Artificial intelligence model training method, device and system based on federated learning

Abstract

The application belongs to the field of electric power automation, and discloses a kind of artificial intelligence model training method, device and system based on federal learning, comprising: obtaining initial artificial intelligence model and sending to each computing node;Receive the initial artificial intelligence model sent by the center node as local artificial intelligence model;Iterative update step is carried out until the preset update iteration threshold or the current artificial intelligence model meets the preset condition, the current artificial intelligence model is used as the training completed artificial intelligence model, and is sent to each computing node;Iterative gradient calculation step is carried out until the preset gradient calculation iteration threshold, and the training completed artificial intelligence model sent by the center node is received, or, iterative gradient calculation step is carried out until the training completed artificial intelligence model sent by the center node is received, and the training completed artificial intelligence model is used to update local artificial intelligence model. Through the asynchronous model parameter updating method, the computing power utilization rate of high-performance computing node is improved, and the model training speed is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of power automation, and relates to an artificial intelligence model training method, device and system based on federated learning. BACKGROUND

[0002] Marketing business is one of the core businesses of a power company, and its operation quality is closely related to the survival and development of the power company. With the continuous expansion and extension of the marketing business scope, marketing data grows rapidly, which contains a large amount of data value. By deeply mining the value of marketing big data, the application of marketing big data is deepened. Specifically, the full-channel customer behavior characteristics are fully utilized, the customer electricity consumption law is dynamically tracked, the electricity consumption customer portrait is constructed, the electricity consumption customer credit risk control system is accelerated to be constructed, the electricity consumption customer credit score is deeply promoted, the core products such as customer confidence index are researched and developed, the State Grid customer credit system is established, the electricity fee recovery early warning monitoring system is enriched, and the electricity consumption fluctuation monitoring, default electricity consumption monitoring and industry risk monitoring services are expanded.

[0003] At present, artificial intelligence technologies such as machine learning and deep learning are usually used to mine the intrinsic information value of marketing data, innovate marketing management mode, and improve management efficiency. In this process, machine learning and deep learning technologies need to use a large amount of data for model training, so that the data sharing needs of horizontal collaboration between internal units of the power company and vertical penetration between departments become increasingly urgent. However, due to the sensitivity of marketing data and the need for privacy protection, it is difficult for each unit in the power company to share marketing data, which easily causes the problem of "data island" between each unit, leads to model training, makes it difficult to fully mine the intrinsic value of marketing data, and causes data value waste.

[0004] At present, the common methods to solve the "data island" problem include data desensitization sharing mechanism and group feature sharing mode. However, the data desensitization sharing mechanism has security risks such as user mismatching and easy backtracking, and even brings legal risks, which does not meet the requirements of power data sharing. While the group feature sharing mode has slightly better privacy protection, but the feature types are limited and cannot be matched with other data, the coverage and effectiveness of sharing are poor, and the data sharing application effect is not good. Therefore, how to ensure the data and user privacy safety of each unit in multi-party data sharing, and on this basis, realize data collaborative training, improve the effectiveness and accuracy of data model, and achieve the effect of data security sharing, is a problem to be solved. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art and provide an artificial intelligence model training method, device and system based on federated learning.

[0006] To achieve the above object, the present application adopts the following technical solutions to achieve it.

[0007] In a first aspect, the present application provides a method for training an artificial intelligence model based on federated learning, applied to a center node of a system for training an artificial intelligence model based on federated learning, the system comprising a center node and a plurality of computing nodes in communication with the center node; the method comprising:

[0008] obtaining an initial artificial intelligence model and sending it to each computing node;

[0009] iteratively performing the updating step until a preset updating iteration threshold is reached or the current artificial intelligence model meets a preset condition, at which point the current artificial intelligence model is taken as a trained artificial intelligence model and sent to each computing node;

[0010] The updating step comprises:

[0011] receiving gradient data sent by a preset number of computing nodes, and entering a global parameter updating process; the gradient data is the gradient of each neural network layer of the local artificial intelligence model calculated by the computing nodes based on local training data;

[0012] The global parameter updating process comprises: updating the global parameters of the artificial intelligence model based on the gradient data sent by the preset number of computing nodes, and sending the updated global parameters to the preset number of computing nodes; the updated global parameters are used to update the model parameters of the local artificial intelligence model of the computing nodes.

[0013] Optionally, the updating step further comprises:

[0014] sending feedback information to the computing nodes that have completed gradient calculation in the global parameter updating process; the feedback information is used to trigger the computing nodes to update the model parameters of the local artificial intelligence model based on the local gradient data.

[0015] Optionally, the updating of the global parameters of the artificial intelligence model based on the gradient data sent by the preset number of computing nodes comprises:

[0016] averaging the gradient data sent by the preset number of computing nodes to obtain average gradient data;

[0017] updating the global parameters of the artificial intelligence model using the backpropagation algorithm based on the average gradient data.

[0018] In a second aspect, the application discloses a method for training an artificial intelligence model based on federated learning, which is applied to a computing node of a system for training an artificial intelligence model based on federated learning, wherein the system comprises a center node and a plurality of computing nodes connected to the center node; and the method comprises the following steps:

[0019] receiving an initial artificial intelligence model sent by the center node as a local artificial intelligence model;

[0020] iteratively performing the gradient calculation step until a preset gradient calculation iteration threshold is reached, and receiving a trained artificial intelligence model sent by the center node, or iteratively performing the gradient calculation step until the trained artificial intelligence model sent by the center node is received, and updating the local artificial intelligence model by using the trained artificial intelligence model;

[0021] wherein the gradient calculation step comprises the following steps:

[0022] calculating the gradients of each neural network layer of the local artificial intelligence model by using the local training data, and sending the gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data when the current node is one of the first preset number of computing nodes that have completed the gradient calculation;

[0023] updating the model parameters of the local artificial intelligence model according to the updated global parameters when the updated global parameters sent by the center node are received.

[0024] Optionally, the gradient calculation step further comprises the following steps:

[0025] updating the model parameters of the local artificial intelligence model according to the local gradient data in response to the feedback information when the feedback information sent by the center node is received.

[0026] Optionally, the step of sending the gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data comprises the following steps:

[0027] calculating the information entropy of the gradients of each neural network layer of the local artificial intelligence model according to the gradients of each neural network layer of the local artificial intelligence model, and determining the gradient threshold of each neural network layer of the local artificial intelligence model according to the information entropy of the gradients of each neural network layer of the local artificial intelligence model;

[0028] clipping the gradients of each neural network layer of the local artificial intelligence model that are smaller than the gradient threshold of each neural network layer according to the gradient threshold of each neural network layer of the local artificial intelligence model, and sending the clipped gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data.

[0029] Optionally, the obtaining, according to the gradient of each neural network layer of the local artificial intelligence model, information entropy of the gradient of each neural network layer of the local artificial intelligence model, and determining, according to the information entropy of the gradient of each neural network layer of the local artificial intelligence model, a gradient threshold of each neural network layer of the local artificial intelligence model comprises:

[0030] Obtaining the maximum gradient and the minimum gradient in the gradient of each neural network layer of the local artificial intelligence model;

[0031] Dividing an interval between the maximum gradient and the minimum gradient into a plurality of continuous sub-intervals;

[0032] Putting the gradient of each neural network layer of the local artificial intelligence model into each sub-interval according to the value size, counting the number of gradients in each sub-interval, and calculating the proportion of the number of gradients in each sub-interval to the total number of gradients of each neural network layer of the local artificial intelligence model to obtain the frequency p of the gradient in each sub-interval i ;

[0033] The information entropy H of the gradient of each neural network layer of the local artificial intelligence model is obtained by the following formula j :

[0034]

[0035] Wherein, j is the neural network layer number, i is the sub-interval number, and N is the total number of sub-intervals;

[0036] The gradient threshold τ of each neural network layer of the local artificial intelligence model is determined by the following formula j :

[0037]

[0038] Wherein, K is a preset hyperparameter.

[0039] In the third aspect of the present application, a federated learning-based artificial intelligence model training method is applied to a federated learning-based artificial intelligence model training system, the federated learning-based artificial intelligence model training system comprising a center node and a plurality of computing nodes in communication connection with the center node; the federated learning-based artificial intelligence model training method comprising:

[0040] The center node obtains an initial artificial intelligence model and sends it to each computing node;

[0041] The computing node receives the initial artificial intelligence model sent by the center node as a local artificial intelligence model;

[0042] The center node iteratively performs the updating step until a preset updating iteration threshold is reached or the current artificial intelligence model meets a preset condition, and then the current artificial intelligence model is taken as a trained artificial intelligence model and is sent to each computing node.

[0043] The updating step includes:

[0044] The gradient data sent by the first preset number of computing nodes is received, and a global parameter updating process is entered;

[0045] The global parameter updating process: according to the gradient data sent by the first preset number of computing nodes, the global parameters of the artificial intelligence model are updated, and the updated global parameters are sent to the first preset number of computing nodes;

[0046] The gradient calculation step is iteratively performed until a preset gradient calculation iteration threshold is reached, and the trained artificial intelligence model sent by the center node is received, or the gradient calculation step is iteratively performed until the trained artificial intelligence model sent by the center node is received, and the trained artificial intelligence model is used to update the local artificial intelligence model;

[0047] The gradient calculation step includes:

[0048] According to the local training data, the gradients of each neural network layer of the local artificial intelligence model are calculated by gradient calculation, and when the current node is one of the first preset number of computing nodes that have completed gradient calculation, the gradients of each neural network layer of the local artificial intelligence model are sent to the center node as gradient data.

[0049] When the updated global parameters sent by the center node are received, the model parameters of the local artificial intelligence model are updated according to the updated global parameters.

[0050] Optionally, the updating step further includes: sending feedback information to the computing nodes that have completed gradient calculation in the global parameter updating process; and the gradient calculation step further includes: when the feedback information sent by the center node is received, the model parameters of the local artificial intelligence model are updated according to the local gradient data in response to the feedback information.

[0051] Optionally, the sending of the gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data includes:

[0052] According to the gradients of each neural network layer of the local artificial intelligence model, the information entropy of the gradients of each neural network layer of the local artificial intelligence model is obtained, and according to the information entropy of the gradients of each neural network layer of the local artificial intelligence model, the gradient threshold of each neural network layer of the local artificial intelligence model is determined;

[0053] According to the gradient threshold of each neural network layer of the local artificial intelligence model, the gradients of each neural network layer of the local artificial intelligence model that are less than the gradient threshold of each neural network layer are pruned, and the pruned gradients of each neural network layer of the local artificial intelligence model are sent to the center node as gradient data.

[0054] In a fourth aspect, the application discloses an artificial intelligence model training device based on federated learning, which is applied to a center node of an artificial intelligence model training system based on federated learning, and the artificial intelligence model training system based on federated learning comprises the center node and a plurality of computing nodes in communication connection with the center node.

[0055] The acquisition module is configured to acquire an initial artificial intelligence model and send the initial artificial intelligence model to the computing nodes.

[0056] The first iteration control module is configured to iteratively perform the updating step until a preset updating iteration threshold is reached or the current artificial intelligence model meets a preset condition, and then the current artificial intelligence model is taken as a trained artificial intelligence model and sent to the computing nodes.

[0057] The updating step comprises the following steps.

[0058] The center node receives gradient data sent by the first preset number of computing nodes and enters a global parameter updating process, wherein the gradient data is the gradient of each neural network layer of the local artificial intelligence model calculated by the computing nodes according to local training data.

[0059] The global parameter updating process comprises the following steps.

[0060] Optionally, the updating step further comprises the following steps.

[0061] The center node sends feedback information to the computing nodes that complete the gradient calculation in the global parameter updating process, wherein the feedback information is used to trigger the computing nodes to update the model parameters of the local artificial intelligence model according to the local gradient data.

[0062] Optionally, the updating of the global parameters of the artificial intelligence model according to the gradient data sent by the first preset number of computing nodes comprises the following steps.

[0063] The center node averages the gradient data sent by the first preset number of computing nodes to obtain average gradient data.

[0064] The center node updates the global parameters of the artificial intelligence model by using a back propagation algorithm according to the average gradient data.

[0065] In a fifth aspect, the present application provides a device for training an artificial intelligence model based on federated learning, which is applied to a computing node of a system for training an artificial intelligence model based on federated learning. The system for training an artificial intelligence model based on federated learning comprises a center node and a plurality of computing nodes connected in communication with the center node. The device for training an artificial intelligence model based on federated learning comprises:

[0066] a receiving module configured to receive an initial artificial intelligence model sent by the center node as a local artificial intelligence model;

[0067] a second iteration control module configured to iteratively perform a gradient calculation step until a preset gradient calculation iteration threshold is reached, and receive a trained artificial intelligence model sent by the center node, or iteratively perform the gradient calculation step until the trained artificial intelligence model sent by the center node is received, and update the local artificial intelligence model using the trained artificial intelligence model;

[0068] wherein the gradient calculation step comprises:

[0069] calculating, according to local training data, gradients of each neural network layer of the local artificial intelligence model, and sending, when the current node is one of the first preset number of computing nodes that have completed the gradient calculation, the gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data;

[0070] updating, when the updated global parameters sent by the center node are received, the model parameters of the local artificial intelligence model according to the updated global parameters.

[0071] Optionally, the gradient calculation step further comprises:

[0072] updating, when the feedback information sent by the center node is received, the model parameters of the local artificial intelligence model according to the local gradient data in response to the feedback information.

[0073] Optionally, the sending of the gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data comprises:

[0074] obtaining, according to the gradients of each neural network layer of the local artificial intelligence model, information entropy of the gradients of each neural network layer of the local artificial intelligence model, and determining, according to the information entropy of the gradients of each neural network layer of the local artificial intelligence model, a gradient threshold of each neural network layer of the local artificial intelligence model;

[0075] clipping, according to the gradient threshold of each neural network layer of the local artificial intelligence model, gradients of each neural network layer of the local artificial intelligence model that are less than the gradient threshold of each neural network layer, and sending, to the center node, the clipped gradients of each neural network layer of the local artificial intelligence model as gradient data.

[0076] Optionally, the obtaining the information entropy of the gradient of each neural network layer of the local artificial intelligence model according to the gradient of each neural network layer of the local artificial intelligence model, and determining the gradient threshold of each neural network layer of the local artificial intelligence model according to the information entropy of the gradient of each neural network layer of the local artificial intelligence model comprises:

[0077] obtaining the maximum gradient and the minimum gradient in the gradient of each neural network layer of the local artificial intelligence model;

[0078] dividing the interval between the maximum gradient and the minimum gradient into a plurality of continuous subintervals;

[0079] putting the gradient of each neural network layer of the local artificial intelligence model into each subinterval according to the value, counting the number of gradients in each subinterval, and calculating the proportion of the number of gradients in each subinterval to the total number of gradients of each neural network layer of the local artificial intelligence model to obtain the frequency p of the gradient in each subinterval i ;

[0080] obtaining the information entropy H of the gradient of each neural network layer of the local artificial intelligence model by the following formula j ;

[0081]

[0082] wherein j is the neural network layer number, i is the subinterval number, and N is the total number of subintervals;

[0083] determining the gradient threshold τ of each neural network layer of the local artificial intelligence model by the following formula j ;

[0084]

[0085] wherein K is a preset hyperparameter.

[0086] In the sixth aspect of the present application, an artificial intelligence model training system based on federated learning comprises a center node and a plurality of computing nodes in communication connection with the center node;

[0087] The center node is provided with the artificial intelligence model training device based on federated learning applied to the center node of the artificial intelligence model training system based on federated learning;

[0088] The computing node is provided with the artificial intelligence model training device based on federated learning applied to the computing node of the artificial intelligence model training system based on federated learning.

[0089] Compared with the prior art, the present application has the following beneficial effects:

[0090] The application is based on a federated learning artificial intelligence model training method, in each iteration process, only the gradient data sent by the first preset number of computing nodes is selected as the basis for updating the global parameters, which will cause the computing nodes participating in the global parameter update to be different in each iteration process, forming an asynchronous model parameter update mechanism, although this increases the number of iteration rounds, but since the high-performance computing nodes do not need to wait for the computing nodes with slow computing speed to complete gradient calculation before updating the global parameters, the utilization rate of the computing resources of each computing node is greatly improved, therefore, the training time of the artificial intelligence model can be effectively shortened.

[0091] Further, the application is based on a federated learning artificial intelligence model training method, fully considering the limitation of limited network resources between the center node and each computing node, using a gradient clipping method based on information entropy to reduce the size of the gradient data uploaded by the computing nodes to the center node, thereby effectively saving network bandwidth, reducing the time of gradient data upload and global parameter download in model training, and improving the model training speed. BRIEF DESCRIPTION OF DRAWINGS

[0092] Figure 1 A typical application scenario diagram of the federated learning artificial intelligence model training method of the embodiment of the application is shown in the figure.

[0093] Figure 2 A federated learning artificial intelligence model training method flowchart applied to the center node of the federated learning artificial intelligence model training system of the embodiment of the application is shown in the figure.

[0094] Figure 3 A federated learning artificial intelligence model training method flowchart applied to the computing node of the federated learning artificial intelligence model training system of the embodiment of the application is shown in the figure.

[0095] Figure 4 A gradient clipping principle diagram of the embodiment of the application is shown in the figure.

[0096] Figure 5 A federated learning artificial intelligence model training method flowchart applied to the federated learning artificial intelligence model training system of the embodiment of the application is shown in the figure.

[0097] Figure 6 A federated learning artificial intelligence model training method principle diagram applied to the federated learning artificial intelligence model training system of the embodiment of the application is shown in the figure.

[0098] Figure 7 A federated learning artificial intelligence model training system structure block diagram applied to the center node of the federated learning artificial intelligence model training system of the embodiment of the application is shown in the figure.

[0099] Figure 8 A federated learning-based artificial intelligence model training system structure block diagram of a computing node applied to a federated learning-based artificial intelligence model training system in an embodiment of the present application. DETAILED DESCRIPTION

[0100] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0101] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0102] First, introduce the related terms involved in the embodiments of the present application:

[0103] Electricity inspection: using machine learning method to detect whether the user has committed electricity stealing and other illegal operations in power marketing business.

[0104] Artificial intelligence model training: artificial intelligence model training usually refers to supervised artificial intelligence model training, which needs to use labeled sample data to train the model.

[0105] Data silos: due to sample data privacy issues, data from various parties cannot be collected together for artificial intelligence model training.

[0106] Federated learning: a distributed model training method that can gather data from various parties for artificial intelligence model training while ensuring the privacy of the data from various parties.

[0107] Horizontal federated learning: in federated learning, the data possessed by each party has the same feature dimension and different sample IDs.

[0108] Global parameter update: In federated learning, there are local model training (performed at local nodes of each party) and global model training, and the global model training is also referred to as global parameter update.

[0109] The application will be further described in detail below with reference to the accompanying drawings:

[0110] To solve the above problems in the prior art, the inventors have found in practical work that federated learning can protect data security and user privacy, and can fully utilize scattered data sources to improve the performance of a model, so that the problem of "data islands" existing between units in a company can be solved by introducing federated learning.

[0111] Federated learning is different from traditional machine learning principles. Traditional machine learning can collect data to a central point by one party, and train a machine learning model at the central point. Federated learning is to collaboratively train one or more machine learning models by multiple scattered parties, which can train machine learning and deep learning models by jointly using data sources of multiple parties without data leaving the domain, and provide model inference services. Therefore, based on federated learning, data sharing between units and departments in a company can be realized, data protection laws and regulations can be met, and data sharing security can be ensured.

[0112] However, in the application of a horizontal federated learning model training in power marketing business, it needs to be based on the existing computing environment of a power grid company, and since there is a large difference in computing capacity between departments of a power grid enterprise, and the network resources in a power enterprise are limited, the horizontal federated learning method model training speed is very slow by using a synchronous model parameter update algorithm, and efficient actual application cannot be performed.

[0113] Therefore, in an embodiment of the application, a federated learning-based artificial intelligence model training method is provided, which adopts an asynchronous model parameter update mechanism to effectively utilize the computing resources of each node.

[0114] Specifically, referring to Figure 1 , a typical application scenario of the federated learning-based artificial intelligence model training method is shown, that is, a federated learning-based artificial intelligence model training system, which generally includes a center node and a plurality of computing nodes in communication connection with the center node. The center node and the computing nodes can be separate electronic computers, can be clusters composed of a plurality of electronic computers, or can be local servers or cloud servers. Based on actual configuration and field application, the computing capacity of each computing node will have certain differences.

[0115] Referring to Figure 2The embodiment of the application is based on a federated learning artificial intelligence model training method applied to a center node of a federated learning artificial intelligence model training system. Specifically, the federated learning artificial intelligence model training method comprises the following steps:

[0116] S11: An initial artificial intelligence model is obtained and sent to each computing node.

[0117] Specifically, the center node obtains an artificial intelligence model that needs to be trained, i.e., an initial artificial intelligence model, and then distributes the initial artificial intelligence model to each computing node for parameter initialization.

[0118] For the center node, the model parameters of the artificial intelligence model in the center node are taken as global parameters, and the model parameters of the initial artificial intelligence model are taken as initial global parameters.

[0119] S12: An update step is iteratively performed until a preset update iteration threshold or a current artificial intelligence model meets a preset condition, the current artificial intelligence model is taken as a trained artificial intelligence model, and the trained artificial intelligence model is sent to each computing node.

[0120] Specifically, the preset update iteration threshold can be designed differently according to different actual trained artificial intelligence models, and can be combined with historical experience and the iteration effect of several times. The preset condition can be that a certain index of the artificial intelligence model, such as the difference between the global parameters of the artificial intelligence models obtained by adjacent two iterations, is within a preset range, or the accuracy of the current artificial intelligence model reaches a preset accuracy threshold.

[0121] The update step comprises: receiving gradient data sent by a preset number of computing nodes, and entering a global parameter update process; the gradient data is gradient of each neural network layer of a local artificial intelligence model obtained by gradient calculation according to local training data by the computing nodes; the global parameter update process: updating the global parameters of the artificial intelligence model according to the gradient data sent by the preset number of computing nodes, and sending the updated global parameters to the preset number of computing nodes; the updated global parameters are used to update the model parameters of the local artificial intelligence model of the computing nodes.

[0122] Specifically, the preset number of computing nodes complete gradient calculation and feedback to the center node as an iteration round, which is the starting signal of the global parameter update process. When the preset number of computing nodes are counted, the CPU clock of the center node is taken as the calculation interval. If more than the preset number of computing nodes complete gradient calculation at the same time within one CPU clock of the center node, these computing nodes are all considered as the preset number of computing nodes.

[0123] In a possible implementation, in the global parameter updating process, updating the global parameters of the artificial intelligence model according to the gradient data sent by the first preset number of computing nodes comprises: averaging the gradient data sent by the first preset number of computing nodes to obtain average gradient data; and updating the global parameters of the artificial intelligence model by using the back propagation algorithm according to the average gradient data. When updating the parameters, the type of the optimizer method used is not limited based on the universality, such as SGD and Adam.

[0124] After the global parameter updating of the artificial intelligence model is completed, the updated global parameters are sent to the first preset number of computing nodes, and then the model parameters of the local artificial intelligence models of the first preset number of computing nodes are updated to the updated global parameters, that is, only the computing nodes that have sent the gradient data are synchronized.

[0125] In a possible implementation, the updating step further comprises: sending feedback information to the computing nodes that have completed the gradient calculation in the global parameter updating process; and the feedback information is used to trigger the computing nodes to update the model parameters of the local artificial intelligence models according to the local gradient data.

[0126] Specifically, after the gradient calculation is completed, each computing node can inform the center node that the gradient calculation is completed in the form of a completion signal feedback to the center node, or the gradient calculation completion of each computing node can be determined in an artificial monitoring manner, which is not limited herein.

[0127] The gradient data of the computing nodes that have completed the gradient calculation in the global parameter updating process does not participate in the current iteration round, until the computing nodes have the condition to participate in the global parameter updating process, that is, the gradient data and the model parameters of the local artificial intelligence models are uploaded and synchronized to the global parameters after the computing nodes participate in an iteration round. In this process, in order to avoid that the computing nodes have a large difference with the gradient data of other computing nodes due to a long time without parameter updating in the continuous iteration process, and thus affect the process of updating the global parameters according to the average gradient data, the center node sends feedback information to the computing nodes that have completed the gradient calculation in the global parameter updating process, so that the computing nodes update the model parameters of the local artificial intelligence models according to the local gradient data, so as to ensure that the gradient data of the computing nodes is approximate to the gradient data of other computing nodes.

[0128] To sum up, the artificial intelligence model training method based on federated learning in the embodiment selects the gradient data sent by the first preset number of computing nodes as the basis for updating the global parameters in each iteration process, which results in that the computing nodes participating in the global parameter update are different in each iteration process, forming an asynchronous model parameter update mechanism. Although this increases the number of iteration rounds, the utilization rate of the computing resources of each computing node is greatly improved because the high-performance computing nodes do not need to wait for the computing nodes with slower computing speed to complete gradient calculation before updating the global parameters, so the training time of the artificial intelligence model can be effectively shortened.

[0129] Referring to Figure 3 In another embodiment of the present application, an artificial intelligence model training method based on federated learning is provided. The artificial intelligence model training method based on federated learning is applied to the computing nodes of an artificial intelligence model training system based on federated learning. Specifically, the artificial intelligence model training method based on federated learning comprises the following steps:

[0130] S21: receiving an initial artificial intelligence model sent by a center node as a local artificial intelligence model.

[0131] Specifically, the computing node receives the initial artificial intelligence model sent by the center node and takes it as the local artificial intelligence model to train the model through local training data.

[0132] S22: iteratively performing the gradient calculation step until a preset gradient calculation iteration threshold is reached, and receiving a training completed artificial intelligence model sent by the center node, or iteratively performing the gradient calculation step until a training completed artificial intelligence model sent by the center node is received; and updating the local artificial intelligence model using the training completed artificial intelligence model.

[0133] Specifically, the computing ability of different computing nodes and the training data are considered, and therefore different gradient calculation iteration thresholds are set for different computing nodes. For computing nodes with weak computing ability and less training data, the gradient calculation iteration is stopped when the preset gradient calculation iteration threshold is reached, and the training completed artificial intelligence model sent by the center node is received. For computing nodes with strong computing ability and more training data, the training completed artificial intelligence model sent by the center node may be received before the preset gradient calculation iteration threshold is reached, and the gradient calculation iteration is also stopped. Finally, the local artificial intelligence model is updated using the training completed artificial intelligence model.

[0134] The gradient calculation step comprises: obtaining, according to the local training data, gradients of each neural network layer of the local artificial intelligence model through gradient calculation, and sending the gradients of each neural network layer of the local artificial intelligence model as gradient data to the center node when the current node is one of the first preset number of computing nodes that complete gradient calculation; and updating the model parameters of the local artificial intelligence model according to the updated global parameters when the updated global parameters sent by the center node are received.

[0135] Specifically, the gradient calculation of each computing node according to the local training data is a repeated process. Whether the current node is one of the first preset number of computing nodes that complete gradient calculation can be determined by the center node directly or indirectly. Those computing nodes that are the first preset number of computing nodes that complete gradient calculation in different iteration rounds can be determined by the center node directly or indirectly. Some computing nodes can complete gradient calculation only once in multiple iteration rounds of the center node, but this does not affect the fact that they are the first preset number of computing nodes that complete gradient calculation in a certain iteration round.

[0136] In a possible implementation, the gradient calculation step further comprises: when the feedback information sent by the center node is received, updating the model parameters of the local artificial intelligence model according to the local gradient data in response to the feedback information.

[0137] Specifically, when the computing node receives the feedback information sent by the center node, it means that the computing node completes gradient calculation in the global parameter updating process of the center node. At this time, in order to avoid the difference between the gradient data of different computing nodes from increasing continuously, the center node sends feedback information to these computing nodes, the computing nodes respond to the feedback information, and then update the model parameters of the local artificial intelligence model according to the local gradient data.

[0138] When the model parameters of the local artificial intelligence model are updated according to the local gradient data, the back propagation algorithm is used for updating, and different optimizers can be used for different computing nodes.

[0139] In a possible implementation, the sending of the gradients of each neural network layer of the local artificial intelligence model as gradient data to the center node comprises: obtaining the information entropy of the gradients of each neural network layer of the local artificial intelligence model according to the gradients of each neural network layer of the local artificial intelligence model, and determining the gradient threshold of each neural network layer of the local artificial intelligence model according to the information entropy of the gradients of each neural network layer of the local artificial intelligence model; clipping the gradients of each neural network layer of the local artificial intelligence model that are less than the gradient threshold of each neural network layer according to the gradient threshold of each neural network layer of the local artificial intelligence model, and sending the clipped gradients of each neural network layer of the local artificial intelligence model as gradient data to the center node.

[0140] Specifically, due to the limited network resources between the center node and each computing node, in order to reduce the influence of network bandwidth on the training speed of the artificial intelligence model based on federated learning, the gradient is pruned based on information entropy before uploading the gradient data to the computing node, thereby effectively saving network bandwidth and ultimately improving the training speed.

[0141] Specifically, the importance of the gradient of each neural network layer is determined by calculating the information entropy of each neural network layer. In information theory, it is believed that all information has redundancy, and the degree of redundancy of information is related to the probability (uncertainty) of each number in the information. Therefore, information entropy is generally used to measure the amount of information. The greater the value of information entropy, the more information the gradient of each neural network layer contains. In the training of a deep neural network, if the information entropy of the gradient of a certain neural network layer is relatively small, the amount of information contained in the gradient of the neural network layer is relatively small, and the influence on the convergence of the artificial intelligence model training should also be relatively small. Therefore, the gradient information dimension of this neural network layer can be pruned to reduce communication consumption.

[0142] The principle of gradient pruning is shown in Figure 4 Figure 4 The upper half is a schematic diagram of the neural network model structure and the gradient value generated during its training process. The model gradient value of the i-th layer is G[i], which is a 1xn vector. mask is the gradient mask obtained by comparing each value in G[i] vector with the pruning threshold. It preserves the gradient values in G[i] vector greater than the pruning threshold by i.e. vector product to obtain the pruned gradient vector Figure 4 The lower half is a schematic diagram of the pruned network model structure and the pruned gradient. The white circles represent the gradient values that are deleted because they are less than the pruning threshold. In a possible implementation, the method further includes: obtaining the maximum gradient and the minimum gradient in the gradient of each neural network layer of the local artificial intelligence model; dividing the interval between the maximum gradient and the minimum gradient into a plurality of continuous subintervals, wherein the number of subintervals is determined by experience and can be set to 5.

[0143] Then, the gradient of each neural network layer of the local artificial intelligence model is placed in each subinterval according to the size of the value, the number of gradients in each subinterval is counted, and the proportion of the number of gradients in each subinterval to the total number of gradients of each neural network layer of the local artificial intelligence model is calculated to obtain the frequency p of the gradient in each subinterval. i ​The frequency of gradient occurrence in a sub-interval can be approximated as the probability that the gradient falls within that sub-interval.

[0144] Then, the information entropy H of the gradient of each neural network layer of the local artificial intelligence model is obtained by the following formula. j : Where j is the neural network layer number, i is the sub-interval number, and N is the total number of sub-intervals; the gradient threshold τ of each neural network layer of the local artificial intelligence model is determined by the following formula. j : Where K is a preset hyperparameter, representing the proportion of gradients to be discarded; typically, K is set to 2. 10 .

[0145] Then, the gradient clipping threshold τ of this neural network layer is used. j The gradient G of this neural network layer j Perform cropping, retaining only those values ​​greater than the gradient cropping threshold τ. j The portion is sent to the central node as gradient data.

[0146] In summary, this embodiment of the AI ​​model training method based on federated learning fully considers the limited network resources between the central node and each computing node. It utilizes an information entropy-based gradient pruning method to reduce the size of gradient data uploaded by computing nodes to the central node, thereby effectively saving network bandwidth, reducing the time for gradient data upload and global parameter download during model training, and improving the model training speed.

[0147] See Figure 5 and 6 In another embodiment of the present invention, a method for training an artificial intelligence model based on federated learning is provided. This method is applied to an artificial intelligence model training system based on federated learning. Specifically, the method includes the following steps:

[0148] S31: The central node obtains the initial artificial intelligence model and sends it to each computing node.

[0149] S32: The computing node receives the initial artificial intelligence model sent by the central node and uses it as its local artificial intelligence model.

[0150] S33: The center node iteratively performs an updating step until a preset updating iteration threshold or the current artificial intelligence model meets a preset condition, and sends the current artificial intelligence model as a trained artificial intelligence model to each computing node; the updating step includes: receiving gradient data sent by the previous preset number of computing nodes, and entering a global parameter updating process; the global parameter updating process: updating the global parameters of the artificial intelligence model according to the gradient data sent by the previous preset number of computing nodes, and sending the updated global parameters to the previous preset number of computing nodes.

[0151] S34: iteratively performing a gradient calculation step until a preset gradient calculation iteration threshold is reached, and receiving a trained artificial intelligence model sent by the center node, or iteratively performing the gradient calculation step until the trained artificial intelligence model sent by the center node is received, and updating the local artificial intelligence model using the trained artificial intelligence model; the gradient calculation step includes: obtaining the gradients of each neural network layer of the local artificial intelligence model through gradient calculation according to the local training data, and when the current node is one of the previous preset number of computing nodes that have completed gradient calculation, sending the gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data; when the updated global parameters sent by the center node are received, updating the model parameters of the local artificial intelligence model according to the updated global parameters.

[0152] In a possible implementation, the updating step further includes: sending feedback information to the computing nodes that have completed gradient calculation in the global parameter updating process; the gradient calculation step further includes: when the feedback information sent by the center node is received, updating the model parameters of the local artificial intelligence model according to the local gradient data in response to the feedback information.

[0153] In a possible implementation, the sending of the gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data includes: obtaining the information entropy of the gradients of each neural network layer of the local artificial intelligence model according to the gradients of each neural network layer of the local artificial intelligence model, and determining the gradient threshold of each neural network layer of the local artificial intelligence model according to the information entropy of the gradients of each neural network layer of the local artificial intelligence model; clipping the gradients of each neural network layer of the local artificial intelligence model that are less than the gradient threshold of each neural network layer according to the gradient threshold of each neural network layer of the local artificial intelligence model, and sending the clipped gradients of each neural network layer of the local artificial intelligence model to the center node as gradient data.

[0154] In a possible implementation, the artificial intelligence model training method based on federated learning can be applied to training of a power consumption inspection model of a power marketing business, so as to avoid the defect of slow training speed caused by the fact that the training of the power consumption inspection model in the power marketing business needs to be based on the existing computing environment of a power grid company, the computing capacity of departments of the power grid enterprise is quite different, and the network resources of the power enterprise are limited.

[0155] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiment, please refer to the method embodiments of the present application.

[0156] Referring to Figure 7 In another embodiment of the present application, an artificial intelligence model training apparatus based on federated learning is provided, which is applied to a center node of an artificial intelligence model training system based on federated learning and can be used to implement the artificial intelligence model training method based on federated learning applied to the center node of the artificial intelligence model training system based on federated learning. Specifically, the artificial intelligence model training apparatus based on federated learning comprises an acquisition module and a first iteration control module.

[0157] The acquisition module is configured to acquire an initial artificial intelligence model and send the initial artificial intelligence model to each computing node. The first iteration control module is configured to iteratively perform an updating step until a preset updating iteration threshold is reached or a current artificial intelligence model meets a preset condition, and then take the current artificial intelligence model as a trained artificial intelligence model and send the trained artificial intelligence model to each computing node. The updating step comprises: receiving gradient data sent by a preset number of previous computing nodes, and entering a global parameter updating process. The gradient data is gradient of each neural network layer of a local artificial intelligence model obtained by the computing node through gradient calculation based on local training data. The global parameter updating process comprises: updating global parameters of the artificial intelligence model according to the gradient data sent by the preset number of previous computing nodes, and sending the updated global parameters to the preset number of previous computing nodes. The updated global parameters are used to update model parameters of the local artificial intelligence model of the computing node.

[0158] In a possible implementation, the updating step further comprises: sending feedback information to the computing node that completes the gradient calculation in the global parameter updating process. The feedback information is used to trigger the computing node to update the model parameters of the local artificial intelligence model according to the local gradient data.

[0159] In a possible implementation, the updating of the global parameters of the artificial intelligence model according to the gradient data sent by the preset number of previous computing nodes comprises: averaging the gradient data sent by the preset number of previous computing nodes to obtain average gradient data; and updating the global parameters of the artificial intelligence model using a back propagation algorithm according to the average gradient data.

[0160] Referring to Figure 8 In still another embodiment of the present application, a device for training an artificial intelligence model based on federated learning is provided, which is applied to a computing node of a system for training an artificial intelligence model based on federated learning and can be used to implement the method for training an artificial intelligence model based on federated learning applied to the computing node of the system for training an artificial intelligence model based on federated learning. Specifically, the device for training an artificial intelligence model based on federated learning comprises a receiving module and a second iteration control module.

[0161] The receiving module is configured to receive an initial artificial intelligence model sent by the center node as a local artificial intelligence model. The second iteration control module is configured to iteratively perform a gradient calculation step until a preset gradient calculation iteration threshold is reached, and receive a trained artificial intelligence model sent by the center node, or iteratively perform the gradient calculation step until the trained artificial intelligence model sent by the center node is received, and update the local artificial intelligence model using the trained artificial intelligence model. The gradient calculation step comprises: obtaining gradients of each neural network layer of the local artificial intelligence model through gradient calculation according to local training data, and sending the gradients of each neural network layer of the local artificial intelligence model as gradient data to the center node when the current node is one of the first preset number of computing nodes that have completed gradient calculation. When the updated global parameters sent by the center node are received, the model parameters of the local artificial intelligence model are updated according to the updated global parameters.

[0162] In a possible implementation, the gradient calculation step further comprises: when the feedback information sent by the center node is received, updating the model parameters of the local artificial intelligence model according to the local gradient data in response to the feedback information.

[0163] In a possible implementation, the sending of the gradients of each neural network layer of the local artificial intelligence model as gradient data to the center node comprises: obtaining the information entropy of the gradients of each neural network layer of the local artificial intelligence model according to the gradients of each neural network layer of the local artificial intelligence model, and determining the gradient threshold of each neural network layer of the local artificial intelligence model according to the information entropy of the gradients of each neural network layer of the local artificial intelligence model. The gradients of each neural network layer of the local artificial intelligence model that are less than the gradient threshold of each neural network layer are pruned according to the gradient threshold of each neural network layer of the local artificial intelligence model, and the pruned gradients of each neural network layer of the local artificial intelligence model are sent to the center node as gradient data.

[0164] In a possible implementation, the obtaining, according to the gradient of each neural network layer of the local artificial intelligence model, the information entropy of the gradient of each neural network layer of the local artificial intelligence model, and determining the gradient threshold of each neural network layer of the local artificial intelligence model according to the information entropy of the gradient of each neural network layer of the local artificial intelligence model comprises: obtaining the maximum gradient and the minimum gradient in the gradient of each neural network layer of the local artificial intelligence model; dividing the interval between the maximum gradient and the minimum gradient into a plurality of continuous sub-intervals; putting the gradient of each neural network layer of the local artificial intelligence model into each sub-interval according to the value, counting the number of gradients in each sub-interval, and calculating the proportion of the number of gradients in each sub-interval to the total number of gradients of each neural network layer of the local artificial intelligence model to obtain the frequency p of the gradient in each sub-interval i ; the information entropy H of the gradient of each neural network layer of the local artificial intelligence model is obtained by the following formula j wherein j is the neural network layer number, i is the sub-interval number, and N is the total number of sub-intervals; the gradient threshold τ of each neural network layer of the local artificial intelligence model is determined by the following formula j wherein K is a preset hyperparameter.

[0165] In another embodiment of the present application, a federated learning-based artificial intelligence model training system is provided, comprising a center node and a plurality of computing nodes in communication connection with the center node; wherein the center node is provided with the federated learning-based artificial intelligence model training device applied to the center node of the federated learning-based artificial intelligence model training system; the computing node is provided with the federated learning-based artificial intelligence model training device applied to the computing node of the federated learning-based artificial intelligence model training system.

[0166] The foregoing embodiments of the federated learning-based artificial intelligence model training method involve all related contents of each step, which can be referred to the function description of the corresponding function module of the federated learning-based artificial intelligence model training device in the embodiments of the present application, and will not be repeated here. The division of modules in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, another division mode can be used. In addition, each function module in each embodiment of the present application can be integrated in one processor, or can be physically separated, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of software function module.

[0167] ​​In another embodiment of the present application, a computer device is provided, which comprises a processor and a memory, the memory is configured to store a computer program, the computer program comprises program instructions, and the processor is configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are particularly suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method process or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the artificial intelligence model training method based on federated learning.

[0168] In another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a computer device, and is configured to store programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the computer device, and of course can also include an expansion storage medium supported by the computer device. The computer readable storage medium provides a storage space, and the storage space stores an operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the artificial intelligence model training method based on federated learning in the above embodiments.

[0169] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0170] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0171] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0173] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing the technical solutions of the present application, but not for limiting it. Although the present application is described in detail with reference to the above embodiments, those skilled in the field should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. A method for training an artificial intelligence model based on federated learning, characterized in that, A central node is applied to a federated learning-based artificial intelligence model training system, the federated learning-based artificial intelligence model training system including a central node and several computing nodes communicatively connected to the central node; the federated learning-based artificial intelligence model training method includes: Obtain the initial artificial intelligence model and send it to each computing node; The update process is iterated until a preset update iteration threshold is reached or the current artificial intelligence model meets preset conditions. Then, the current artificial intelligence model is sent as the trained artificial intelligence model to each computing node. The update step includes: After receiving gradient data sent by a preset number of computing nodes, the system enters the global parameter update process. The gradient data consists of the gradients of each neural network layer of the local artificial intelligence model, which are calculated by the computing nodes based on the local training data. Global parameter update process: Based on the gradient data sent by the previous preset number of computing nodes, update the global parameters of the artificial intelligence model, and send the updated global parameters to the previous preset number of computing nodes; wherein, the updated global parameters are used to update the model parameters of the local artificial intelligence model of the computing nodes. The update step also includes: Feedback information is sent to the computing nodes that have completed gradient calculations during the global parameter update process; the feedback information is used to trigger the computing nodes to update the model parameters of the local artificial intelligence model based on the local gradient data. Among them, the preset number of computing nodes are calculated with the CPU computing clock of the central node as the computing interval. If more than the preset number of computing nodes complete the gradient calculation simultaneously within the CPU computing clock of a central node, then these computing nodes are all considered as the first preset number of computing nodes.

2. The method for training an artificial intelligence model based on federated learning according to claim 1, characterized in that, The step of updating the global parameters of the artificial intelligence model based on the gradient data sent by the pre-set number of computing nodes includes: The average gradient data is obtained by averaging the gradient data sent by a preset number of computing nodes. Based on the average gradient data, the backpropagation algorithm is used to update the global parameters of the artificial intelligence model.

3. A method for training an artificial intelligence model based on federated learning, characterized in that, A computing node applied to a federated learning-based artificial intelligence model training system, the federated learning-based artificial intelligence model training system including a central node and several computing nodes communicatively connected to the central node; the federated learning-based artificial intelligence model training method includes: Receive the initial artificial intelligence model sent by the central node and use it as the local artificial intelligence model; The gradient calculation steps are iterated until a preset gradient calculation iteration threshold is reached, and the trained artificial intelligence model is received from the central node; or, the gradient calculation steps are iterated until the trained artificial intelligence model is received from the central node, and the trained artificial intelligence model is used to update the local artificial intelligence model. The gradient calculation step includes: Based on local training data, the gradients of each neural network layer of the local artificial intelligence model are obtained through gradient calculation. When the current node is one of the first preset number of calculation nodes that have completed gradient calculation, the gradients of each neural network layer of the local artificial intelligence model are sent to the central node as gradient data. When the updated global parameters are received from the central node, the model parameters of the local artificial intelligence model are updated according to the updated global parameters. The gradient calculation step further includes: when receiving feedback information sent by the central node, responding to the feedback information and updating the model parameters of the local artificial intelligence model based on the local gradient data; Among them, the preset number of computing nodes are calculated with the CPU computing clock of the central node as the computing interval. If more than the preset number of computing nodes complete the gradient calculation simultaneously within the CPU computing clock of a central node, then these computing nodes are all considered as the first preset number of computing nodes.

4. The method for training an artificial intelligence model based on federated learning according to claim 3, characterized in that, The step of sending the gradients of each neural network layer of the local artificial intelligence model as gradient data to the central node includes: Based on the gradients of each neural network layer of the local artificial intelligence model, the information entropy of the gradients of each neural network layer of the local artificial intelligence model is obtained, and the gradient threshold of each neural network layer of the local artificial intelligence model is determined based on the information entropy of the gradients of each neural network layer of the local artificial intelligence model. Based on the gradient threshold of each neural network layer in the local AI model, the gradients of each neural network layer in the local AI model that are smaller than the gradient threshold of each neural network layer are clipped, and the clipped gradients of each neural network layer in the local AI model are sent as gradient data to the central node.

5. The method for training an artificial intelligence model based on federated learning according to claim 4, characterized in that, The step of obtaining the information entropy of the gradient of each neural network layer of the local artificial intelligence model based on the gradient of each neural network layer, and determining the gradient threshold of each neural network layer of the local artificial intelligence model based on the information entropy of the gradient of each neural network layer of the local artificial intelligence model includes: Obtain the maximum and minimum gradients among the gradients of each neural network layer in the local artificial intelligence model; Divide the interval between the maximum gradient and the minimum gradient into several consecutive sub-intervals; The gradients of each neural network layer in the local AI model are placed into sub-intervals according to their values. The number of gradients in each sub-interval is counted, and the proportion of the number of gradients in each sub-interval to the total number of gradients in all neural network layers of the local AI model is calculated to obtain the frequency of gradient occurrence in each sub-interval. ; The information entropy of the gradients of each neural network layer in the local artificial intelligence model is obtained by the following formula. : in, Number the layers of the neural network. For sub-interval numbering, This represents the total number of sub-intervals. The gradient thresholds for each neural network layer of the local AI model are determined using the following formula. : in, These are the preset hyperparameters.

6. A method for training an artificial intelligence model based on federated learning, characterized in that, An AI model training system based on federated learning is provided, comprising a central node and several computing nodes communicatively connected to the central node; the AI ​​model training method based on federated learning includes: The central node acquires the initial artificial intelligence model and sends it to each computing node; The computing node receives the initial artificial intelligence model sent by the central node and uses it as its local artificial intelligence model; The central node iterates through the update steps until a preset update iteration threshold is reached or the current artificial intelligence model meets preset conditions. Then, the current artificial intelligence model is sent as the trained artificial intelligence model to each computing node. The update step includes: After receiving gradient data from a predetermined number of computing nodes, the global parameter update process begins. Global parameter update process: Based on the gradient data sent by the previous preset number of computing nodes, update the global parameters of the artificial intelligence model, and send the updated global parameters to the previous preset number of computing nodes; The gradient calculation steps are iterated until a preset gradient calculation iteration threshold is reached, and the trained artificial intelligence model is received from the central node; or, the gradient calculation steps are iterated until the trained artificial intelligence model is received from the central node, and the trained artificial intelligence model is used to update the local artificial intelligence model. The gradient calculation step includes: Based on local training data, the gradients of each neural network layer of the local artificial intelligence model are obtained through gradient calculation. When the current node is one of the first preset number of calculation nodes that have completed gradient calculation, the gradients of each neural network layer of the local artificial intelligence model are sent to the central node as gradient data. When the updated global parameters are received from the central node, the model parameters of the local artificial intelligence model are updated according to the updated global parameters. The update step further includes: sending feedback information to the computing node that completes gradient calculation during the global parameter update process; the gradient calculation step further includes: when receiving feedback information sent by the central node, responding to the feedback information and updating the model parameters of the local artificial intelligence model according to the local gradient data; Among them, the preset number of computing nodes are calculated with the CPU computing clock of the central node as the computing interval. If more than the preset number of computing nodes complete the gradient calculation simultaneously within the CPU computing clock of a central node, then these computing nodes are all considered as the first preset number of computing nodes.

7. The method for training an artificial intelligence model based on federated learning according to claim 6, characterized in that, The step of sending the gradients of each neural network layer of the local artificial intelligence model as gradient data to the central node includes: Based on the gradients of each neural network layer of the local artificial intelligence model, the information entropy of the gradients of each neural network layer of the local artificial intelligence model is obtained, and the gradient threshold of each neural network layer of the local artificial intelligence model is determined based on the information entropy of the gradients of each neural network layer of the local artificial intelligence model. Based on the gradient threshold of each neural network layer in the local AI model, the gradients of each neural network layer in the local AI model that are smaller than the gradient threshold of each neural network layer are clipped, and the clipped gradients of each neural network layer in the local AI model are sent as gradient data to the central node.

8. A training device for an artificial intelligence model based on federated learning, characterized in that, A central node is used in a federated learning-based artificial intelligence model training system, the federated learning-based artificial intelligence model training system including a central node and several computing nodes communicatively connected to the central node; the federated learning-based artificial intelligence model training device includes: The acquisition module is used to acquire the initial artificial intelligence model and send it to each computing node; The first iteration control module is used to iteratively perform update steps. When the preset update iteration threshold or the current artificial intelligence model meets the preset conditions, the current artificial intelligence model is sent as the trained artificial intelligence model to each computing node. The update step includes: After receiving gradient data sent by a preset number of computing nodes, the system enters the global parameter update process. The gradient data consists of the gradients of each neural network layer of the local artificial intelligence model, which are calculated by the computing nodes based on the local training data. Global parameter update process: Based on the gradient data sent by the previous preset number of computing nodes, update the global parameters of the artificial intelligence model, and send the updated global parameters to the previous preset number of computing nodes; wherein, the updated global parameters are used to update the model parameters of the local artificial intelligence model of the computing nodes. The update step also includes: Feedback information is sent to the computing nodes that have completed gradient calculations during the global parameter update process; the feedback information is used to trigger the computing nodes to update the model parameters of the local artificial intelligence model based on the local gradient data. Among them, the preset number of computing nodes are calculated with the CPU computing clock of the central node as the computing interval. If more than the preset number of computing nodes complete the gradient calculation simultaneously within the CPU computing clock of a central node, then these computing nodes are all considered as the first preset number of computing nodes.

9. The artificial intelligence model training device based on federated learning according to claim 8, characterized in that, The step of updating the global parameters of the artificial intelligence model based on the gradient data sent by the pre-set number of computing nodes includes: The average gradient data is obtained by averaging the gradient data sent by a preset number of computing nodes. Based on the average gradient data, the backpropagation algorithm is used to update the global parameters of the artificial intelligence model.

10. A training device for an artificial intelligence model based on federated learning, characterized in that, A computing node applied to a federated learning-based artificial intelligence model training system, the federated learning-based artificial intelligence model training system including a central node and several computing nodes communicatively connected to the central node; the federated learning-based artificial intelligence model training device includes: The receiving module is used to receive the initial artificial intelligence model sent by the central node and use it as the local artificial intelligence model. The second iteration control module is used to iterate the gradient calculation steps until a preset gradient calculation iteration threshold is reached, and to receive the trained artificial intelligence model sent by the central node, or to iterate the gradient calculation steps until the trained artificial intelligence model is received from the central node, and to update the local artificial intelligence model using the trained artificial intelligence model. The gradient calculation step includes: Based on local training data, the gradients of each neural network layer of the local artificial intelligence model are obtained through gradient calculation. When the current node is one of the first preset number of calculation nodes that have completed gradient calculation, the gradients of each neural network layer of the local artificial intelligence model are sent to the central node as gradient data. When the updated global parameters are received from the central node, the model parameters of the local artificial intelligence model are updated according to the updated global parameters. The gradient calculation step further includes: when receiving feedback information sent by the central node, responding to the feedback information and updating the model parameters of the local artificial intelligence model based on the local gradient data; Among them, the preset number of computing nodes are calculated with the CPU computing clock of the central node as the computing interval. If more than the preset number of computing nodes complete the gradient calculation simultaneously within the CPU computing clock of a central node, then these computing nodes are all considered as the first preset number of computing nodes.

11. The artificial intelligence model training device based on federated learning according to claim 10, characterized in that, The step of sending the gradients of each neural network layer of the local artificial intelligence model as gradient data to the central node includes: Based on the gradients of each neural network layer of the local artificial intelligence model, the information entropy of the gradients of each neural network layer of the local artificial intelligence model is obtained, and the gradient threshold of each neural network layer of the local artificial intelligence model is determined based on the information entropy of the gradients of each neural network layer of the local artificial intelligence model. Based on the gradient threshold of each neural network layer in the local AI model, the gradients of each neural network layer in the local AI model that are smaller than the gradient threshold of each neural network layer are clipped, and the clipped gradients of each neural network layer in the local AI model are sent as gradient data to the central node.

12. The artificial intelligence model training device based on federated learning according to claim 11, characterized in that, The step of obtaining the information entropy of the gradient of each neural network layer of the local artificial intelligence model based on the gradient of each neural network layer, and determining the gradient threshold of each neural network layer of the local artificial intelligence model based on the information entropy of the gradient of each neural network layer of the local artificial intelligence model includes: Obtain the maximum and minimum gradients among the gradients of each neural network layer in the local artificial intelligence model; Divide the interval between the maximum gradient and the minimum gradient into several consecutive sub-intervals; The gradients of each neural network layer in the local AI model are placed into sub-intervals according to their values. The number of gradients in each sub-interval is counted, and the proportion of the number of gradients in each sub-interval to the total number of gradients in all neural network layers of the local AI model is calculated to obtain the frequency of gradient occurrence in each sub-interval. ; The information entropy of the gradients of each neural network layer in the local artificial intelligence model is obtained by the following formula. : in, Number the layers of the neural network. For sub-interval numbering, This represents the total number of sub-intervals. The gradient thresholds for each neural network layer of the local AI model are determined using the following formula. : in, These are the preset hyperparameters.

13. A federated learning-based artificial intelligence model training system, characterized in that, It includes a central node and several computing nodes that are connected to the central node in communication. The central node is equipped with an artificial intelligence model training device based on federated learning as described in any one of claims 8 to 9; The computing node is equipped with an artificial intelligence model training device based on federated learning as described in any one of claims 10 to 12.

Images