Model training method and device, storage medium and computer program product
By using dimensionality reduction seeds in federated learning to reduce the dimensionality reduction process of model update information, the problem of low communication efficiency in federated learning is solved, and more efficient data transmission and model training is achieved.
Patent Information
- Application Number
- CN202411417368.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2025-01-07
AI Technical Summary
In federated learning, due to the high frequency transmission of model update information, the communication efficiency is low and the data transmission volume is large, which affects the training efficiency.
By receiving the dimensionality reduction seeds sent by the second device, the model update information of the first device is processed to reduce the dimensionality reduction model update information, and transmit it between the first device and the second device to reduce the transmission Data volume and communication bandwidth.
The transmission communication bandwidth is reduced, the load pressure of the second device is reduced, the overall communication efficiency and model training efficiency are improved, and training interruptions or performance degradation due to transmission delay or bandwidth limitations are avoided.
Smart Images
Figure CN119272901A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, device, storage medium and computer program product. Background Art
[0002] With the strengthening of data privacy protection regulations and the diversification of data generation locations, federated learning has received widespread attention as a distributed machine learning method. Under the federated learning framework, multiple participants (such as clients) can collaborate to train a shared machine learning model without directly exchanging data, which helps protect personal privacy and data security. However, this method requires high-frequency transmission of model update information (such as gradient information) between participants (such as clients) and coordinators (such as servers), resulting in large amounts of data transmission and low communication efficiency.
[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0004] The main purpose of this application is to provide a model training method, device, storage medium and computer program product to solve the technical problem of low communication efficiency caused by large data transmission volume in existing federated learning.
[0005] To achieve the above object, the present application proposes a model training method, which is applied to any first device among multiple first devices, and the model training method includes:
[0006] receiving a dimension reduction seed sent by a second device, where the dimension reduction seed is used to indicate a projection direction;
[0007] Training model parameters of a local model based on the sample data held by the first device to obtain first model update information;
[0008] Performing dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed to obtain the first model update information after dimensionality reduction, wherein the transmission communication bandwidth required for the first model update information after dimensionality reduction is less than that for the first model update information;
[0009] Sending the first model update information after the dimension reduction to the second device;
[0010] receiving second model update information sent by the second device, where the second model update information is obtained by aggregating the dimension-reduced first model update information sent by each of the first devices by the second device;
[0011] The model parameters are updated based on the second model update information.
[0012] Furthermore, the first model update information includes a local gradient vector of the model parameter; and the step of performing dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed to obtain the first model update information after dimensionality reduction includes:
[0013] Selecting a preset second number of dimensionality reduction seeds from the preset first number of dimensionality reduction seeds sent by the second device as target seeds, the preset first number being greater than the preset second number;
[0014] Calculating projection values of the local gradient vector in the projection directions indicated by the target seeds based on the target seeds respectively, to obtain local gradient projection values corresponding to the target seeds;
[0015] Generate first model update information after dimensionality reduction including local gradient projection values corresponding to each of the target seeds.
[0016] Further, the step of calculating the projection values of the local gradient vector in the projection directions indicated by the target seeds respectively based on the target seeds to obtain the local gradient projection values corresponding to the target seeds comprises:
[0017] Generate a random vector corresponding to each of the target seeds;
[0018] The projection values of the local gradient vectors in the projection directions indicated by the target seeds are calculated based on the random vectors corresponding to the target seeds to obtain the local gradient projection values corresponding to the target seeds.
[0019] Further, the step of calculating the projection values of the local gradient vectors in the projection directions indicated by the target seeds based on the random vectors corresponding to the target seeds to obtain the local gradient projection values corresponding to the target seeds comprises:
[0020] Normalizing the random vector generated corresponding to the target seed to a unit vector;
[0021] An inner product of the local gradient vector and the unit vector is calculated to calculate a projection value of the local gradient vector in the projection direction indicated by the target seed, so as to obtain a local gradient projection value corresponding to the target seed.
[0022] Further, the step of calculating the projection values of the local gradient vectors in the projection directions indicated by the target seeds based on the random vectors corresponding to the target seeds to obtain the local gradient projection values corresponding to the target seeds comprises:
[0023] Normalizing the random vector generated corresponding to the target seed to a unit vector;
[0024] An inner product of the local gradient vector and the unit vector is calculated to calculate a projection value of the local gradient vector in the projection direction indicated by the target seed, so as to obtain a local gradient projection value corresponding to the target seed.
[0025] Further, the second model update information includes the aggregated gradient value corresponding to each of the target seeds; the aggregated gradient value corresponding to the target seed is obtained by aggregating the local gradient projection values corresponding to the corresponding target seeds in the first model update information after dimensionality reduction sent by each of the first devices by the second device;
[0026] The step of updating the model parameters based on the second model update information comprises:
[0027] The model parameters are updated based on the aggregated gradient projection values corresponding to the target seeds and the random vectors corresponding to the target seeds.
[0028] In addition, to achieve the above purpose, the present application also proposes a model training method, which is applied to the second device, and the method includes:
[0029] Sending dimension reduction seeds to each first device;
[0030] Receiving first model update information after dimensionality reduction sent by each of the first devices, wherein the first model update information after dimensionality reduction is obtained by the first device training the model parameters of the local model based on the sample data held by the first device, and then performing dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed, wherein the transmission communication bandwidth required for the first model update information after dimensionality reduction is less than that for the first model update information;
[0031] Aggregating the first model update information after dimension reduction sent by each of the first devices to obtain second model update information;
[0032] The second model update information is sent to each of the first devices, so that each of the first devices updates the model parameters of the corresponding first device based on the second model update information.
[0033] Further, the first model update information includes a local gradient vector of the model parameter, the dimension reduction seeds sent by the second device include a preset first number, and the first model update information after dimension reduction includes a local gradient projection value corresponding to each target seed, and the local gradient projection value corresponding to each target seed is obtained by the first device selecting a preset second number of dimension reduction seeds from the preset first number of dimension reduction seeds sent by the second device as the target seeds, and calculating the projection value of the local gradient vector in the projection direction indicated by each target seed;
[0034] The step of aggregating the first model update information after dimension reduction sent by each of the first devices to obtain the second model update information comprises:
[0035] Aggregating all local gradient projection values associated with the same target seed to obtain aggregated gradient projection values corresponding to each target seed;
[0036] Second model update information including aggregated gradient projection values corresponding to each of the target seeds is generated.
[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a model training device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the model training method described above.
[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the model training method described above are implemented.
[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the model training method described above.
[0040] One or more technical solutions proposed in this application have at least the following technical effects:
[0041] After each first device receives the dimension reduction seed sent by the second device and each first device obtains the first model update information by training the model locally, the first model update information is subjected to dimension reduction processing based on the projection direction indicated by the dimension reduction seed to obtain the first model update information after dimension reduction. By reducing the dimension of the first model update information, the data amount of the first model update information can be reduced, so that the transmission communication bandwidth required for the first model update information after dimension reduction can be less than that of the original first model update information. Then, the first model update information after dimension reduction is sent to the second device by the first device. Since the transmission communication bandwidth cost of the first model update information after dimension reduction is lower than that of the original first model update information, the load pressure of the second device when processing a large amount of data can be reduced, and the system resource consumption of the second device can be reduced. Then, the second model update information sent by the second device is received by each first device. The second model update information is obtained by aggregating the first model update information after dimension reduction sent by each first device by the second device. Since the second model update information is generated on the basis of the first model update information after dimension reduction, the dimension of the second model update information received by the second device is also reduced accordingly, which can achieve the purpose of reducing the amount of data received by the second device. Therefore, in this application, by reducing the dimensionality of the original model update information, it is not necessary to transmit the complete model update information between the first device and the second device, but the model update information after dimensionality reduction is transmitted between the first device and the second device, which can reduce the amount of data transmission, thereby improving the overall communication efficiency. In addition, by reducing the amount of transmitted data, it is also possible to reduce training interruptions or performance degradation caused by transmission delays or bandwidth limitations, which can improve model training efficiency and model stability. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0044] Figure 1 This is an exemplary schematic diagram of an application scenario involved in the model training method of this application;
[0045] Figure 2 A flowchart of the first embodiment of the model training method of the present application is provided;
[0046] Figure 3A flowchart of the second embodiment of the model training method of the present application is provided;
[0047] Figure 4 A flowchart of the third embodiment of the model training method of the present application is provided;
[0048] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the model training method in the embodiment of the present application.
[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0051] With the development of artificial intelligence technology, deep neural networks have gradually been applied to risk assessment, speech recognition, natural language processing and other fields. However, the network structure in different application scenarios is relatively fixed. Therefore, in order to achieve better model performance, it is necessary to train the model with more training data. In order to solve this problem, federated learning came into being. In a federated learning system, multiple participants use their own sample data to independently train the global model, and send the trained model gradient information to the coordinator for gradient aggregation, thereby collaboratively training the global model. However, with the gradual increase in task complexity and performance requirements, and the gradual deepening of the number of network layers of the model, the model parameters are increasing, and because there are many training members participating in federated learning, the data received by the coordinator increases exponentially, which easily causes communication congestion and seriously affects the overall training efficiency.
[0052] To solve the above problems, the present application provides a solution, in which the first device performs dimensionality reduction processing on the original model update information in the projection direction indicated by the dimensionality reduction seed given by the second device to obtain the reduced model update information, and transmits the reduced model update information between the first device and the second device for model training, so that the first device and the second device do not need to transmit the complete model update information during the training process, which can reduce the amount of data transmission during the training process and improve communication efficiency.
[0053] See also Figure 1 , Figure 1 This is an exemplary schematic diagram of an application scenario involved in the model training method of this application. Figure 1As shown, the application scenario may include: a first device 1 and a second device 2 participating in federated learning. Among them, the first device 1 as a participant in federated learning may include multiple (only two are shown in the figure), and the second device 2 serves as a coordinator of federated learning. Each first device 1 is respectively connected to the second device 2 through a network 3. The first device 1 and the second device 2 can both be a computing service device with data processing, network communication and program running functions, such as a server, a tablet computer, a personal computer, a mobile phone, etc. The second device 2 can generate a dimension reduction seed locally and send the dimension reduction seed to each first device 1. Each first device 1 can train its own local model to obtain the model update information of each first device (also referred to as "first model update information", hereinafter collectively referred to as "first model update information"), and then perform dimension reduction processing on the first model update information based on the projection direction indicated by the dimension reduction seed to obtain the first model update information after dimension reduction, so as to compress the original first model update information. After that, each first device 1 sends the first model update information after dimension reduction generated by each first device 1 to the second device 2, and the second device 2 aggregates the first model update information after dimension reduction sent by all first devices 1 to obtain the second model update information, and sends the second model update information to each first device 1. Each first device 1 updates the parameters of the local model based on the received second model update information, and performs the next round of iterative training based on the updated local model until the preset training iteration stop condition is reached. During the entire training process, the model update information after dimension reduction is transmitted between the first device and the second device, so that the first device and the second device do not need to transmit the complete model parameters, which can reduce the amount of data transmission and improve communication efficiency.
[0054] In some embodiments, the sample data may be data related to a specified prediction task.
[0055] In an example scenario, if the specified prediction task is a medical image diagnosis task, the sample data may be a medical scan image of a specified body part in the medical field. In one embodiment, the sample data is data annotated with a data label, and the data label is used to indicate a reference label corresponding to the sample data and the prediction result, for example, the label may be sick or not sick. Specifically, when diagnosing medical images in the medical field, the second device serves as a designated server, in which a small amount of medical images are stored as sample data, the first device may be a terminal of each hospital, and the terminal of each hospital stores medical scan images collected within the hospital, and each medical scan image is annotated with the diagnosis result (for example, sick or not sick) made by the doctor after distinguishing the medical scan image. The second device generates a dimension reduction seed locally and sends the dimension reduction seed to each first device, so that the terminal of each hospital can train the local model based on the medical scan image sample data held locally to obtain the first model update information, and then perform dimension reduction processing on the original first model update information based on the projection direction indicated by the dimension reduction seed to compress the model update information. At the same time, after the second device receives the dimension-reduced first model update information sent by each first device, it aggregates the dimension-reduced first model update information sent by each first device to obtain the second model update information. Afterwards, the second device sends the second model update information to each first device, so that each first device updates the model parameters based on the second model update information.
[0056] In another example scenario, if the specified prediction task is a financial risk prediction task, the sample data may be user behavior data collected by a bank or other financial institution. In one embodiment, the sample data is data annotated with a data label, and the data label is used to indicate a reference label corresponding to the sample data and the prediction result, for example, the label may be good credit or poor credit. Specifically, the first device may be a terminal of each bank or other financial institution, and the terminal of each financial institution stores user behavior data collected internally by each financial institution, and the user behavior data is annotated with label information of good credit or poor credit. The second device may store a small amount of user behavior sample data. When the model starts training, the second device distributes dimensionality reduction seeds to each first device, and the first device may train the local model based on the user behavior sample data held locally to obtain the first model update information, and then perform dimensionality reduction processing on the first model update information based on the projection direction indicated by the seed, so that the model update information transmitted during the model training process is effectively compressed, thereby achieving the purpose of reducing the amount of data transmission during the model training process.
[0057] A model training method provided in an embodiment of the present application is described in detail below with reference to the accompanying drawings.
[0058] Reference Figure 2 , Figure 2 This is a flow chart of the first embodiment of the model training method of the present application. In this embodiment, the model training method is applied to any one of a plurality of first devices, and the first device is connected to a second device for communication. In this embodiment, the model training method includes steps S10 to S60:
[0059] Step S10: The first device receives a dimensionality reduction seed sent by the second device, where the dimensionality reduction seed is used to indicate a projection direction.
[0060] Specifically, when the model starts training, the first device can receive a preset first number of dimensionality reduction seeds, for example, S dimensionality reduction seeds, sent by the second device, where S is a positive integer. Each dimensionality reduction seed is randomly generated locally by the second device, and the dimensionality reduction seed can be understood as a random value. Each dimensionality reduction seed can be used to indicate a projection direction.
[0061] Step S20: The first device trains model parameters of a local model based on sample data held by the first device to obtain first model update information.
[0062] Specifically, in the first training round, the first device receives a global model broadcast from the second device, and the global model includes model parameters. After receiving the global model, the first device uses the received global model as the local model of the first device. Then, in each subsequent training round, the first device trains or updates the model parameters in the local model based on the sample data it holds, and obtains the first model update information. The first model update information may include updated model parameter information, or the gradient value of the model parameter of the first device. Since the model parameters are generally multi-dimensional data, the calculated gradient value is also multi-dimensional data and can be represented in the form of a vector. For ease of explanation, the gradient value of the model parameter calculated by the first device during the model training process is referred to as a local gradient vector in this article, and the dimension of the local gradient vector is the same as the dimension of the model parameter.
[0063] Step S30: the first device performs dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed to obtain the first model update information after dimensionality reduction, wherein the transmission communication bandwidth required for the first model update information after dimensionality reduction is less than that for the first model update information.
[0064] In this embodiment, the original first model update information is processed by dimensionality reduction in the projection direction indicated by the dimensionality reduction seed (essentially, the data set of the first model update information is projected), so that the original high-dimensional first model update information can be mapped to a low-dimensional space, thereby achieving the purpose of reducing the data volume of the first model update information, thereby making the transmission communication bandwidth required for the first model update information after dimensionality reduction less than that of the first model update information.
[0065] In a feasible implementation, the first model update information may include model parameters / local gradient vectors of model parameters. The above step S30 may include: after receiving the various dimensionality reduction seeds sent by the second device, the first device performs dimensionality reduction processing on the model parameters / local gradient vectors in the first model update information based on the projection directions indicated by the various dimensionality reduction seeds, to obtain the projection values of the model parameters / local gradient vectors in the projection directions indicated by the various dimensionality reduction seeds, and then aggregates the projection values of the model parameters / local gradient vectors in the projection directions indicated by the various dimensionality reduction seeds to obtain the first model update information after dimensionality reduction.
[0066] In another feasible implementation, the first model update information may include model parameters / local gradient vectors of model parameters, and the above-mentioned step S30 may include: the first device randomly selects a preset second number of dimensionality reduction seeds as target seeds from a preset first number of dimensionality reduction seeds sent by the second device, wherein the preset first number is greater than the preset second number; then, the model parameters / local gradient vectors in the first model update information are subjected to dimensionality reduction processing based on the projection directions indicated by each target seed, respectively, to obtain the projection values of the model parameters / local gradient vectors in the projection directions indicated by each target seed, and then the projection values of the model parameters / local gradient vectors in the projection directions indicated by each target seed are aggregated to obtain the first model update information after dimensionality reduction.
[0067] Step S40: The first device sends the first model update information after dimensionality reduction to the second device.
[0068] In this embodiment, after receiving the dimension reduction seed sent by the second device and training the local model to obtain the first model update information of the current training round, the first device projects the first model update information based on the projection direction indicated by the dimension reduction seed, thereby compressing the first model update information and sending the dimension-reduced first model update information to the second device. Since the first model update information transmitted to the second device is dimensionally reduced compared to the original first model update information, the data transmission volume can be reduced, and the load pressure of the second device can also be reduced, thereby preventing excessive data from affecting the processing speed of the second device.
[0069] Step S50: The first device receives second model update information sent by the second device, where the second model update information is obtained by aggregating the first model update information sent by each first device after dimension reduction by the second device.
[0070] In this embodiment, each first device sends its own first model update information after dimensionality reduction to the second device, so that the second device aggregates the first model update information after dimensionality reduction of all first devices to obtain second model update information. Since the second model update information is obtained by aggregating the first model update information after dimensionality reduction of each first device, the amount of data it has is less than the original model update information.
[0071] Step S60: The first device updates the model parameters based on the second model update information.
[0072] Specifically, after receiving the second model update information, the first device uses the second model update information to update the model parameters of the local model, and performs the next round of iterative training based on the updated local model until a preset training iteration stop condition is reached.
[0073] In the above-mentioned embodiment scheme, each first device receives the dimension reduction seed sent by the second device, and each first device obtains the first model update information of the model by training the model locally. Then, the first model update information is subjected to dimension reduction processing based on the projection direction indicated by the dimension reduction seed to obtain the first model update information after dimension reduction. By reducing the dimension of the first model update information, the data amount of the first model update information can be reduced, so that the transmission communication bandwidth required for the first model update information after dimension reduction can be less than that of the original first model update information; then, the first model update information after dimension reduction is sent to the second device by the first device. Since the transmission communication bandwidth cost of the first model update information after dimension reduction is lower than that of the original first model update information, the load pressure of the second device when processing a large amount of data can be reduced, and the system resource consumption of the second device can be reduced; then, the second model update information sent by the second device is received by each first device. The second model update information is obtained by aggregating the first model update information after dimension reduction sent by each first device by the second device. Since the second model update information is generated on the basis of the first model update information after dimension reduction, the dimension of the second model update information received by the second device is also reduced accordingly, which can achieve the purpose of reducing the amount of data received by the second device. Therefore, in this application, by reducing the dimensionality of the original model update information, it is not necessary to transmit the complete model update information between the first device and the second device, but the model update information after dimensionality reduction is transmitted between the first device and the second device, which can reduce the amount of data transmission, thereby improving the overall communication efficiency. In addition, by reducing the amount of transmitted data, it is also possible to reduce training interruptions or performance degradation caused by transmission delays or bandwidth limitations, which can improve model training efficiency and model stability.
[0074] Taking into account the need to balance privacy and communication bandwidth costs when transmitting data between the first device and the second device during model training, in some embodiments, the gradient of the model parameters can be selected as the model update information for transmission between the first device and the second device. Therefore, in some embodiments, the first model update information may include a local gradient vector of the model parameters of the first device. The following takes the example of the model update information including the local gradient vector of the model parameters to introduce the specific implementation method of the dimensionality reduction processing of the first model update information. Specifically, the above step S30 may include steps S301 to S303:
[0075] Step S301, selecting a preset second number of dimensionality reduction seeds from a preset first number of dimensionality reduction seeds sent by a second device as target seeds; wherein the preset first number is greater than the preset second number.
[0076] Specifically, after receiving the preset first number of seeds sent by the second device, the first device can randomly select a preset second number of seeds as target seeds to perform gradient projection processing of the local gradient vector. The preset first number is set by the second device according to actual conditions, and this application does not limit this. The preset second number can be determined by the first device based on system resources, required communication bandwidth, and model training accuracy.
[0077] The first device may select different dimensionality reduction seeds as target seeds in each training round, and different first devices may select different target seeds in the same training round.
[0078] Step S302 : calculating the projection values of the local gradient vectors in the projection directions indicated by the target seeds based on the target seeds respectively, and obtaining the local gradient projection values corresponding to the target seeds.
[0079] In some implementations, the above step S302 may include steps S3021 to S3022:
[0080] Step S3021, generating a random vector corresponding to each target seed.
[0081] Specifically, the first device can input each target seed into the agreed pseudo-random number generator respectively, and generate a random vector corresponding to each target seed respectively. Among them, the pseudo-random number generator is used to calculate a pseudo-random number (or random vector) through a seed value when the system needs a random number. The key to the pseudo-random number generator is the seed. The seed is the starting point of the random vector, and the same seed will generate the same random vector. Each first device can pre-agree on the pseudo-random number generator to be used.
[0082] Step S3022, calculating the projection values of the local gradient vectors in the projection directions indicated by the target seeds based on the random vectors corresponding to the target seeds, to obtain the local gradient projection values corresponding to the target seeds.
[0083] Specifically, after generating the random vectors corresponding to the target seeds, the first device may perform the following processing for each random vector corresponding to the target seed, including steps S30221 to S30222:
[0084] Step S30221, normalize the random vector generated corresponding to the target seed into a unit vector.
[0085] It should be noted that normalizing a random vector to a unit vector means converting the length (or dimension, modulus) of the random vector to 1 while keeping its direction unchanged. A unit vector is a vector with a modulus equal to 1 and a definite direction. The process of normalizing to a unit vector is actually scaling the vector so that its length (or dimension) becomes 1. The normalized unit vector only indicates the direction and does not contain length information, so no additional errors will be introduced in subsequent calculations due to differences in the length of the vector.
[0086] In a specific implementation, the first device normalizing the random vector to a unit vector may include: dividing the random vector by the modulus (or Euclidean norm) of the random vector. For example, assuming there is a random vector r, which is (x, y, z), its modulus ||r|| can be calculated by the following formula:
[0087] ||r||=sqrt(x^2+y^2+z^2);
[0088] Then, divide each component of vector r by ||r|| to get the unit vector r_norm:
[0089] r_norm=(x / |||r||y / ||r||,z / ||r||).
[0090] Step S30222, calculating the inner product of the local gradient vector and the unit vector to calculate the projection value of the local gradient vector in the projection direction indicated by the target seed, and obtaining the local gradient projection value corresponding to the target seed.
[0091] Specifically, after normalizing the random vector generated corresponding to the target seed to a unit vector, the first device can calculate the inner product of the local gradient and the unit vector of the random vector corresponding to the target seed, thereby obtaining the score of the local gradient vector on the corresponding random vector (or called the projection direction), and the obtained score is the projection value of the local gradient vector in the corresponding projection direction.
[0092] Step S303: Generate first model update information after dimensionality reduction including local gradient projection values corresponding to each target seed.
[0093] Specifically, each dimensionality reduction seed sent by the second device carries a seed identifier, and the first device can combine the local gradient projection values corresponding to the target seeds according to the seed identifiers corresponding to the target seeds to obtain the first model update information after dimensionality reduction. For example, assuming that the original local gradient vector is [0.11, 0.13, 0.5, 0.1, 0.01], the local gradient vector [0.11, 0.13, 0.5, 0.1, 0.01] is reduced in dimension by target seeds A, target seeds B, and target seeds C to obtain 0.11, 0.13, and 0.5, respectively, and assuming that the seed identifiers corresponding to target seeds A, target seeds B, and target seeds C are k1, k2, and k3, respectively, then the generated first model update information after dimensionality reduction can be [0.11 (k1) ,0.13 (k2) ,0.5 (k3) ].
[0094] In the above scheme, the first device selects multiple target seeds from a preset first number of dimensionality reduction seeds to perform projection processing on the local gradient vector, thereby obtaining local gradient projection values in multiple projection directions, and then can approximate the original local gradient information through the local gradient projection values in multiple projection directions, and can compress the gradient information on the basis of restoring the original gradient information as much as possible, thereby reducing the amount of data transmission during the model training process while ensuring the model training effect. In addition, in the above scheme, the first device only selects a preset second number of dimensionality reduction seeds to calculate the projection value, and there is no need to perform projection processing on the local gradient vector in the projection directions corresponding to all dimensionality reduction seeds, which can reduce the amount of calculation of the first device and is suitable for enterprises with limited computing resources.
[0095] In a feasible implementation, the above-mentioned aggregated gradient value includes the aggregated gradient value corresponding to each target seed; the aggregated gradient value corresponding to the target seed is obtained by aggregating the local gradient projection value corresponding to the corresponding target seed in the first model update information after dimensionality reduction sent by each first device by the second device; accordingly, the above-mentioned step S60, updating the local model based on the second model update information may include:
[0096] The parameters of the local model are updated based on the aggregated gradient values corresponding to each target seed and the random vectors corresponding to each target seed.
[0097] Specifically, after receiving the aggregated gradient values corresponding to each target seed, the first device can restore the gradient vector based on the aggregated gradient value corresponding to the target seed and the random vector corresponding to each target seed to obtain a unified gradient vector, and update the model parameters based on the unified gradient vector to update the local model, and return to the above step S20 for the next round of iterative training.
[0098] In some implementations, the first device may restore the gradient vector of the aggregated gradient value corresponding to each target seed and the random vector corresponding to each target seed based on the following formula to obtain a unified gradient vector delta W_G:
[0099]
[0100] Among them, k is the number of target seeds, s i is the aggregate gradient value corresponding to the i-th target seed sent by the second device, r i is the random vector corresponding to the i-th target seed.
[0101] Through the above implementation, the first device can update the parameters of the local model based on the aggregated gradient value and the corresponding random vector corresponding to each target seed. Since the aggregated gradient value brings together the gradient information from different first devices, each first device can learn the knowledge and capabilities of other first devices, thereby significantly improving the effect of model training.
[0102] See also Figure 3 , Figure 3 This is a flow chart of the second embodiment of the model training method of the present application. In this embodiment, the model training method is applied to a second device (such as a server), and the model training method includes steps A10 to A40:
[0103] Step A10: The second device sends the dimension reduction seed to each first device.
[0104] Specifically, the second device, as a coordinator, can generate a dimensionality reduction seed locally when the model starts training. The generated dimensionality reduction seed may include multiple dimensionality reduction seeds, and send multiple dimensionality reduction seeds to each first device. The generated dimensionality reduction seed is essentially a random value.
[0105] In step A20, the second device receives the first model update information after dimensionality reduction sent by each first device, wherein the first model update information after dimensionality reduction is obtained by the first device training a local model based on sample data held by the first device, and then performing dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed, wherein the transmission communication bandwidth required for the first model update information after dimensionality reduction is less than that for the first model update information.
[0106] Specifically, the second device sends a dimensionality reduction seed to each first device, so that after each first device trains the model parameters of the local model based on the locally held sample data to obtain the first model update information, it performs dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed to obtain the first model update information after dimensionality reduction, wherein the first model update information may include the local gradient vector of the model parameters or the updated model parameters.
[0107] Further, when the first model update information includes local gradient vectors of model parameters, the first model update information after dimensionality reduction received by the second device may include local gradient projection values corresponding to each target seed, and the local projection values corresponding to each target seed are obtained by the first device selecting a preset second number of dimensionality reduction seeds from a preset first number of dimensionality reduction seeds sent by the second device as target seeds, and calculating the projection values of the local gradient vectors in the projection directions indicated by each target seed.
[0108] Step A30: The second device aggregates the first model update information after dimension reduction sent by each first device to obtain second model update information.
[0109] Specifically, when the first model update information after dimensionality reduction includes the local gradient projection value corresponding to the target seed, the second device can aggregate the local gradient projection values corresponding to the corresponding target seeds sent by each first device to obtain an aggregated gradient value, and the obtained aggregated gradient value is the second model update information. The aggregation method can be direct aggregation, weighted aggregation or encrypted aggregation. Since the aggregation method is a prior art, it will not be repeated here.
[0110] Further, in the case where the first model update information after dimensionality reduction includes local gradient projection values corresponding to multiple target seeds, the second device can aggregate all local gradient projection values associated with the same target seed to obtain aggregated gradient projection values corresponding to each target seed. Then, the second model update information including the aggregated gradient projection values corresponding to each target seed is generated. Specifically, the generation principle of the second model update information is consistent with the first model update information after dimensionality reduction, and will not be repeated here.
[0111] In step A40, the second device sends the second model update information to each first device, so that each first device updates the model parameters of the corresponding first device based on the second model update information.
[0112] In this embodiment, the second device sends second model update information to each first device to share the model update information. When the second model update information includes the aggregated gradient projection value corresponding to each target seed, each first device updates the model parameters of the local model of the corresponding first device based on the aggregated gradient value corresponding to each target seed, and performs the next round of iterative training based on the updated local model.
[0113] In the above implementation scheme, the dimension reduction seed is sent to each first device by the second device, and each first device obtains the first model update information of the model by training the model locally. Then, the first model update information is subjected to dimension reduction processing based on the projection direction indicated by the dimension reduction seed to obtain the first model update information after dimension reduction. By reducing the dimension of the first model update information, the data amount of the first model update information can be reduced, so that the transmission communication bandwidth required for the first model update information after dimension reduction can be less than the original first model update information; then, the first model update information after dimension reduction is sent to the second device by the first device. Since the transmission communication bandwidth cost of the first model update information after dimension reduction is lower than that of the original first model update information, the load pressure of the second device when processing a large amount of data can be reduced, and the system resource consumption of the second device can be reduced; then, the second model update information sent by the second device is received by each first device. The second model update information is obtained by aggregating the first model update information after dimension reduction sent by each first device by the second device. Since the second model update information is generated on the basis of the first model update information after dimension reduction, the dimension of the second model update information received by the second device is also reduced accordingly, which can achieve the purpose of reducing the amount of data received by the second device. Therefore, in this application, by reducing the dimensionality of the original model update information, it is not necessary to transmit the complete model update information between the first device and the second device, but the model update information after dimensionality reduction is transmitted between the first device and the second device, which can reduce the amount of data transmission, thereby improving the overall communication efficiency. In addition, by reducing the amount of transmitted data, it is also possible to reduce training interruptions or performance degradation caused by transmission delays or bandwidth limitations, which can improve model training efficiency and model stability.
[0114] In order to help understand the above embodiments of the model training method of the present application, an example of the model training method combined with the above embodiments is proposed, see Figure 4 , Figure 4 It is a flowchart of the third embodiment of the model training method of the present application, including steps S1 to S9:
[0115] Step S1: The second device generates S dimensionality reduction seeds.
[0116] Wherein, each dimension reduction seed is used to indicate the projection direction, and S is an integer greater than 1.
[0117] Step S2: The second device sends the generated S dimensionality reduction seeds to each first device.
[0118] Step S3: The first device trains model parameters of the local model based on local sample data to obtain local gradient vectors of the model parameters.
[0119] In step S4, the first device selects K dimension reduction seeds from the S dimension reduction seeds as target seeds, and generates K random vectors corresponding to the K target seeds.
[0120] Specifically, the first device may generate a random vector using a predetermined pseudo-random number generator.
[0121] Step S5: The first device calculates the projection values of the local gradient vectors in the projection directions indicated by the respective seeds based on the random vectors corresponding to the respective target seeds, to obtain the local gradient projection values corresponding to the respective target seeds.
[0122] Step S6: The first device sends the local gradient projection value corresponding to each target seed to the second device.
[0123] Step S7: The second device receives the local gradient projection values corresponding to the target seeds sent by the first devices, and aggregates the local gradient projection values corresponding to the same target seeds to obtain aggregated gradient values corresponding to the target seeds.
[0124] Step S8: The second device sends the aggregated gradient value corresponding to each target seed to each first device.
[0125] Step S9: The first device receives the aggregated gradient values corresponding to the target seeds, and updates the model parameters based on the aggregated gradient values corresponding to the target seeds.
[0126] Specifically, after receiving the aggregated gradient values corresponding to each target seed, the first device performs gradient vector recovery based on the aggregated gradient values corresponding to each target seed and the random vectors corresponding to each target seed to obtain a unified gradient vector, updates the model parameters based on the unified gradient vector, and performs the next round of iterative training based on the updated model until the preset training iteration stop condition is reached.
[0127] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the model training method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0128] An embodiment of the present application provides a model training device, which may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model training method in the above-mentioned embodiment one.
[0129] Reference below Figure 5 , which shows a schematic diagram of the structure of a model training device suitable for implementing an embodiment of the present application. The model training device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The model training device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0130] like Figure 5 As shown, the model training device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the model training device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the model training device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a model training device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.
[0131] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0132] Compared with the prior art, the beneficial effects of the model training device provided in the embodiment of the present application are the same as the beneficial effects of the model training method provided in the above embodiment, and other technical features in the model training device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0133] It should be understood that the various parts disclosed in the embodiments of the present application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0134] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0135] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the model training method in the above-mentioned embodiment.
[0136] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0137] The above-mentioned computer-readable storage medium may be included in the model training device; or it may exist independently without being assembled into the model training device.
[0138] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the model training device, the model training device executes: receiving the dimensionality reduction seed sent by the second device, the dimensionality reduction seed is used to indicate the projection direction; performing dimensionality reduction processing on the first model update information based on the projection direction corresponding to the dimensionality reduction seed to obtain the first model update information after dimensionality reduction, wherein the transmission communication bandwidth required for the first model update information after dimensionality reduction is less than that of the first model update information; sending the first model update information after dimensionality reduction to the second device; receiving the second model update information sent by the second device, the second model update information is obtained by aggregating the first model update information after dimensionality reduction sent by each first device by the second device; and updating the model parameters based on the second model update information. Alternatively, the model training device executes: sending a dimensionality reduction seed to each first device; receiving first model update information after dimensionality reduction sent by each first device, wherein the first model update information after dimensionality reduction is obtained by the first device training a local model based on sample data held by the first device, and then reducing the dimensionality of the first model update information based on the projection direction indicated by the dimensionality reduction seed, wherein the transmission communication bandwidth required for the first model update information after dimensionality reduction is less than that for the first model update information; aggregating the first model update information after dimensionality reduction sent by each first device to obtain second model update information; and sending the second model update information to each first device, so that each first device updates the model parameters of the corresponding first device based on the second model update information.
[0139] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0141] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0142] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned model training method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the model training method provided in the above-mentioned embodiment, which will not be repeated here.
[0143] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned model training method when executed by a processor.
[0144] Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of the present application are the same as the beneficial effects of the model training method provided in the above embodiments, which will not be elaborated here.
[0145] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A model training method, characterized in that: Applied to any first device among a plurality of first devices, the method includes: receiving a dimension reduction seed sent by a second device, where the dimension reduction seed is used to indicate a projection direction; Training model parameters of a local model based on the sample data held by the first device to obtain first model update information; Performing dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed to obtain the first model update information after dimensionality reduction, wherein the transmission communication bandwidth required for the first model update information after dimensionality reduction is less than that for the first model update information; Sending the first model update information after the dimension reduction to the second device; receiving second model update information sent by the second device, where the second model update information is obtained by aggregating the dimension-reduced first model update information sent by each of the first devices by the second device; The model parameters are updated based on the second model update information.
2. The method according to claim 1, characterized in that The first model update information includes a local gradient vector of the model parameter; the step of performing dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed to obtain the first model update information after dimensionality reduction includes: Selecting a preset second number of reduced-dimensionality seeds from the preset first number of reduced-dimensionality seeds sent by the second device as target seeds, the preset first number being greater than the preset second number; Calculating projection values of the local gradient vector in the projection directions indicated by the target seeds based on the target seeds respectively, to obtain local gradient projection values corresponding to the target seeds; Generate first model update information after dimensionality reduction including local gradient projection values corresponding to each of the target seeds.
3. The method according to claim 2, characterized in that The step of calculating the projection values of the local gradient vectors in the projection directions indicated by the target seeds based on the target seeds to obtain the local gradient projection values corresponding to the target seeds comprises: Generate a random vector corresponding to each of the target seeds; The projection values of the local gradient vectors in the projection directions indicated by the target seeds are calculated based on the random vectors corresponding to the target seeds to obtain the local gradient projection values corresponding to the target seeds.
4. The method according to claim 3, characterized in that The step of calculating the projection values of the local gradient vectors in the projection directions indicated by the target seeds based on the random vectors corresponding to the target seeds to obtain the local gradient projection values corresponding to the target seeds comprises: Normalizing the random vector generated corresponding to the target seed to a unit vector; An inner product of the local gradient vector and the unit vector is calculated to calculate a projection value of the local gradient vector in the projection direction indicated by the target seed, so as to obtain a local gradient projection value corresponding to the target seed.
5. The method according to claim 2, characterized in that The second model update information includes the aggregated gradient value corresponding to each of the target seeds; the aggregated gradient value corresponding to the target seed is obtained by aggregating, by the second device, the local gradient projection values corresponding to the corresponding target seeds in the first model update information after dimensionality reduction sent by each of the first devices; The step of updating the model parameters based on the second model update information comprises: The model parameters are updated based on the aggregated gradient projection values corresponding to the target seeds and the random vectors corresponding to the target seeds.
6. A model training method, characterized in that: Applied to the second device, the method includes: Sending dimension reduction seeds to each first device; Receiving first model update information after dimensionality reduction sent by each of the first devices, wherein the first model update information after dimensionality reduction is obtained by the first device training the model parameters of the local model based on the sample data held by the first device, and then performing dimensionality reduction processing on the first model update information based on the projection direction indicated by the dimensionality reduction seed, wherein the transmission communication bandwidth required for the first model update information after dimensionality reduction is less than that for the first model update information; Aggregating the first model update information after dimension reduction sent by each of the first devices to obtain second model update information; The second model update information is sent to each of the first devices, so that each of the first devices updates the model parameters of the corresponding first device based on the second model update information.
7. The method according to claim 6, characterized in that The first model update information includes a local gradient vector of the model parameter, the dimension reduction seeds sent by the second device include a preset first number, the first model update information after dimension reduction includes a local gradient projection value corresponding to each target seed, and the local gradient projection value corresponding to each target seed is obtained by the first device selecting a preset second number of dimension reduction seeds from the preset first number of dimension reduction seeds sent by the second device as target seeds, and calculating the projection value of the local gradient vector in the projection direction indicated by each target seed, wherein the preset first number is less than the preset second number; The step of aggregating the first model update information after dimension reduction sent by each of the first devices to obtain the second model update information includes: Aggregating all local gradient projection values associated with the same target seed to obtain aggregated gradient projection values corresponding to each target seed; Second model update information including aggregated gradient projection values corresponding to each of the target seeds is generated.
8. A model training device, characterized in that: The model training device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the model training method as described in any one of claims 1 to 7.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the model training method according to any one of claims 1 to 7 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the steps of the model training method according to any one of claims 1 to 7.