Network model training method and device, electronic equipment and storage medium
Patent Information
- Application Number
- CN202410500530.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-24
- Publication Date
- 2026-11-03
- Estimated Expiration
- 2044-04-24
AI Technical Summary
In federated learning, the use of additional training sets may raise data privacy issues, and in real-world applications, it is difficult to obtain training sets with a distribution consistent with the test data, making it difficult to obtain weight shrinkage factors and aggregate weights.
By calculating the global gradient value and preset hyperparameters, the shrinkage factor of the client model parameters is determined, thereby shrinking the weights of the network model and avoiding the use of additional training sets.
This effectively avoids data privacy issues, improves the model's generalization ability, and reduces the need for additional training sets.
Smart Images
Figure CN118799698B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing, and in particular to network model training methods, apparatus, electronic devices and storage media. Background Technology
[0002] With the rapid development of computing and storage capabilities, machine learning, utilizing large-scale data to solve problems, has brought significant benefits to scientific research and real-world applications. Therefore, machine learning is increasingly being applied in various fields such as finance, healthcare, and business. To avoid user privacy issues during model training, Federated Learning (FL) has emerged.
[0003] In related technologies, a global weight shrinkage phenomenon was discovered during the weighted aggregation process of federated learning. Additional training sets were used to learn the weight shrinkage factor and aggregation weights separately. However, such additional training sets in the federated learning framework may raise data privacy issues, and in real-world applications, it is difficult to obtain training sets with a distribution consistent with the test data; acquiring additional training sets is quite challenging. Summary of the Invention
[0004] The main objective of this application is to provide a network model training method, apparatus, electronic device, and storage medium, aiming to avoid learning weight shrinkage factors and aggregate weights through additional training sets, thereby avoiding data privacy issues and the acquisition of additional training sets. The technical solution is as follows:
[0005] In a first aspect, embodiments of this application provide a network model training method, including:
[0006] Send the first global model parameters of the network model to at least two clients, trigger each client to train the network model based on the received first global model parameters and local data, and return the trained client model parameters to the server, wherein the local data is an image carrying a species category identifier;
[0007] The received client model parameters are aggregated, and a global gradient value is determined based on the aggregated second global model parameters and the first global model parameters.
[0008] Obtain the preset hyperparameters and global learning rate;
[0009] Based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate, a first shrinkage factor is determined, wherein the first shrinkage factor is the shrinkage factor corresponding to the client model parameters.
[0010] The network model is subjected to first weight shrinkage based on the first shrinkage factor and the second global model parameters to obtain a network model after first weight shrinkage. The network model is used to predict the species category of the content in the input image.
[0011] Secondly, embodiments of this application provide a network model training apparatus, comprising:
[0012] The sending unit sends the first global model parameters of the network model to at least two clients, triggering each client to train the network model based on the received first global model parameters and local data, and returns the trained client model parameters to the server, wherein the local data is an image carrying a species category identifier;
[0013] The first determining unit is used to aggregate the received multiple client model parameters and determine the global gradient value based on the aggregated second global model parameters and the first global model parameters.
[0014] The acquisition unit is used to acquire preset hyperparameters and the global learning rate;
[0015] The second determining unit is used to determine a first shrinkage factor based on the first global model parameters, the global gradient value, the preset hyperparameters and the global learning rate, wherein the first shrinkage factor is the shrinkage factor corresponding to the client model parameters;
[0016] The weight shrinking unit is used to perform a first weight shrinking on the network model based on the first shrinking factor and the second global model parameters to obtain a network model after the first weight shrinking. The network model is used to predict the species category of the content in the input image.
[0017] Thirdly, embodiments of this application provide an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described above.
[0018] Fourthly, embodiments of this application provide a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.
[0019] In this embodiment, by sending the first global model parameters of the network model to at least two clients, each client is triggered to train the network model based on the received first global model parameters and local data, and the trained client model parameters are returned to the server. The local data is an image carrying a species category identifier. The received client model parameters are aggregated, and a global gradient value is determined based on the aggregated second global model parameters and the first global model parameters. Preset hyperparameters and a global learning rate are obtained. A first shrinkage factor is determined based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate. The first shrinkage factor is the shrinkage factor corresponding to the client model parameters. A first weight shrinkage is performed on the network model based on the first shrinkage factor and the second global model parameters to obtain a first weight-shrinkable network model. This network model is used to predict the species category of the content in the input image. By calculating the first shrinkage factor, the first weight shrinkage of the network model is achieved, thereby avoiding the need to learn the weight shrinkage factor and aggregate weights through additional training sets, thus avoiding data privacy issues and the need to obtain additional training sets. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a schematic diagram of a network model training method provided in an embodiment of this application.
[0022] Figure 2 This is a flowchart illustrating a network model training method provided in an embodiment of this application.
[0023] Figure 3 This is a schematic diagram of the structure of a network model training device provided in an embodiment of this application.
[0024] Figure 4 This is a schematic diagram of the structure of a network model training device provided in an embodiment of this application. Detailed Implementation
[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.
[0026] This application provides a network model training method, apparatus, storage medium, and computer device. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of a network model training method provided in an embodiment of this application. The system may include at least two clients 1000, at least one electronic device 2000, at least one database 3000, and a network 4000. Clients 1000 can connect to electronic devices 2000 via network 4000. Electronic device 2000 is any device with computing hardware capable of supporting and executing services related to network model training. Network 4000 can be a wireless network or a wired network, such as a wireless local area network (WLAN), local area network (LAN), cellular network, 2G network, 3G network, 4G network, 5G network, etc. Furthermore, different clients 1000 can also connect to other terminals or to electronic devices 2000 using their own Bluetooth networks or hotspot networks. For example, one client 1000 connects to electronic device 2000 via Bluetooth, and another client 1000 connects to electronic device 2000 via a hotspot network. In addition, the system may include multiple databases 3000, which are coupled to different electronic devices 2000, and can store the network model parameters of the network model before and after training in the databases 3000.
[0027] It should be noted that, Figure 1 The system diagram of the network model training system shown is merely an example. The network model training system and scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of network model training systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.
[0028] In this embodiment, the description will be from the perspective of a network model training device, which can be integrated into a computer device that has a storage unit and a microprocessor and thus computing power.
[0029] Please see Figure 2 , Figure 2This document provides a flowchart illustrating a network model training method as described in an embodiment of this application. Figure 2 As shown, the method described in this application embodiment may include the following steps S101-S105.
[0030] S101, send the first global model parameters of the network model to at least two clients, trigger each client to train the network model based on the received first global model parameters and local data, and return the trained client model parameters to the server, wherein the local data is an image carrying a species category identifier;
[0031] S102, aggregate the received multiple client model parameters, and determine the global gradient value based on the aggregated second global model parameters and the first global model parameters;
[0032] S103, obtain the preset hyperparameters and global learning rate;
[0033] S104, based on the first global model parameters, the global gradient value, the preset hyperparameters and the global learning rate, determine the first shrinkage factor, where the first shrinkage factor is the shrinkage factor corresponding to the client model parameters;
[0034] S105, perform first weight shrinkage on the network model based on the first shrinkage factor and the second global model parameters to obtain the network model after first weight shrinkage.
[0035] In the weighted aggregation process of federated learning, a global weight shrinkage phenomenon has been observed. Additional training sets are used to learn the weight shrinkage factor and aggregation weights separately. However, such additional training sets in the federated learning framework may raise data privacy concerns, and in real-world applications, it is difficult to obtain training sets with a distribution consistent with the test data, making the acquisition of additional training sets challenging. Therefore, this application provides a method for determining the shrinkage factor in the weight shrinkage process through computation, thereby avoiding the need for additional training sets to learn the weight shrinkage factor and aggregation weights, thus avoiding data privacy issues and the need to acquire additional training sets.
[0036] The following will provide a detailed explanation of each step:
[0037] S101, send the first global model parameters of the network model to at least two clients, trigger each client to train the network model based on the received first global model parameters and local data, and return the trained client model parameters to the server, wherein the local data is an image carrying a species category identifier.
[0038] Federated Learning (FL) is a distributed training method designed to facilitate the training of high-quality models by coordinating multiple clients or devices while preserving the privacy of their local datasets. It provides a privacy protection mechanism that effectively utilizes the computing resources of client devices to train the model, and then the trained models are uniformly processed and merged on the server side to obtain the overall trained model. This avoids the transmission and sharing of private data during training, thus preventing the leakage of privacy information during data transmission. Federated Learning can fully utilize the large amounts of valuable dataset resources on client devices.
[0039] Specifically, in the application scenarios of the network model provided in this application embodiment, the network model is used for image classification, for example, to distinguish species categories of images input to the trained network model. For example, given an image of a kitten, the trained network model determines whether the species in the image is a cat or a dog. The network model can also be used for speech noise reduction, for example, to remove noise from speech input to the trained network model. The network model can also be applied to various other scenarios, which are not limited here.
[0040] In this federated learning framework, the electronic device sends the first global model parameters of the network model obtained in the previous training round to each of the two clients, triggering each client to train the network model based on the received first global model parameters and local data, and then returns the trained client model parameters to the server.
[0041] S102, aggregate the received multiple client model parameters, and determine the global gradient value based on the aggregated second global model parameters and the first global model parameters.
[0042] The aggregation process can be performed using FedAvg, FedAvgM, CCVR, FedDF, FeddBE, FedDisco, FedLAW, and L-DAWA, among other methods.
[0043] FedAvg is the standard algorithm for federated learning, and most federated learning algorithms follow its basic framework. It assumes there is one server and K clients, each client having a private dataset D. k FedAvg trains for T communication rounds until the model converges. The main process of each communication round t can be summarized as follows: the server aggregates the models uploaded by the client to obtain a new global model. Where λ k It is aggregate weight. These are the client-side model parameters obtained after training each client. FedAvg often uses a simple average weighting. Or weighted according to the size of the local dataset, i.e.
[0044] FedAvgM stabilizes the training process through server-side momentum updates. CCVR uses dummy representations to tune the global model's classifier. FedDF and FeddBE fine-tune the global model on the server using an additional dataset. FedDisco recalculates aggregation weights based on dataset size and the degree of difference between local and global class distributions. FedLAW re-examines the weighted aggregation process and discovers global weight shrinkage. It learns weight shrinkage factors and aggregation weights separately using an additional surrogate dataset. L-DAWA uses the cosine similarity between the local and global models as aggregation weights for model aggregation, while also considering the differences between different layers in the model.
[0045] Specifically, by aggregating multiple client-side model parameters using any of the aforementioned aggregation methods, the aggregated second global model parameters are obtained. global gradient value Based on the second global model parameters and the first global model parameters It's confirmed.
[0046] In some implementations, determining the global gradient value based on the aggregated second global model parameters and the first global model parameters includes:
[0047] The difference between the aggregated second global model parameters and the first global model parameters is calculated to obtain the global gradient value.
[0048] Among them, the global gradient value For the second global model parameters With the first global model parameters The difference.
[0049] S103, obtain the preset hyperparameters and global learning rate.
[0050] Among them, the preset hyperparameter τ and the global learning rate η g It was pre-set.
[0051] Specifically, in some implementations, the global learning rate η g =1.
[0052] S104, based on the first global model parameters, the global gradient value, the preset hyperparameters and the global learning rate, determine the first shrinkage factor, whereby the first shrinkage factor is the shrinkage factor corresponding to the client model parameters.
[0053] Among them, in determining the global gradient value Then, the first global model parameters can be combined. Preset hyperparameter τ and the global learning rate η g To determine the first contraction factor γ.
[0054] In some implementations, determining the first shrinkage factor based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate includes:
[0055] (1) Calculate the norm of the product of the global gradient value and the global learning rate to obtain the first calculation result;
[0056] (2) Calculate the product of the norm of the first global model parameter and the preset hyperparameter to obtain the second calculation result;
[0057] (3) Calculate the sum of the first calculation result and the second calculation result to obtain the third calculation result;
[0058] (4) Calculate the ratio of the second calculation result to the third calculation result to obtain the first contraction factor.
[0059] Among them, federated learning algorithm aggregation is achieved through Perform model aggregation, using Let k represent the pseudo-gradient of client k in the t-th communication round, then the updated local model It can be represented as
[0060] The global learning rate η is set. g =1. Therefore, the polymerization process after FedLAW introduces a shrinkage factor. It can be represented as follows:
[0061]
[0062] in For global gradient, This can be viewed as a global gradient update (optimization term). This can be viewed as a pseudo-gradient (regularization term) of global weight shrinkage. Therefore, the value of the shrinkage factor γ affects the regularization strength during model aggregation; a smaller shrinkage factor indicates smaller gradient updates and stronger regularization. When FedLAW learned the optimal shrinkage factor using additional data, it was found that the ratio of the norms of the optimization term and the regularization term in the above formula stabilized around a constant τ, expressed by the following formula:
[0063]
[0064] However, FedLAW used this to demonstrate that the learned shrinkage factor can maintain the stability between the regularization term and the optimization term and improve the model performance, but did not further utilize this phenomenon. Therefore, this phenomenon can be used to calculate the shrinkage factor.
[0065] Based on the above phenomena, the method FedLWS (Federate Learning with Layer-wise Weight Shrinking) is proposed. It improves the generalization ability of federated learning by shrinking the layer-wise weights. It can be seen that in the aggregation process, if the value of the constant τ in the above formula is known, the other parameters are easily obtained during training. Then the only unknown in the formula is the shrinkage factor γ. Therefore, by taking the constant τ as a hyperparameter, it is easy to deduce the expression of the shrinkage factor γ from the above formula as follows:
[0066]
[0067] The value of the shrinkage factor γ is calculated directly using the above expression. The numerator is the second calculation result obtained by multiplying the norm of the first global model parameters by the preset hyperparameters; the denominator includes the sum of the first and second calculation results obtained by calculating the norm of the product of the global gradient value and the global learning rate, which is the third calculation result; the shrinkage factor γ is the ratio of the second and third calculation results.
[0068] S105, the network model is subjected to first weight shrinkage based on the first shrinkage factor and the second global model parameters to obtain the network model after first weight shrinkage. The network model is used to predict the species category of the content in the input image.
[0069] After calculating the first contraction factor γ, the aggregated second global model parameters can be used as a reference. The network model undergoes a first weight reduction process to obtain the reduced-weight network model. This reduced-weight network model is used in species category prediction applications to predict the species category of the content in an input image.
[0070] In some implementations, the step of performing a first weight shrinkage on the network model based on the first shrinkage factor and the second global model parameters to obtain a network model with the first weight shrinkage includes:
[0071] (1) Calculate the product of the first contraction factor and the second global model parameter to obtain the third global model parameter;
[0072] (2) Update the model parameters in the network model to the third global model parameters to obtain the network model after the first weight shrinkage.
[0073] The third network model parameter of the subsequent network model is the product of the first shrinkage factor and the second global model parameter.
[0074] Specifically, upon obtaining the network model after the first weight shrinkage, it is necessary to determine whether the network model after the first weight shrinkage has been trained. If it is determined that the network model after the first weight shrinkage has been trained, then the training of the network model is completed, and a trained network model is obtained. If it is determined that the training of the network model after the first weight shrinkage has not been completed, then the network model parameters corresponding to the network model after the first weight shrinkage are determined as the first global model parameters, and the process returns to step 101, which involves sending the first global model parameters of the network model to at least two clients, triggering each client to train the network model based on the received first global model parameters and local data, and returning the trained client model parameters to the server. The local data is an image carrying a species category identifier. This process is then used to further optimize the network model parameters.
[0075] Specifically, the method for determining whether the network model after the first weight shrinkage has completed training is as follows: Calculate the difference (i.e., the loss value) between the labels on the data in the validation set and the output data of the network model after the first weight shrinkage. Compare the current loss value with the loss value obtained in the previous training round. If the loss value remains unchanged for multiple consecutive rounds, or is less than the loss value of multiple consecutive historical rounds, it indicates that the trained model is stabilizing, i.e., converged. Alternatively, set a preset loss value and compare the loss value of the network model after the first weight shrinkage with the preset loss value. If the loss value exceeds the preset loss value, it is determined that the network model after the first weight shrinkage has not completed training; if the loss value does not exceed the preset loss value, it is determined that the network model after the first weight shrinkage has completed training.
[0076] Taking species category prediction as an example, the loss value can be calculated as follows: set the category label for cat to 1 and the category label for dog to 0. When the model outputs the maximum probability of cat, which is 0.8, calculate the difference between the category label for cat (1) and the predicted probability (0.8) to obtain the loss value.
[0077] Another way to determine whether the network model after the first weight shrinkage has been trained is as follows: set a preset training round, and when the network model after the first weight shrinkage is obtained, determine whether the current training round has reached the preset training round. If the preset training round has not been reached, it is determined that the training of the network model after the first weight shrinkage has not been completed; if the preset training round has been reached, it is determined that the training of the network model after the first weight shrinkage has been completed.
[0078] The preset number of training epochs can be determined based on the model's performance. For example, if the experimental data shows that the trained model performs best when the number of training epochs is 200, then 200 can be used as the preset number of training epochs.
[0079] This embodiment of the application sends first global model parameters of a network model to at least two clients, triggering each client to train the network model based on the received first global model parameters and local data, and returns the trained client model parameters to the server. The local data consists of images carrying species category identifiers. The received client model parameters are aggregated, and a global gradient value is determined based on the aggregated second global model parameters and the first global model parameters. Preset hyperparameters and a global learning rate are obtained. A first shrinkage factor is determined based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate. The first shrinkage factor is the shrinkage factor corresponding to the client model parameters. A first weight shrinkage is performed on the network model based on the first shrinkage factor and the second global model parameters to obtain a first weight-shrinkable network model. This network model is used to predict the species category of the content in the input image. By calculating the first shrinkage factor, the first weight shrinkage of the network model is achieved, thereby avoiding the need to learn the weight shrinkage factor and aggregate weights through additional training sets, thus avoiding data privacy issues and the need to obtain additional training sets.
[0080] In some embodiments, after performing a first weight shrinkage on the network model based on the first shrinkage factor and the second global model parameters to obtain a network model with the first weight shrinkage, the method further includes:
[0081] (1) Obtain the first network layer model parameters corresponding to each network layer in the network model from the first global network model parameters;
[0082] (2) Obtain the second network layer model parameters corresponding to each network layer in the network model from the second global model parameters;
[0083] (3) Based on the first network layer model parameters, the second network layer model parameters, the preset hyperparameters and the global learning rate, determine the second shrinkage factor corresponding to each network layer;
[0084] (4) Based on the second network layer model parameters and the corresponding second shrinkage factor of each network layer, perform second weight shrinkage on the weight-shrinked network model to obtain the second weight-shrinked network model.
[0085] FedLAW directly shrinks the weights of the entire model. However, due to the differences between different layers in a deep neural network, simply applying a single shrinkage factor to the entire model may not effectively improve the generalization performance of the global model. Therefore, assigning different shrinkage factors γ to each layer of the model in a hierarchical manner, instead of using a single γ for the entire model, can further improve the generalization performance of the model. To address this issue, embodiments of this application further calculate a second shrinkage factor for each network layer in the network model.
[0086] Specifically, using γ l The shrinkage factor of the first layer in the network model is represented by the following formula, which is used to calculate the shrinkage factor of the first layer:
[0087]
[0088] In some implementations, determining the second shrinkage factor for each network layer based on the model parameters of the first network layer corresponding to each network layer, the model parameters of the corresponding second network layer, the preset hyperparameters, and the global learning rate includes:
[0089] (1) Calculate the product of the norm of the second network layer model parameters corresponding to each network layer and the preset hyperparameter to obtain the fourth calculation result;
[0090] (2) Calculate the range of the difference between the first network layer model parameters and the corresponding second network layer model parameters for each network layer to obtain the fifth calculation result;
[0091] (3) Calculate the product of the fifth calculation result and the global learning rate to obtain the sixth calculation result;
[0092] (4) Calculate the sum of the sixth calculation result and the fourth calculation result to obtain the seventh calculation result;
[0093] (5) Calculate the ratio of the fourth calculation result to the seventh calculation result to obtain the second shrinkage factor corresponding to each network layer.
[0094] The fourth calculation result is... The fifth calculation result is The sixth calculation result is The seventh calculation result is The fourth calculation result is the numerator in the following formula, and the seventh calculation result is the denominator in the following formula, thus obtaining the network layer model parameter γ corresponding to network layer l. l .
[0095]
[0096] In some implementations, the step of performing a second weight shrinkage on the learned network model based on the network layer model parameters corresponding to each network layer and the corresponding second shrinkage factor to obtain a second weight-shrinked network model includes:
[0097] (1) Calculate the product of the network layer model parameters corresponding to each network layer and the corresponding second shrinkage factor to obtain the network layer model parameters after second weight shrinkage for each network layer;
[0098] (2) Synthesize the network layer model parameters corresponding to the second weight shrinkage of multiple network layers to obtain the fourth global model parameters;
[0099] (3) Update the model parameters in the network model to the fourth global model parameters to obtain the network model after the second weight shrinkage.
[0100] The learned network model is subjected to a second weight shrinkage using the following formula to obtain the network model with the second weight shrinkage.
[0101]
[0102] Specifically, the product of the second shrinkage factor corresponding to the network layer model parameters for each network layer is calculated to obtain the network layer model parameters after second weight shrinkage for each network layer. For example, the second shrinkage factor for network layer 1 is γ1, and the corresponding network layer model parameters are... Then the network model after shrinking the second weight corresponding to network layer 1 is: This process is repeated to calculate the second-weighted, shrunk network model for each network layer. The second-weighted, shrunk network models for multiple network layers are then synthesized to obtain the fourth global model parameters. The model parameters in the current network model are then updated to these fourth global model parameters to obtain the second-weighted, shrunk network model.
[0103] In this embodiment, by calculating the second shrinkage factor corresponding to each network layer, the network layer model parameters after second weight shrinkage are obtained based on the product of the second network layer model parameters and the corresponding second shrinkage factor during the second weight shrinkage process. These second-weight-shrinked network layer model parameters are then synthesized to obtain the fourth global model parameters, resulting in the second-weight-shrinked network model. This approach takes into account the differences between different layers of the model, thus improving model performance.
[0104] The following will be combined with the appendix Figure 3 This application provides a detailed description of the network model training apparatus provided in its embodiments. It should be noted that the appendix... Figure 3 The network model training device described herein is used to execute the instructions. Figure 2 The methods shown in the embodiments are for illustrative purposes only, illustrating the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this specification. Figure 2 The example shown.
[0105] Please see Figure 3 This illustration shows a schematic diagram of a network model training apparatus provided in an exemplary embodiment of this application. The network model training apparatus can be implemented as all or part of a device through software, hardware, or a combination of both. The apparatus 1 includes a sending unit 10, a first determining unit 20, an acquiring unit 30, a second determining unit 40, and a weight shrinking unit 50.
[0106] Sending unit 10 sends the first global model parameters of the network model to at least two clients, triggering each client to train the network model based on the received first global model parameters and local data, and returns the trained client model parameters to the server, wherein the local data is an image carrying a species category identifier;
[0107] The first determining unit 20 is used to aggregate the received multiple client model parameters and determine the global gradient value based on the aggregated second global model parameters and the first global model parameters.
[0108] Acquisition unit 30 is used to acquire preset hyperparameters and the global learning rate;
[0109] The second determining unit 40 is used to determine a first shrinkage factor based on the first global model parameters, the global gradient value, the preset hyperparameters and the global learning rate, wherein the first shrinkage factor is the shrinkage factor corresponding to the client model parameters;
[0110] The weight shrinking unit 50 is used to perform a first weight shrinking on the network model based on the first shrinking factor and the second global model parameters to obtain a network model after the first weight shrinking. The network model is used to predict the species category of the content in the input image.
[0111] Optionally, the second determining unit 40 is specifically used to calculate the norm of the product of the global gradient value and the global learning rate to obtain a first calculation result;
[0112] Calculate the product of the norm of the first global model parameter and the preset hyperparameter to obtain the second calculation result;
[0113] The sum of the first calculation result and the second calculation result is calculated to obtain the third calculation result;
[0114] The ratio of the second calculation result to the third calculation result is calculated to obtain the first contraction factor.
[0115] Optionally, the weighted shrinkage unit 50 is specifically used to calculate the product of the first shrinkage factor and the second global model parameter to obtain the third global model parameter;
[0116] The model parameters in the network model are updated to the third global model parameters to obtain the network model after the first weight shrinkage.
[0117] Optionally, the device is further configured to obtain the first network layer model parameters corresponding to each network layer in the network model from the first global network model parameters;
[0118] Obtain the second network layer model parameters corresponding to each network layer in the network model from the second global model parameters;
[0119] Based on the first network layer model parameters, the second network layer model parameters, the preset hyperparameters, and the global learning rate, a second shrinkage factor is determined for each network layer.
[0120] Based on the second network layer model parameters corresponding to each network layer and the corresponding second shrinkage factor, the weight-shrinked network model is subjected to second weight shrinkage to obtain the second weight-shrinked network model.
[0121] Optionally, the device is further configured to calculate the product of the norm of the second network layer model parameters corresponding to each network layer and the preset hyperparameters to obtain a fourth calculation result;
[0122] Calculate the norm of the difference between the first network layer model parameters and the corresponding second network layer model parameters for each network layer to obtain the fifth calculation result;
[0123] The product of the fifth calculation result and the global learning rate is calculated to obtain the sixth calculation result;
[0124] The sum of the sixth calculation result and the fourth calculation result is calculated to obtain the seventh calculation result;
[0125] The ratio of the fourth calculation result to the seventh calculation result is calculated to obtain the second shrinkage factor corresponding to each network layer.
[0126] Optionally, the first determining unit 20 is specifically used to calculate the product of the second network layer model parameters corresponding to each network layer and the corresponding second shrinkage factor to obtain the network layer model parameters after second weight shrinkage for each network layer.
[0127] By synthesizing the network layer model parameters corresponding to the second weight shrinkage of multiple network layers, a fourth global model parameter is obtained;
[0128] The model parameters in the network model are updated to the fourth global model parameters to obtain the network model after the second weight shrinkage.
[0129] In this embodiment, by sending the first global model parameters of the network model to at least two clients, each client is triggered to train the network model based on the received first global model parameters and local data, and the trained client model parameters are returned to the server. The local data is an image carrying a species category identifier. The received client model parameters are aggregated, and a global gradient value is determined based on the aggregated second global model parameters and the first global model parameters. Preset hyperparameters and a global learning rate are obtained. A first shrinkage factor is determined based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate. The first shrinkage factor is the shrinkage factor corresponding to the client model parameters. A first weight shrinkage is performed on the network model based on the first shrinkage factor and the second global model parameters to obtain a first weight-shrinkable network model. This network model is used to predict the species category of the content in the input image. By calculating the first shrinkage factor, the first weight shrinkage of the network model is achieved, thereby avoiding the need to learn the weight shrinkage factor and aggregate weights through additional training sets, thus avoiding data privacy issues and the need to obtain additional training sets.
[0130] It should be noted that the network model training device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the network model training method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the network model training device and the network model training method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.
[0131] The sequence numbers of the embodiments described above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0132] This application embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described functionality. Figure 2 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figure 2 The specific details of the illustrated embodiments will not be elaborated here.
[0133] Please refer to Figure 4 This diagram illustrates the structure of a network model training device provided in an exemplary embodiment of this specification. The network model training device in this specification may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected via the bus 150.
[0134] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the network model training device using various interfaces and lines, and executes various functions of terminal 100 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user page, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.
[0135] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described above, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems.
[0136] The memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, while native and third-party applications run in user space. To ensure that different third-party applications can achieve good running performance, the operating system allocates corresponding system resources for each application. However, different application scenarios within the same third-party application have different requirements for system resources. For example, in local resource loading scenarios, third-party applications have high requirements for disk read speed; in animation rendering scenarios, third-party applications have high requirements for GPU performance. Since the operating system and third-party applications are independent of each other, the operating system often cannot promptly perceive the current application scenario of a third-party application, resulting in the operating system's inability to adapt system resources accordingly.
[0137] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.
[0138] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 can be a touch display screen.
[0139] The touch display screen can be designed as a full-screen, curved screen, or irregularly shaped screen. It can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; however, this application does not limit the specific design in this regard.
[0140] In addition, those skilled in the art will understand that the structure of the network model training device shown in the above figures does not constitute a limitation on the network model training device. The network model training device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the network model training device may also include radio frequency circuits, input units, sensors, audio circuits, WiFi modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.
[0141] exist Figure 4 In the network model training device shown, the processor 110 can be used to call the network model training application stored in the memory 120 and specifically perform the following operations:
[0142] Send the first global model parameters of the network model to at least two clients, trigger each client to train the network model based on the received first global model parameters and local data, and return the trained client model parameters to the server, wherein the local data is an image carrying a species category identifier;
[0143] The received client model parameters are aggregated, and a global gradient value is determined based on the aggregated second global model parameters and the first global model parameters.
[0144] Obtain the preset hyperparameters and global learning rate;
[0145] Based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate, a first shrinkage factor is determined, wherein the first shrinkage factor is the shrinkage factor corresponding to the client model parameters.
[0146] The network model is subjected to first weight shrinkage based on the first shrinkage factor and the second global model parameters to obtain a network model after first weight shrinkage. The network model is used to predict the species category of the content in the input image.
[0147] In one embodiment, when the processor 110 performs the operation of determining the first shrinkage factor based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate, it specifically performs the following operations:
[0148] Calculate the norm of the product of the global gradient value and the global learning rate to obtain the first calculation result;
[0149] Calculate the product of the norm of the first global model parameter and the preset hyperparameter to obtain the second calculation result;
[0150] The sum of the first calculation result and the second calculation result is calculated to obtain the third calculation result;
[0151] The ratio of the second calculation result to the third calculation result is calculated to obtain the first contraction factor.
[0152] In one embodiment, when the processor 110 performs the first weight shrinkage on the network model based on the first shrinkage factor and the second global model parameters to obtain the first weight-shrinkable network model, it specifically performs the following operations:
[0153] Calculate the product of the first contraction factor and the second global model parameter to obtain the third global model parameter;
[0154] The model parameters in the network model are updated to the third global model parameters to obtain the network model after the first weight shrinkage.
[0155] In one embodiment, after the processor 110 performs a first weight shrinkage on the network model based on the first shrinkage factor and the second global model parameters to obtain a network model with the first weight shrinkage, it further performs the following operations:
[0156] Obtain the first network layer model parameters corresponding to each network layer in the network model from the first global network model parameters;
[0157] Obtain the second network layer model parameters corresponding to each network layer in the network model from the second global model parameters;
[0158] Based on the first network layer model parameters, the second network layer model parameters, the preset hyperparameters, and the global learning rate, a second shrinkage factor is determined for each network layer.
[0159] Based on the second network layer model parameters corresponding to each network layer and the corresponding second shrinkage factor, the weight-shrinked network model is subjected to second weight shrinkage to obtain the second weight-shrinked network model.
[0160] In one embodiment, when the processor 110 executes the process of determining the second shrinkage factor corresponding to each network layer based on the first network layer model parameters, the corresponding second network layer model parameters, the preset hyperparameters, and the global learning rate, it specifically performs the following operations:
[0161] The fourth calculation result is obtained by multiplying the norm of the second network layer model parameters corresponding to each network layer with the preset hyperparameters;
[0162] Calculate the norm of the difference between the first network layer model parameters and the corresponding second network layer model parameters for each network layer to obtain the fifth calculation result;
[0163] The product of the fifth calculation result and the global learning rate is calculated to obtain the sixth calculation result;
[0164] The sum of the sixth calculation result and the fourth calculation result is calculated to obtain the seventh calculation result;
[0165] The ratio of the fourth calculation result to the seventh calculation result is calculated to obtain the second shrinkage factor corresponding to each network layer.
[0166] In one embodiment, when the processor 110 performs second weight shrinking on the weight-shrinked network model based on the second network layer model parameters corresponding to each network layer and the corresponding second shrinking factor to obtain a second weight-shrinked network model, it specifically performs the following operations:
[0167] Calculate the product of the second network layer model parameters corresponding to each network layer and the corresponding second shrinkage factor to obtain the network layer model parameters after second weight shrinkage for each network layer.
[0168] By synthesizing the network layer model parameters corresponding to the second weight shrinkage of multiple network layers, a fourth global model parameter is obtained;
[0169] The model parameters in the network model are updated to the fourth global model parameters to obtain the network model after the second weight shrinkage.
[0170] In one embodiment, when the processor 110 determines the global gradient value based on the aggregated second global model parameters and the first global model parameters, it specifically performs the following operations:
[0171] The difference between the aggregated second global model parameters and the first global model parameters is calculated to obtain the global gradient value.
[0172] In this embodiment, by sending the first global model parameters of the network model to at least two clients, each client is triggered to train the network model based on the received first global model parameters and local data, and the trained client model parameters are returned to the server. The local data is an image carrying a species category identifier. The received client model parameters are aggregated, and a global gradient value is determined based on the aggregated second global model parameters and the first global model parameters. Preset hyperparameters and a global learning rate are obtained. A first shrinkage factor is determined based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate. The first shrinkage factor is the shrinkage factor corresponding to the client model parameters. A first weight shrinkage is performed on the network model based on the first shrinkage factor and the second global model parameters to obtain a network model with the first weight shrinkage. By calculating the first shrinkage factor, the first weight shrinkage of the network model is achieved, thereby avoiding the need to learn the weight shrinkage factor and aggregate weights through additional training sets, thus avoiding data privacy issues and the need to obtain additional training sets.
[0173] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0174] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.
Claims
1. A network model training method, characterized in that, include: Send the first global model parameters of the network model to at least two clients, trigger each client to train the network model based on the received first global model parameters and local data, and return the trained client model parameters to the server, wherein the local data is an image carrying a species category identifier; The received client model parameters are aggregated, and the global gradient value is determined based on the aggregated second global model parameters and the first global model parameters. Obtain the preset hyperparameters and global learning rate; Based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate, a first shrinkage factor is determined, wherein the first shrinkage factor is the shrinkage factor corresponding to the client model parameters. Calculate the product of the first contraction factor and the second global model parameter to obtain the third global model parameter; The model parameters in the network model are updated to the third global model parameters to obtain the network model after the first weight shrinkage. The network model is used to predict the species category of the content in the input image. Obtain the first network layer model parameters corresponding to each network layer in the network model from the first global network model parameters; Obtain the second network layer model parameters corresponding to each network layer in the network model from the second global model parameters; The fourth calculation result is obtained by multiplying the norm of the second network layer model parameters corresponding to each network layer with the preset hyperparameters; Calculate the norm of the difference between the first network layer model parameters and the corresponding second network layer model parameters for each network layer to obtain the fifth calculation result; The product of the fifth calculation result and the global learning rate is calculated to obtain the sixth calculation result; The sum of the sixth calculation result and the fourth calculation result is calculated to obtain the seventh calculation result; Calculate the ratio of the fourth calculation result to the seventh calculation result to obtain the second shrinkage factor corresponding to each network layer; Based on the second network layer model parameters corresponding to each network layer and the corresponding second shrinkage factor, the weight-shrinked network model is subjected to second weight shrinkage to obtain the second weight-shrinked network model.
2. The method as described in claim 1, characterized in that, The step of determining the first shrinkage factor based on the first global model parameters, the global gradient value, the preset hyperparameters, and the global learning rate includes: Calculate the norm of the product of the global gradient value and the global learning rate to obtain the first calculation result; Calculate the product of the norm of the first global model parameter and the preset hyperparameter to obtain the second calculation result; The sum of the first calculation result and the second calculation result is calculated to obtain the third calculation result; The ratio of the second calculation result to the third calculation result is calculated to obtain the first contraction factor.
3. The method as described in claim 1, characterized in that, The step of performing a second weight shrinkage on the weight-shrinked network model based on the second network layer model parameters corresponding to each of the network layers and the corresponding second shrinkage factor to obtain a second weight-shrinked network model includes: Calculate the product of the second network layer model parameters corresponding to each network layer and the corresponding second shrinkage factor to obtain the network layer model parameters after second weight shrinkage for each network layer. By synthesizing the network layer model parameters corresponding to the second weight shrinkage of multiple network layers, a fourth global model parameter is obtained; The model parameters in the network model are updated to the fourth global model parameters to obtain the network model after the second weight shrinkage.
4. The method as described in claim 1, characterized in that, The determination of the global gradient value based on the aggregated second global model parameters and the first global model parameters includes: The difference between the aggregated second global model parameters and the first global model parameters is calculated to obtain the global gradient value.
5. A network model training device, characterized in that, The device includes: The sending unit sends the first global model parameters of the network model to at least two clients, triggering each client to train the network model based on the received first global model parameters and local data, and returns the trained client model parameters to the server. The local data is an image carrying a species category identifier. The first determining unit is used to aggregate the received multiple client model parameters and determine the global gradient value based on the aggregated second global model parameters and the first global model parameters. The acquisition unit is used to acquire preset hyperparameters and the global learning rate; The second determining unit is used to determine a first shrinkage factor based on the first global model parameters, the global gradient value, the preset hyperparameters and the global learning rate, wherein the first shrinkage factor is the shrinkage factor corresponding to the client model parameters; A weight shrinking unit is used to perform a first weight shrinking on the network model based on the first shrinking factor and the second global model parameters to obtain a network model after the first weight shrinking. The network model is used to predict the species category of the content in the input image. The apparatus is used to perform the steps of the method as described in any one of claims 1-4.
6. An electronic device, characterized in that, include: Processor and memory; The memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as described in any one of claims 1 to 4.
7. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.