Model training method and electronic device based on federated learning

Abstract

The embodiment of the present application is applicable to the field of artificial intelligence technology, and discloses a model training method and electronic device based on federated learning, the method comprising: a first electronic device sends a model to be trained to a second electronic device; the first electronic device receives one or more models to be fused sent by the second electronic device, the model to be fused is a model obtained by the second electronic device training the model to be trained according to a local training sample data set; the first electronic device uses a cross-validation data set to cross-validate the model to be fused, and obtains a cross-validation result; the first electronic device assigns fusion weights to one or more models to be fused according to the cross-validation result; the first electronic device performs model weighted fusion on the model to be fused according to the fusion weight of each model to be fused, and obtains an updated cloud model. The technical solution provided by the embodiment of the present application can improve the robustness of the cloud model obtained by cloud fusion.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method and electronic device based on federated learning. Background Art

[0002] As users’ awareness of data security and privacy continues to increase, and as data security issues are exposed by the frequent leakage of users’ personal privacy data, the protection of user data security and privacy is also constantly being strengthened, which in turn poses new challenges to the data processing mode and model training mode of traditional artificial intelligence (AI).

[0003] In order to meet the new challenges, a model training method of federated learning is proposed. In federated learning, the training data of the model is stored in each client device and does not need to be uploaded to the cloud server; each client device uses local data to train the model and uploads the obtained model parameters to the cloud server. Specifically, the cloud server first sends the model to be trained to each client device; after each client device receives the model, it uses the training data stored locally to train the model and obtain new model parameters; then, the client device encrypts the new model parameters and uploads them to the cloud server; finally, the cloud server performs model fusion update based on the model parameters uploaded by each client device to obtain a new model.

[0004] However, in existing federated learning, the new model obtained by integrating cloud servers has poor robustness. Summary of the invention

[0005] The present application provides a model training method and electronic device based on federated learning to solve the problem of poor robustness of the new model obtained by fusion of cloud servers in existing federated learning.

[0006] In a first aspect, an embodiment of the present application provides a model training method based on federated learning, the method comprising: a first electronic device sends a model to be trained to a second electronic device; then, the first electronic device receives one or more models to be fused sent by the second electronic device, the model to be fused being a model obtained by the second electronic device by training the model to be trained based on a local training sample data set; then, the first electronic device uses a cross-validation data set to cross-validate the model to be fused to obtain a cross-validation result; finally, after assigning fusion weights to one or more models to be fused based on the cross-validation result, the first electronic device performs model weighted fusion on the models to be fused based on the fusion weights of each model to be fused to obtain an updated cloud model.

[0007] The first electronic device is a cloud device, such as a server. The second electronic device is a terminal device or a client device, such as a mobile phone and a tablet. The model to be integrated refers to the model uploaded to the cloud by the end side. This model is generally used to provide personalized services to users, so it can be called a personalized model. The model to be integrated can also be called an end-side model or an end-side personalized model.

[0008] It should be pointed out that the model to be fused can be a model that has been trained on the device side but not model-validated, or a model that has been trained on the device side and model-validated.

[0009] It can be seen that the embodiment of the present application cross-validates the model to be fused before fusion of the model uploaded by the end-side device, that is, the credibility and contribution of the model are judged through cross-validation. Then, different fusion weights are assigned to each model to be fused according to the cross-validation results, that is, different weights are assigned to each model according to the confidence level of the model. Finally, weighted fusion is performed on each model to be fused according to different fusion weights to obtain a new cloud model after update. In this way, the robustness of the cloud model obtained by cloud fusion is improved.

[0010] In some possible implementations of the first aspect, after the first electronic device receives the models to be fused sent by one or more second electronic devices, the first electronic device can also determine the training results of each model to be fused according to a preset sufficient training condition, and the training results include sufficient training and insufficient training.

[0011] At this time, the specific process of the first electronic device assigning fusion weights to each model to be fused according to the cross-validation result may include: the first electronic device determines the first weight of each model to be fused according to the cross-validation result; the first electronic device adjusts the first weight for each model to be fused according to the training result to obtain the second weight; the first electronic device uses the second weight of each model to be fused as the fusion weight.

[0012] Specifically, after the cloud obtains the models to be fused uploaded by the end side, it can perform statistical analysis on the models to be fused, count the number of end-side training times and the amount of end-side training data for each model to be fused, and then determine whether the models to be fused are fully trained based on the statistical analysis results. The results of whether the models to be fused are fully trained can be used to assist in determining the fusion weight of each model to be fused.

[0013] Specifically, when allocating fusion weights for each model to be fused according to the cross-validation results, the cross-validation results and the results of whether the model to be fused is fully trained can be comprehensively considered to allocate fusion weights to the user.

[0014] The above-mentioned sufficient training conditions may include the number of end-side training times and the amount of end-side training data. The number of end-side training times and the amount of end-side training data can be obtained by analyzing the model parameters of the uploaded model to be integrated. Whether the model to be integrated is fully trained can be determined by counting the end-side training times and the amount of end-side training data of each model to be integrated.

[0015] In other words, in some possible implementations of the first aspect, the specific process of the first electronic device determining the training result of each model to be fused according to a pre-set sufficient training condition may include: if the number of end-side training times of the model to be fused is greater than a preset number threshold, and the amount of end-side data of the model to be fused is greater than a preset threshold, the first electronic device determines that the training result of the model to be fused is sufficient training; if the number of end-side training times of the model to be fused is less than the preset number threshold, and / or the amount of end-side data of the model to be fused is less than the preset threshold, the first electronic device determines that the training result of the model to be fused is insufficient training.

[0016] The number of terminal-side training times is the number of training times of the model to be trained on the second electronic device, and the amount of terminal-side data is the amount of training sample data locally on the second electronic device.

[0017] The above-mentioned preset number threshold and preset threshold can be set according to actual application requirements. For example, the prediction number threshold is 1000 and the preset threshold is 100.

[0018] After determining whether each model to be fused is sufficiently trained, a fusion weight can be assigned to the model based on the results of sufficient training and the cross-validation results. In this process, a first weight can be assigned to the model to be fused based only on the cross-validation results, and then the first weight can be adjusted based on whether it is sufficiently trained to obtain a second weight. The second weight is the fusion weight of the model to be fused.

[0019] Specifically, in some possible implementations of the first aspect, the first electronic device adjusts the first weight according to the training result, and the process of obtaining the second weight may include: if the training result of the model to be fused is sufficient training, the first electronic device adds the first weight and a preset value to obtain the second weight; if the training result of the model to be fused is insufficient training, the first electronic device subtracts the first weight from the preset value to obtain the second weight.

[0020] The above preset value can be set according to actual application requirements. For example, the preset value is 0.001.

[0021] In some possible implementations of the first aspect, the process of the first electronic device determining the first weight of each model to be fused based on the cross-validation results may include: the first electronic device adding the cross-validation results of each model to be fused to obtain the sum of the cross-validation results; the first electronic device using the ratio between the cross-validation result of each model to be fused and the sum of the cross-validation results as the first weight of the model to be fused.

[0022] In specific applications, in addition to assigning fusion weights to models based on whether the models to be fused are fully trained and the model cross-validation results, you can also assign fusion weights to models based solely on the model cross-validation results.

[0023] In some possible implementations of the first aspect, the process of the first electronic device assigning fusion weights to each model to be fused based on the cross-validation results may also include: the first electronic device adds the cross-validation results of each model to be fused to obtain the sum of the cross-validation results; the first electronic device uses the ratio between the cross-validation results of each model to be fused and the sum of the cross-validation results as the fusion weight of the model to be fused.

[0024] At this time, the fusion weight is assigned to the model only based on the model cross-validation results.

[0025] It should be pointed out that the method of assigning weights to models by comprehensively considering the sufficiency of the models to be fused and model cross-validation is more accurate than the method of assigning fusion weights to models only based on the results of model cross-validation.

[0026] In some possible implementations of the first aspect, the first electronic device uses a cross-validation data set to cross-validate the model to be fused, and the process of obtaining the cross-validation result may include: the first electronic device divides the cross-validation data set into sub-cross-validation data sets; the first electronic device uses each sub-cross-validation data set to validate each model to be fused, respectively, to obtain a sub-validation result, wherein one sub-cross-validation data set corresponds to one sub-validation result; the first electronic device obtains the cross-validation result of each model to be fused based on the sub-validation result of each model to be fused.

[0027] As an example but not a limitation, the cross-validation data set is divided into three sub-cross-validation data sets. For a certain model A to be fused, the three sub-validation results corresponding to these three sub-cross-validation data sets are 90%, 85%, and 95%, respectively. For the sake of simplicity of calculation, the average value of these three sub-validation results is calculated, that is, (90%+85%+95%) / 3=90%. At this time, the average value is used as the cross-validation result of the model A to be fused, that is, the cross-validation result of the model A to be fused is 90%.

[0028] In some possible implementations of the first aspect, the method may also include: the first electronic device obtains cloud verification data uploaded by the second electronic device, the cloud verification data is data generated by the second electronic device by simulating data on training samples; the first electronic device constructs a cross-validation data set based on the cloud verification data and cloud data, the cloud data is data stored locally in the cloud.

[0029] It should be noted that the end-side device may or may not perform the data simulation process. If the end-side device does not perform the data simulation process, the cloud device can build a cross-validation data set based only on the cloud data. If the end-side device performs the data simulation process and uploads the data obtained in the data simulation process to the cloud as the data for building the cloud cross-validation data set, the cloud device can build a cross-validation data set by combining the cloud verification data uploaded by the end-side device and the local data in the cloud.

[0030] Cloud data refers to data stored locally in the cloud. The data can be obtained by the cloud based on the model parameters of the model to be fused uploaded by the client, or it can be training data used to train the model to be trained.

[0031] In comparison, combining the cloud verification data uploaded on the client side with the cloud data to build a cross-validation dataset can enrich the scenarios and complexity of model verification, thereby further improving the robustness of the new model obtained by cloud fusion.

[0032] In some possible implementations of the first aspect, after the first electronic device uses the cross-validation data set to cross-validate the model to be fused and obtains the cross-validation result, the method may further include the following steps: if the model structure of the model to be fused is different from the model structure of the model to be fused in the previous round of training, the first electronic device determines a target cloud model from pre-stored cloud models according to the model structure of the model to be fused; the first electronic device performs model validation on the target cloud model; if the validation passes, the first electronic device uses the target cloud model as the updated cloud model;

[0033] If the model structure of the model to be fused is the same as the model structure of the model to be fused in the previous round of training, the first electronic device enters the step of assigning fusion weights to each model to be fused according to the cross-validation result.

[0034] Specifically, after the cloud performs reliability cross-validation on the model to be fused, it can directly perform the model weighted fusion step, or it can first determine whether the model structure of the model to be fused has changed, and only perform model weighted fusion when the model structure has not changed. If the model structure has changed, a new cloud model can be obtained through model search. In this way, an updated cloud model can still be obtained when the model result changes.

[0035] In some possible implementations of the first aspect, the first electronic device performs model weighted fusion on the to-be-fused model according to the fusion weight of each to-be-fused model to obtain an updated cloud model, which may include: the first electronic device determines whether the to-be-fused model needs to be retrained according to a preset retraining condition; if retraining is required, the first electronic device trains the to-be-fused model according to the fusion weight by weighted federated meta-learning to obtain the fused model; and performs model verification on the fused model, and if the verification passes, uses the fused model as the updated cloud model;

[0036] If retraining is not required, the first electronic device performs weighted fusion on the models to be fused according to the fusion weights to obtain a fusion model; and performs model verification on the fusion model. If the verification passes, the fusion model is used as an updated cloud model.

[0037] It should be pointed out that the retraining conditions are mainly set according to the training tasks and model characteristics. For some simple models, the model obtained by directly performing weighted fusion or weighted averaging is very effective, so there is no need to retrain, such as random forest; while for some models with complex structures, the model obtained by directly performing weighted fusion or weighted averaging is less effective, so re-weighted training is required, such as deep learning models.

[0038] In the second aspect, an embodiment of the present application provides a model training method based on federated learning, which may include: a second electronic device receives a model to be trained sent by a first electronic device; then, the second electronic device verifies the data validity of a training sample data set to obtain a verified training sample data set; then, the second electronic device performs incremental training on the model to be trained based on the verified training sample data set to obtain an end-side model; finally, the second electronic device verifies the end-side model, and transmits the end-side model that has passed the model verification to the first electronic device as a model to be fused.

[0039] It can be seen that compared with the existing federated learning, when using local training data to train the model, there is no validation of the local training data, nor is there validation of the trained model. The embodiment of the present application validates the local training data, and then uses the validated training data to train the model. After the model training is completed, the client device can also perform model validation on the trained model, and after the validation is passed, the trained model is uploaded to the cloud. In this way, the effect of the model uploaded to the cloud can be guaranteed, and the robustness of the new model integrated with the cloud can be further improved.

[0040] In some possible implementations of the second aspect, the second electronic device performs data validity verification on the training sample data set, and the process of obtaining the verified training sample data set may include: the second electronic device performs data classification on the training sample data set to determine the first valid data, redundant data and noise data in the training sample data; the second electronic device removes the redundant data; the second electronic device performs noise data repair on the noise data to obtain the second valid data; the second electronic device composes the verified training sample data set based on the first valid data and the second valid data.

[0041] In some possible implementations of the second aspect, the second electronic device performs incremental training on the to-be-trained model according to the verified training sample data set to obtain the terminal-side model, which may include:

[0042] If the type of the model training task is different from the type of the model training task in the previous round of training, the second electronic device performs incremental training on the training model by multi-task incremental learning according to the verified training sample data set to obtain a terminal-side model;

[0043] If the type of the model training task is the same as the type of the model training task in the previous round of training, and the base model type of the model to be trained is a machine learning model, the second electronic device performs incremental training on the model to be trained by machine incremental learning according to the verified training sample data set to obtain a terminal-side model;

[0044] If the type of model training task is the same as the type of model training task in the previous round of training, and the base model type of the model to be trained is a non-machine learning model, the second electronic device performs incremental training on the model to be trained by means of neural network incremental learning based on the verified training sample data set to obtain an end-side model.

[0045] As an example but not limitation, when the model training task is a classification task, if the current model training task is a 3-classification task and the previous round of model training task is a 2-classification task, it is considered that the type of the model training task has changed.

[0046] It should be pointed out that this implementation method selects the optimal or most appropriate incremental training method for the end-side model according to the type of the end-side model and the end-side model training task, which further improves the training efficiency and training effect of the end-side model and enables the model to converge quickly.

[0047] In some possible implementations of the second aspect, the second electronic device may perform incremental training on the training model by means of neural network incremental learning according to the verified training sample data set to obtain the terminal-side model, and the process may include:

[0048] If the model to be trained meets the first condition, the second electronic device performs incremental training on the model to be trained by small sample learning and convolution layer updating according to the verified training sample data set to obtain a device-side model.

[0049] If the model to be trained meets the second condition, the second electronic device performs incremental training on the model to be trained by knowledge distillation and convolution layer solidification according to the verified training sample data set to obtain a terminal-side model.

[0050] In some possible implementations of the second aspect, the method may further include a data enhancement process, which may specifically include: the second electronic device obtains end-side sample data; the second electronic device performs data simulation based on the end-side sample data to generate training data and cloud verification data for constructing a cloud cross-validation data set; the second electronic device uploads the cloud verification data to the first electronic device; the second electronic device constructs a training sample data set based on the end-side sample data and the training data.

[0051] It should be pointed out that this implementation method first performs data enhancement on a small number of training samples to simulate and generate a large number of training samples, and then uses the training sample data generated by data simulation to train the model, which improves the model training efficiency and makes the trained model more personalized.

[0052] In a third aspect, an embodiment of the present application provides a model training device based on federated learning, which may include:

[0053] A model to be trained sending module, used for sending the model to be trained to the second electronic device;

[0054] A model receiving module, used to receive a model to be fused sent by one or more second electronic devices, where the model to be fused is a model obtained by the second electronic device training the model to be trained according to a local training sample data set;

[0055] A cross-validation module is used to cross-validate the fusion model using a cross-validation data set to obtain a cross-validation result;

[0056] A weight allocation module is used to allocate fusion weights to one or more models to be fused according to the cross-validation results;

[0057] The model fusion module is used to perform weighted fusion of the models to be fused according to the fusion weight of each model to be fused, so as to obtain an updated cloud model.

[0058] In some possible implementations of the third aspect, the apparatus may further include:

[0059] A statistical analysis module, used to determine the training results of each model to be fused according to a pre-set sufficient training condition, the training results including sufficient training and insufficient training;

[0060] At this time, the above-mentioned weight allocation module is specifically used to: determine the first weight of each model to be fused according to the cross-validation result; for each model to be fused, adjust the first weight according to the training result to obtain the second weight; and use the second weight of each model to be fused as the fusion weight.

[0061] In some possible implementations of the third aspect, the statistical analysis module is specifically used to:

[0062] If the number of terminal-side training times of the model to be fused is greater than a preset number threshold, and the amount of terminal-side data of the model to be fused is greater than a preset threshold, it is determined that the training result of the model to be fused is fully trained;

[0063] If the number of terminal-side training times of the model to be fused is less than a preset number threshold, and / or the amount of terminal-side data of the model to be fused is less than a preset threshold, it is determined that the training result of the model to be fused is insufficient training;

[0064] The number of terminal-side training times is the number of training times of the model to be trained on the second electronic device, and the amount of terminal-side data is the amount of training sample data locally on the second electronic device.

[0065] In some possible implementations of the third aspect, the above-mentioned weight allocation module is specifically used to: if the training result of the model to be fused is sufficient training, add the first weight and the preset value to obtain the second weight; if the training result of the model to be fused is insufficient training, subtract the first weight and the preset value to obtain the second weight.

[0066] In some possible implementations of the third aspect, the above-mentioned weight distribution module is specifically used to: add the cross-validation results of each model to be fused to obtain the sum of the cross-validation results; and use the ratio between the cross-validation results of each model to be fused and the sum of the cross-validation results as the first weight of the model to be fused.

[0067] In some possible implementations of the third aspect, the above-mentioned weight distribution module is specifically used to: add the cross-validation results of each model to be fused to obtain the sum of the cross-validation results; and use the ratio between the cross-validation results of each model to be fused and the sum of the cross-validation results as the fusion weight of the model to be fused.

[0068] In some possible implementations of the third aspect, the cross-validation module is specifically used to: divide the cross-validation data set into sub-cross-validation data sets; use each sub-cross-validation data set to validate each model to be fused, and obtain sub-validation results, wherein one sub-cross-validation data set corresponds to one sub-validation result; and obtain the cross-validation results of each model to be fused based on the sub-validation results of each model to be fused.

[0069] In some possible implementations of the third aspect, the apparatus may further include:

[0070] The cross-validation data set construction module is used to obtain the cloud verification data uploaded by the second electronic device. The cloud verification data is data generated by the second electronic device through data simulation of training samples; a cross-validation data set is constructed based on the cloud verification data and cloud data. The cloud data is data stored locally in the cloud.

[0071] In some possible implementations of the third aspect, the apparatus may further include:

[0072] The model search module is used to determine the target cloud model from the pre-stored cloud models according to the model structure of the model to be fused if the model structure of the model to be fused is different from the model structure of the model to be fused in the previous round of training; perform model verification on the target cloud model; if the verification passes, use the target cloud model as the updated cloud model; if the model structure of the model to be fused is the same as the model structure of the model to be fused in the previous round of training, enter the step of assigning fusion weights to each model to be fused according to the cross-validation result.

[0073] In some possible implementations of the third aspect, the model fusion module is specifically used to: determine whether the model to be fused needs to be retrained according to a pre-set retraining condition; if retraining is required, train the fused model through weighted federated meta-learning according to the fusion weight to obtain the fused model; perform model verification on the fused model, and if the verification passes, use the fused model as the updated cloud model; if retraining is not required, perform weighted fusion on the fused model according to the fusion weight to obtain the fused model; perform model verification on the fused model, and if the verification passes, use the fused model as the updated cloud model.

[0074] The above-mentioned model training device based on federated learning has the function of implementing the model training method based on federated learning of the first aspect mentioned above. This function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions, and the modules can be software and / or hardware.

[0075] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0076] In a fourth aspect, an embodiment of the present application provides a model training device based on federated learning, which may include the following modules:

[0077] A model to be trained receiving module, used for receiving the model to be trained sent by the first electronic device;

[0078] A data validity verification module is used to verify the data validity of the training sample data set to obtain a verified training sample data set;

[0079] A model training module is used to perform incremental training on the to-be-trained model according to the verified training sample data set to obtain a terminal-side model;

[0080] Model verification module, used to verify the end-side model;

[0081] The module for uploading models to be integrated is used to transmit the terminal-side model that has passed the model verification as the model to be integrated to the first electronic device.

[0082] In some possible implementations of the fourth aspect, the data validity verification module is specifically used to: perform data classification on the training sample data set to determine the first valid data, redundant data and noise data in the training sample data; remove redundant data; perform noise data repair on the noise data to obtain second valid data; and form a verified training sample data set based on the first valid data and the second valid data.

[0083] In some possible implementations of the fourth aspect, the model training module is specifically used to:

[0084] If the type of the model training task is different from that of the previous round of training, incremental training is performed on the training model using multi-task incremental learning based on the verified training sample dataset to obtain the client-side model.

[0085] If the type of the model training task is the same as that of the model training task in the previous round of training, and the base model type of the model to be trained is a machine learning model, incremental training is performed on the model to be trained using machine incremental learning based on the verified training sample dataset to obtain the client-side model.

[0086] If the type of the model training task is the same as that of the model training task in the previous round of training, and the base model type of the model to be trained is a non-machine learning model, incremental training is performed on the model to be trained using the neural network incremental learning method based on the verified training sample data set to obtain the end-side model.

[0087] In some possible implementations of the fourth aspect, the model training module is specifically used to:

[0088] If the model to be trained meets the first condition, the second electronic device performs incremental training on the model to be trained by small sample learning and convolution layer updating according to the verified training sample data set to obtain a device-side model.

[0089] If the model to be trained meets the second type of conditions, the second electronic device performs incremental training on the model to be trained by means of knowledge distillation and convolutional layer solidification according to the verified training sample data set to obtain a terminal-side model.

[0090] In some possible implementations of the fourth aspect, the apparatus may further include:

[0091] A data enhancement module is used to obtain end-side sample data; perform data simulation based on the end-side sample data to generate training data and cloud verification data for constructing a cloud cross-validation data set; upload the cloud verification data to the first electronic device; and construct a training sample data set based on the end-side sample data and the training data.

[0092] The above-mentioned model training device based on federated learning has the function of implementing the model training method based on federated learning of the second aspect mentioned above. This function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions, and the modules can be software and / or hardware.

[0093] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0094] In a fifth aspect, an embodiment of the present application provides a model training method based on federated learning, which may include:

[0095] The first electronic device sends the model to be trained to the second electronic device.

[0096] The second electronic device verifies the data validity of the training sample data set to obtain a verified training sample data set; then, incrementally trains the model to be trained based on the verified training sample data set to obtain an end-side model; then, verifies the end-side model; and then transmits the end-side model that has passed the model verification to the first electronic device as the model to be fused.

[0097] The first electronic device uses a cross-validation data set to cross-validate the model to be fused to obtain a cross-validation result; then, according to the cross-validation result, a fusion weight is assigned to one or more models to be fused; finally, according to the fusion weight of each model to be fused, a model weighted fusion is performed on the model to be fused to obtain an updated cloud model.

[0098] In some possible implementations of the fifth aspect, the second electronic device verifies data validity of the training sample data set to obtain the verified training sample data set, which may include:

[0099] The second electronic device performs data classification on the training sample data set to determine first valid data, redundant data and noise data in the training sample data; the second electronic device removes the redundant data; the second electronic device performs noise data repair on the noise data to obtain second valid data; the second electronic device composes a verified training sample data set based on the first valid data and the second valid data.

[0100] In some possible implementations of the fifth aspect, the second electronic device performs incremental training on the to-be-trained model according to the verified training sample data set to obtain the terminal-side model, which may include:

[0101] If the type of the model training task is different from the type of the model training task in the previous round of training, the second electronic device performs incremental training on the training model by multi-task incremental learning according to the verified training sample data set to obtain a terminal-side model;

[0102] If the type of the model training task is the same as the type of the model training task in the previous round of training, and the base model type of the model to be trained is a machine learning model, the second electronic device performs incremental training on the model to be trained by machine incremental learning according to the verified training sample data set to obtain a terminal-side model;

[0103] If the type of model training task is the same as the type of model training task in the previous round of training, and the base model type of the model to be trained is a non-machine learning model, the second electronic device performs incremental training on the model to be trained by means of neural network incremental learning based on the verified training sample data set to obtain an end-side model.

[0104] Further, in some possible implementations of the fifth aspect, the second electronic device performs incremental training on the training model by means of neural network incremental learning according to the verified training sample data set to obtain the terminal-side model, including:

[0105] If the model to be trained meets the first type of conditions, the second electronic device performs incremental training on the model to be trained by means of small sample learning and convolutional layer updating according to the verified training sample data set to obtain the terminal side model; if the model to be trained meets the second type of conditions, the second electronic device performs incremental training on the model to be trained by means of knowledge distillation and convolutional layer solidification according to the verified training sample data set to obtain the terminal side model.

[0106] In some possible implementations of the fifth aspect, the second electronic device may further perform a data simulation process. The data simulation process may include:

[0107] The second electronic device obtains the end-side sample data; the second electronic device performs data simulation based on the end-side sample data to generate training data and cloud verification data for constructing a cloud cross-validation data set; the second electronic device uploads the cloud verification data to the first electronic device; the second electronic device constructs a training sample data set based on the end-side sample data and the training data.

[0108] In some possible implementations of the fifth aspect, the method may further include: the first electronic device determines the training result of each model to be fused according to a preset sufficient training condition, where the training results include sufficient training and insufficient training.

[0109] At this time, the process of the first electronic device assigning fusion weights to each model to be fused according to the cross-validation result may include: the first electronic device determines the first weight of each model to be fused according to the cross-validation result; the first electronic device adjusts the first weight for each model to be fused according to the training result to obtain the second weight; the first electronic device uses the second weight of each model to be fused as the fusion weight.

[0110] In some possible implementations of the fifth aspect, the process in which the first electronic device determines the training result of each model to be fused according to a pre-set sufficient training condition may include: if the number of end-side training times of the model to be fused is greater than a preset number threshold, and the amount of end-side data of the model to be fused is greater than a preset threshold, the first electronic device determines that the training result of the model to be fused is sufficient training; if the number of end-side training times of the model to be fused is less than the preset number threshold, and / or the amount of end-side data of the model to be fused is less than the preset threshold, the first electronic device determines that the training result of the model to be fused is insufficient training.

[0111] The number of terminal-side training times is the number of training times of the model to be trained on the second electronic device, and the amount of terminal-side data is the amount of training sample data locally on the second electronic device.

[0112] In some possible implementations of the fifth aspect, the first electronic device adjusts the first weight according to the training result, and the process of obtaining the second weight may include: if the training result of the model to be fused is sufficient training, the first electronic device adds the first weight and a preset value to obtain the second weight; if the training result of the model to be fused is insufficient training, the first electronic device subtracts the first weight from the preset value to obtain the second weight.

[0113] In some possible implementations of the fifth aspect, the process of the first electronic device determining the first weight of each model to be fused based on the cross-validation results may include: the first electronic device adding the cross-validation results of each model to be fused to obtain the sum of the cross-validation results; the first electronic device using the ratio between the cross-validation result of each model to be fused and the sum of the cross-validation results as the first weight of the model to be fused.

[0114] In some possible implementations of the fifth aspect, the process of the first electronic device assigning fusion weights to each model to be fused based on the cross-validation results may include: the first electronic device adds the cross-validation results of each model to be fused to obtain the sum of the cross-validation results; the first electronic device uses the ratio between the cross-validation result of each model to be fused and the sum of the cross-validation results as the fusion weight of the model to be fused.

[0115] In some possible implementations of the fifth aspect, the first electronic device uses a cross-validation data set to cross-validate the model to be fused, and the process of obtaining the cross-validation result may include: the first electronic device divides the cross-validation data set into sub-cross-validation data sets; the first electronic device uses each sub-cross-validation data set to validate each model to be fused, and obtains a sub-validation result, wherein one sub-cross-validation data set corresponds to one sub-validation result; the first electronic device obtains the cross-validation result of each model to be fused based on the sub-validation result of each model to be fused.

[0116] In some possible implementations of the fifth aspect, the method may also include: the first electronic device obtains cloud verification data uploaded by the second electronic device, the cloud verification data is data generated by the second electronic device by simulating data on training samples; the first electronic device constructs a cross-validation data set based on the cloud verification data and cloud data, the cloud data is data stored locally in the cloud.

[0117] In some possible implementations of the fifth aspect, after the first electronic device uses a cross-validation data set to cross-validate the model to be fused and obtains the cross-validation result, the method may also include: if the model structure of the model to be fused is different from the model structure of the model to be fused in the previous round of training, the first electronic device determines the target cloud model from the pre-stored cloud models according to the model structure of the model to be fused; the first electronic device performs model verification on the target cloud model; if the verification passes, the first electronic device uses the target cloud model as the updated cloud model; if the model structure of the model to be fused is the same as the model structure of the model to be fused in the previous round of training, the first electronic device enters the step of assigning fusion weights to each model to be fused according to the cross-validation result.

[0118] In some possible implementations of the fifth aspect, the first electronic device performs model weighted fusion on the models to be fused according to the fusion weights of each model to be fused to obtain an updated cloud model. The process may include: the first electronic device determines whether the model to be fused needs to be retrained according to a preset retraining condition; if retraining is required, the first electronic device trains the model to be fused by weighted federated meta-learning according to the fusion weights to obtain a fusion model; performs model verification on the fusion model, and if the verification passes, uses the fusion model as the updated cloud model; if retraining is not required, the first electronic device performs weighted fusion on the models to be fused according to the fusion weights to obtain a fusion model; performs model verification on the fusion model, and if the verification passes, uses the fusion model as the updated cloud model.

[0119] In a sixth aspect, an embodiment of the present application provides a model training device based on federated learning, which includes a first electronic device and a second electronic device, and the device is used to implement the function of the model training method based on federated learning in the fifth aspect mentioned above.

[0120] In the seventh aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method of any one of the first aspect or the second aspect is implemented.

[0121] In an eighth aspect, an embodiment of the present application provides a model training system based on federated learning, which may include a first electronic device and a second electronic device.

[0122] The first electronic device is used to execute the model training method of any one of the first aspects above, and the second electronic device is used to execute the model training method of any one of the second aspects above.

[0123] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of any one of the first aspect or the second aspect is implemented.

[0124] In a tenth aspect, an embodiment of the present application provides a chip system, the chip system comprising a processor, the processor coupled to a memory, the processor executing a computer program stored in the memory to implement the method as described in any one of the first aspect or the second aspect. The chip system can be a single chip, or a chip module composed of multiple chips.

[0125] In an eleventh aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes any one of the methods in the first or second aspect above.

[0126] It can be understood that the beneficial effects of the third to eleventh aspects mentioned above can be found in the relevant descriptions in the first or second aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 A schematic block diagram of the model training system architecture provided in an embodiment of the present application;

[0128] Figure 2 A schematic diagram of a data simulation process provided in an embodiment of the present application;

[0129] Figure 3 A schematic diagram of the end-side data validity verification provided in an embodiment of the present application;

[0130] Figure 4 A schematic diagram of the end-side model training process provided in an embodiment of the present application;

[0131] Figure 5 A schematic diagram of the end-cloud collaboration process provided in an embodiment of the present application;

[0132] Figure 6 A schematic diagram of the model training process provided in the embodiment of the present application;

[0133] Figure 7 A schematic diagram of a model training process provided in an embodiment of the present application;

[0134] Figure 8 Another schematic diagram of the model training process provided in the embodiment of the present application;

[0135] Fig. 9 Another schematic diagram of the model training process provided in the embodiment of the present application;

[0136] Fig.10 Another schematic diagram of model training provided in an embodiment of the present application;

[0137] Fig.11 A schematic diagram of the regression prediction effect provided in the embodiment of the present application;

[0138] Fig.12 A schematic diagram of picture recommendations provided in the embodiment of the present application;

[0139] Fig.13 A schematic diagram of the interaction between the client device and the cloud device provided in the embodiment of the present application;

[0140] Fig.14 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0141] During the research, the inventors found that when the existing technology uses federated learning to train models, the cloud server does not verify the reliability of the model uploaded by the client device before model fusion, and adopts an average update strategy during model fusion, without assigning different weights according to the different confidence levels of the model. This will result in poor robustness of the new model obtained by the cloud server fusion, and cannot ensure the effect of the cloud aggregation model.

[0142] In addition, in existing federated learning, when client devices use local training data to train models, they do not verify the validity of the local training data, nor do they verify the trained models. In this way, the effect of the model uploaded to the cloud cannot be guaranteed, which leads to poor robustness of the new model fused by the cloud server.

[0143] In addition, in existing federated learning, the number of training samples of the local training data of client devices is small, resulting in low training efficiency and difficulty in fast model convergence.

[0144] In response to the problems existing in traditional federated learning, the embodiment of the present application provides a model training solution based on federated learning. In the embodiment of the present application, before the model is integrated, the cloud device verifies the reliability of the model uploaded by the client device to judge the credibility and contribution of the model; then calculates the weights corresponding to each model based on the reliability verification results, that is, assigns different weights to each model according to the confidence level of each model; finally, the model is weightedly integrated according to the weights corresponding to each model to obtain a new model aggregated in the cloud. In this way, the robustness of the new model aggregated in the cloud can be improved.

[0145] Furthermore, in an embodiment of the present application, the client device may first perform data validity verification on the local training data, and then use the training data after validity verification to train the model. After the model training is completed, the client device may also perform model verification on the trained model, and after the verification is passed, the trained model is uploaded to the cloud. In this way, the effect of the model uploaded to the cloud can be guaranteed, and the robustness of the new model integrated in the cloud can be further improved.

[0146] Furthermore, in an embodiment of the present application, the client device may first perform data enhancement on a small number of training samples to simulate and generate a large number of training samples, and then use the training sample data generated by data simulation to train the model, thereby improving the efficiency of model training and making the trained model more personalized.

[0147] In order to better introduce the technical solutions of the embodiments of the present application, the relevant contents will be elaborated in detail below.

[0148] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0149] The following first introduces the system architecture that may be involved in the embodiments of the present application.

[0150] See also Figure 1 The schematic block diagram of the model training system architecture provided by the embodiment of the present application is shown as follows: Figure 1 As shown, the system includes a cloud device 11 and a client device 12, and the cloud device 11 and the client device 12 are connected to each other through a communication network 13, and the communication network 13 can be, for example, the Internet. The number and type of the client devices 12 can be arbitrary. Generally, there are at least two client devices 12, that is, at least two client devices can participate in model training. The type of client device can be, but is not limited to, a mobile phone, a tablet, a car terminal or a computer, etc., as an example and not a limitation, Figure 1 The client device 12 includes mobile phones and tablets. The cloud device 11 is generally a cloud server. The cloud device 11 can send the model to be trained to the client device 12. The model is generally an AI model. The type of AI model can be arbitrary. As an example but not limitation, the AI ​​model can be a deep learning model or a machine learning model. More specifically, the AI ​​model will be different depending on the actual application scenario. The AI ​​model can provide personalized services for users and can also be called a user personalized model. For example, in the voice wake-up scenario, the AI ​​model is a voice wake-up model; in the charging control scenario, the AI ​​model is a charging control model; in the news recommendation scenario, the AI ​​model is a news recommendation model.

[0151] It should be pointed out that the above-mentioned model to be trained can be the model updated by the cloud device after the last round of model training. That is to say, the cloud device and the client device can perform multiple rounds of model training. In one round of model training, the cloud device can send the model obtained by the last round of fusion update to the client device as the model to be trained. After the client device uses local data to train the model to be trained, it uploads the trained model to the cloud. The cloud device performs model fusion update based on the models uploaded by each client device to obtain an updated model. In this cycle, one or more rounds of model training are performed. Of course, in the first round of model training, the model to be trained can be a general initial model pre-trained by the cloud device.

[0152] For example, in the voice wake-up scenario, the model to be trained is a voice wake-up model. Before the first round of model training, the cloud server will first collect a large amount of legal and compliant voice wake-up data, which may include voice data of multiple users, and the voice data may be collected under the premise of obtaining user authorization. The cloud trains the constructed voice wake-up model based on the collected voice wake-up data. The trained voice wake-up model is used as the initial voice wake-up model. The initial voice wake-up model can well recognize the wake-up voice of standard Mandarin. However, for some Mandarin mixed with dialects or local accents, the voice wake-up effect may not be good. Or, in some cases where there is a lot of background noise, the voice wake-up effect of the initial voice wake-up model is not good. After the cloud sends the initial voice wake-up model to the user's mobile phone, the mobile phone can use the initial voice wake-up model for wake-up voice recognition, and in this process, the mobile phone can collect end-side data related to voice wake-up after obtaining clear user authorization. At this time, the end-side data may include voice data corresponding to successful voice wake-up, and voice data corresponding to failed voice wake-up but successful wake-up by other means (for example, successful facial wake-up when voice wake-up fails), etc.

[0153] After receiving the model to be trained sent by the cloud device, the client device can use the training sample data set on the client side to train the model to be trained.

[0154] For example, in the news recommendation scenario, the model to be trained is a news recommendation model, which can recommend news that meets the user's preferences based on data such as the user's browsing history. At this time, after obtaining clear user authorization, the mobile phone can collect relevant end-side data through data dotting. The relevant end-side data may include the time period when the user browses news (for example, morning or evening) and the news category (for example, sports news or entertainment news). After obtaining clear user authorization, the mobile phone can combine the relevant end-side data into training sample data for the news recommendation model.

[0155] In some embodiments, after collecting a small amount of training sample data, the client device can form a training sample data set based on the small amount of sample data, and use the training sample data set to train the model to be trained. However, when training a model using a training sample data set including a small amount of samples, the model training efficiency is low due to the small amount of sample data, and it is difficult to quickly converge to a personalized model that meets the personalized needs of the user.

[0156] In other embodiments, in order to improve the model training efficiency, the model can quickly converge to a personalized model. After collecting a small amount of sample data, the client device 12 can perform data enhancement on the small amount of sample data to generate a large amount of training sample data through data simulation. Then, the training sample data generated based on the data simulation is used to form a training sample data set, and the training sample data set is used to train the model to be trained. In this way, a large amount of training sample data is generated through data enhancement, and then a large amount of training sample data is used to train the model, which can improve the model training efficiency, enable the model to converge quickly, and make the trained model more personalized.

[0157] As an example and not a limitation, the following Figure 2 A schematic diagram of the data simulation process is shown for introduction.

[0158] like Figure 2 As shown, the client device obtains an initial small amount of samples from the end-side data pool, which may include but is not limited to historical behavior data and dot data. After the client device obtains a small amount of sample data from the end-side data pool, it can perform preliminary data analysis on the small amount of sample data. The preliminary data analysis may refer to operations such as data classification, clustering or regression. Then, based on the preliminary data analysis results, the user behavior on the client side is reinforced learning modeled to obtain a corresponding model, and then the model is used to generate AI simulation data. The generated data can be added to the training data set as training data for model training of the client device, or added to the cloud verification data, which can be used to construct a cloud cross-validation data set. Next, determine whether the amount of training data is sufficient. If the amount of data is not sufficient, re-generate AI simulation data; if the amount of data is sufficient, perform data preprocessing operations on the training data to obtain end-side data. At this point, the end-side data preparation is complete.

[0159] It should be pointed out that by performing data simulation on a small amount of sample data to generate training sample data with a larger amount of sample data, the training sample data set on the client device side is expanded, thereby improving the model training efficiency on the client device side and making the model trained on the client side more personalized.

[0160] In addition, after obtaining a small amount of sample data or generating a large amount of sample data through data simulation, before using the training sample data to train the model, the obtained training sample data can also be verified for data validity to obtain the optimal training data set, and then the optimal training data set is used to train the model. It should be understood that the training sample data here can be sample data obtained after data enhancement or sample data without data enhancement.

[0161] Data validity verification of training sample data may include but is not limited to steps such as data classification, redundant data removal and noise data repair. Data classification refers to classifying training sample data to determine valid data, redundant data and noise data in the training sample data. Redundant data refers to repeated data in a data set, that is, there are multiple identical data in a data set. Noise data refers to meaningless data in a data set, which generally refers to data with errors or anomalies in the data. Valid data refers to data in a data set that meets data quality specifications, and the quality of data can be evaluated by the integrity, consistency and accuracy of the data.

[0162] For redundant data, the redundant data can be removed, and for noise data, the noise data can be repaired to convert the noise data into valid data. Finally, an optimal training data set is formed based on the valid data. The valid data in the optimal training data set may include valid data obtained by repairing the noise data and valid data obtained by data classification.

[0163] Of course, after obtaining the training sample data and before verifying the data validity, you can also perform data cleaning operations on the training sample data to perform preliminary data cleaning on the training sample data and screen out data that is obviously erroneous, incomplete, and incorrectly formatted.

[0164] As an example and not a limitation, the following Figure 3 The schematic diagram of the end-side data validity verification is shown for introduction.

[0165] like Figure 3As shown, after the client device obtains the end-side data, it can first clean the end-side data. The end-side data can be sample data with a large amount of data obtained through data enhancement, or sample data with a small amount of data without data enhancement. Then, the end-side data validity verification is performed. The process of end-side data validity verification may include: classifying the training sample data in an active learning manner to determine the valid data, redundant data and noise data in the training sample data. Specifically, in the process of active learning, a supervised learning process and an unsupervised learning process can be performed separately based on whether there are labels or not. Specifically, it can be determined whether the sample data has a label first. If there is a label, the data is classified by supervised learning in active learning. If there is no label, the unlabeled data is automatically labeled by unsupervised learning in active learning, and then the data is classified to obtain the data classification result. In a specific application, the label of the training sample data can be specifically expressed as 0 and 1. For example, when the training sample data is voice data in a voice wake-up scenario, the label is divided into 0 and 1, 0 indicates that the training sample data is the voice data corresponding to the voice wake-up, and 1 indicates that the training sample data is the voice data corresponding to the voice wake-up.

[0166] After obtaining the data classification result, it can be determined whether there is redundant data in the end-side data. If so, the redundant data is stored in the data cache area. If there is no redundant data, it is determined whether there is noise data in the end-side data. If there is noise data, the noise data is repaired and then it is determined whether there is valid data in the end-side data. If there is no noise data, it is determined whether there is valid data in the end-side data. If there is valid data in the end-side data, a end-side training set is formed based on the valid data, and the end-side training set is the optimal training sample data set. If there is no valid data in the end-side data, the end-side data is stored in the data cache area.

[0167] It should be noted that Figure 3 The end-side data in the above may refer to the training sample data used for end-side model training. Figure 2 The client-side data in the example may include cloud-based validation data for building a cloud-based cross-validation dataset and training sample data for client-side model training. Figure 3 The end-side data may also include cloud verification data for constructing a cloud cross-validation data set and training sample data for end-side model training. That is, in addition to data validity verification of the training sample data, data validity verification may also be performed on the cloud verification data.

[0168] It should be pointed out that before model training, the training sample data is verified for data validity to obtain the optimal training data set, and then the model is trained using the optimal training data set to ensure the effect of the model uploaded to the cloud by the client device as much as possible, thereby further improving the robustness of the new model integrated into the cloud.

[0169] After the client device obtains the training sample data set, it can use the training sample data to train the model. At this time, the training sample data set can be Figure 3 The end-side training set, that is, the training sample data set used in the end-side model training is a data set that has been verified for data validity. Of course, the training sample data set may also be a data set that has not been verified for data validity. In addition, the training sample data set may be a data set obtained after data enhancement or a data set without data enhancement. In specific applications, the training process of the end-side model may also be different depending on the training sample data set. It can be understood that the model training method on the client device is incremental training.

[0170] As an example and not a limitation, the following Figure 4 The schematic diagram of the end-side model training process is shown for introduction.

[0171] like Figure 4 As shown, after the client device obtains the end-side training set, it first determines whether the obtained model training task has changed. Whether the model training task has changed can be determined by determining whether the model training task has increased or decreased. Specifically, the current model training task can be compared with the previous model training task to determine whether the current training task has increased or decreased. For example, if the previous training task is a three-category task and the current training task is a four-category task, it can be determined that the current model training task has increased.

[0172] If the model training task changes, multi-task incremental learning is used to perform incremental model training to obtain a trained model. The multi-task incremental learning algorithm may be, for example, LWF (learning without forgetting) or EWC (elastic weight consolidation).

[0173] If the model training task has not changed, the base model type is then determined, and different incremental learning methods are used according to the base model type. The base model type may include deep learning models and machine learning models. If the base model type is a machine learning model, the model is incrementally trained using the machine learning incremental training method to obtain a trained model. If the base model type is not a machine learning model, the model is incrementally trained using the neural network incremental learning method.

[0174] In the incremental learning method of neural networks, the difficulty of updating the model can be determined first. Specifically, the difficulty of updating the model can be determined by judging the model structure and the amount of training sample data. If the model is difficult to update, the model is incrementally trained by using meta-learning and convolutional layer updates; conversely, if the model is easy to update, the model is incrementally trained by using knowledge distillation and convolutional layer solidification.

[0175] When judging the difficulty of model updating, a sample data volume threshold and a threshold for the number of layers in the model can be set in advance. The amount of training sample data can be judged by judging the size relationship between the training sample data volume and the set sample data volume threshold, and the complexity of the model structure can be judged by judging the size relationship between the layers of the model to be trained and the threshold for the number of layers. For example, if the model to be trained has only 3 fully connected layers, the structure of the model to be trained is judged to be simple, and if it has 50 fully connected layers, the structure of the model to be trained can be judged to be complex. In general, if the model structure is complex and the amount of training sample data is small, the model is judged to be difficult to update. Conversely, if the model structure is simple and the amount of training sample data is large, the model is judged to be easy to update.

[0176] It should be noted that if the end-side training set is a data set obtained after data enhancement, at this time, since the training set contains a large amount of training sample data, it can be determined that the model is easy to update, and the knowledge distillation and convolution layer solidification method is used to perform incremental model training. If the end-side training set is a data set that has not undergone data enhancement, at this time, the training set contains a small amount of training sample data, and it can be determined that the model is difficult to update, then meta-learning and convolution layer update methods are used to perform incremental model training. Specifically, meta-learning few-shot learning can be used to perform incremental model training. Of course, it is also necessary to combine the complexity of the model structure to determine whether the model is easy to update.

[0177] Of course, in some other embodiments, the client-side model training process may be different from the above-mentioned process. For example, it may not be necessary to determine whether the task has changed, or to determine the difficulty of model update. Figure 4 In the model training process mentioned above, different incremental training methods are used depending on whether the task changes. In this way, the model can still be incrementally trained when the task changes. In addition, different incremental training methods are used according to the type of base model. In this way, even in the case of a small number of samples, the model can still be incrementally trained through the incremental training method of meta-learning, that is, through the small sample learning in meta-learning and the convolution layer update method, to obtain a trained model. In other words, Figure 4The corresponding end-side model training method can select the optimal or most appropriate end-side model incremental training method according to the type of end-side model and the end-side model training task, further improving the training efficiency and training effect of the end-side model, so that the model can converge quickly.

[0178] After the client device performs incremental training of the end-side model, it can verify the trained model and upload the model to the cloud only after the verification is passed. This can ensure the effect of the model uploaded to the cloud as much as possible, thereby further improving the robustness of the new model integrated in the cloud. Model verification can refer to verifying the trained model using the corresponding verification data set to obtain the model verification result. When the model verification result meets certain conditions, it can be determined that the model verification has passed and the result is Figure 4 The terminal-side personalized model in the test result is then uploaded to the cloud; conversely, when the model test result does not meet the corresponding conditions, it can be determined that the model verification has not passed. At this time, the terminal-side training set can be used again for model incremental training.

[0179] In specific applications, the model verification result is generally expressed as the accuracy of the model. The corresponding accuracy threshold is pre-set. If the accuracy of the model is greater than the set accuracy threshold, the model verification is determined to have passed. Conversely, if the accuracy of the model is less than the set accuracy threshold, the model verification is determined to have failed. For example, in the voice wake-up scenario, the set accuracy threshold is 90%. The incrementally trained voice wake-up model is verified using the verification data set, and the wake-up success rate of the voice wake-up model is 91%. Since 91%>90%, the model verification is determined to have passed, and the incrementally trained voice wake-up model can be uploaded to the cloud.

[0180] Of course, in some other embodiments, after obtaining the trained end-side model, the client device may not perform model verification, but directly upload the trained model to the cloud.

[0181] After the client device obtains a personalized model that can be uploaded to the cloud through the end-side model training, it uploads the personalized model to the cloud device. The cloud device can first count the number of personalized models received, and only perform model fusion when the personalized model data reaches a certain amount.

[0182] After obtaining a sufficient number of personalized models, the cloud device can first perform reliability cross-validation on the personalized model based on the cross-validation data set, and then determine the fusion weight of each personalized model based on the model reliability cross-validation result. Finally, the model is weighted fused according to the fusion weights corresponding to each personalized model to obtain a new cloud model after fusion and update.

[0183] In some embodiments, the cloud cross-validation data set may be composed of cloud validation data and cloud data generated by the client device during the data simulation process. Figure 2 The cloud verification data in the cloud. The cloud data can be generated by the cloud device according to the parameters or feature information of the personalized model uploaded by the client device, that is, the cloud device can receive the personalized models uploaded by each client device, and then generate the corresponding cloud data according to the features or parameters of each personalized model and the data pre-stored in the cloud, and construct a cross-validation data set based on the generated cloud data and the cloud verification data generated by the client device.

[0184] The data pre-stored in the cloud may refer to data used in the cloud for training the initial model. For example, in a voice wake-up scenario, the model to be trained is a voice wake-up model. The cloud will first collect a large amount of voice data as a training data set for training the initial voice wake-up model. At this time, the training data set used in the cloud for training the initial voice wake-up model is the data pre-stored in the cloud.

[0185] In other embodiments, the client device may not perform the data simulation process and does not generate corresponding cloud verification data. In this case, the cloud cross-validation data set may only consist of cloud data. Cloud data is data generated by the cloud device based on the parameters or feature information of the personalized model uploaded by the client device and the data pre-stored in the cloud.

[0186] In some other embodiments, the cross-validation dataset in the cloud may also consist only of data pre-stored in the cloud.

[0187] In comparison, the cross-validation dataset constructed based on the cloud verification data generated by the client device and the cloud data generated by the cloud can enrich the scenarios and complexity of model validation, thereby further improving the robustness of the new model obtained by cloud fusion.

[0188] After constructing the cross-validation data set, the cloud device can use the cross-validation data set to perform model reliability cross-validation on each personalized model to obtain a cross-validation result. The process of model reliability cross-validation may include: dividing the constructed cross-validation data set into multiple sub-validation data sets, and using the multiple sub-validation data sets to verify each end-side personalized model to obtain a verification result of each sub-validation data set; for each end-side personalized model, a verification result can be calculated based on the verification results corresponding to each sub-validation data set, and this verification result is used as the cross-validation result of the end-side personalized model.

[0189] Model reliability cross-validation refers to cross-validation of the model, the purpose of which is to obtain a model with higher reliability. For example, in the voice wake-up scenario, the terminal-side personalized model is the voice wake-up model, and the model reliability cross-validation result is specifically manifested as the voice wake-up success rate. After obtaining the voice wake-up success rate of each voice wake-up model through model cross-validation, the voice wake-up model with a relatively high voice wake-up success rate is selected for model weighted fusion. As an example and not a limitation, the accuracy threshold is set to 95%, that is, the voice wake-up model with a voice wake-up success rate greater than or equal to 95% is selected for model weighted fusion. Of course, the voice wake-up model for model weighted fusion can also be selected by the quantity threshold. As an example and not a limitation, the top 100 voice wake-up models are selected for model weighted fusion based on the voice wake-up success rate.

[0190] Of course, after the model reliability is cross-validated, the cloud device does not need to select the end-side personalized model for model weighted fusion based on the cross-validation results, but can use all the end-side personalized models for model weighted fusion.

[0191] After the cloud device uses the cross-validation data set to cross-validate the model reliability, it can obtain the corresponding cross-validation results. Then, based on the cross-validation results, it calculates the fusion weights of each personalized model. Finally, it performs weighted fusion of the models based on the fusion weights to obtain an updated new model.

[0192] Of course, in some other embodiments, in addition to calculating the model fusion weight according to the cross-validation results, the personalized model can also be statistically analyzed, and the model fusion weight can be determined by the statistical analysis results. Statistical analysis can refer to separately counting the number of terminal-side training times and the size of the terminal-side training data set of the personalized model uploaded by each client device, etc., to obtain statistical analysis results. After the cloud device obtains the statistical analysis results, it can determine the fusion weight of the terminal-side personalized model in combination with the statistical analysis results and the model reliability cross-validation results.

[0193] That is to say, the fusion weight of the terminal-side personalized model can be determined only according to the model reliability cross-validation result, or the fusion weight of the terminal-side personalized model can be determined in combination with the model reliability cross-validation result and the statistical analysis result.

[0194] The following first introduces the process of determining the fusion weight of the terminal-side personalized model based only on the model reliability cross-validation result.

[0195] After the cloud device constructs the cloud cross-validation data set and obtains the end-side personalized model uploaded by each client device, it first divides the cloud cross-validation data set into multiple sub-validation data sets, where the number of sub-validation data sets can be set according to actual needs and actual application scenarios. Then, each end-side personalized model is verified using each sub-validation data set to obtain a verification result. At this time, since there are multiple sub-validation data sets, each end-side personalized model corresponds to multiple verification results. The multiple verification results of each end-side personalized model are weighted averaged or arithmetic averaged to obtain an average value, which is used as the model cross-validation result of the end-side personalized model.

[0196] After obtaining the model cross-validation results of each end-side personalized model, the end-side personalized model used for model weighted fusion can be screened out according to the pre-set screening conditions. The screening condition can be a quantity threshold or a model accuracy threshold. For example, according to the model cross-validation results, the top 100 end-side personalized models are selected, or the end-side personalized models with a model accuracy higher than 90% are selected, and the selected models are used for model weighted fusion.

[0197] After the models for weighted fusion are selected according to the model cross-validation results, corresponding weights can be assigned to each model based on the model cross-validation results.

[0198] For example, in the voice wake-up scenario, the cloud obtains 1,000 voice wake-up models, and divides the cloud cross-validation data set into 3 sub-validation data sets, each of which includes the user's voice data.

[0199] The 1000 voice wake-up models were verified using the three sub-validation datasets to obtain the voice wake-up success rate of the model, which is the model verification result. For each voice wake-up model, there are three voice wake-up success rates corresponding to the three sub-validation datasets, that is, each voice wake-up model has three corresponding voice wake-up success rates.

[0200] The three voice wake-up success rates of one of the voice wake-up models are 90%, 80% and 70% respectively. For the convenience of calculation, the average value of the three voice wake-up success rates is calculated here, that is, 90%+80%+70%=240%, 240% / 3=80%. The calculated average value is used as the voice wake-up success rate of the voice wake-up model, that is, the voice wake-up success rate of the voice wake-up model is 80%.

[0201] Similarly, the voice wake-up success rates of the 1000 voice wake-up models are calculated accordingly.

[0202] After calculating the voice wake-up success rates of 1,000 voice wake-up models, the voice wake-up models used for model weighted fusion can be screened out from the 1,000 voice wake-up models according to the voice wake-up success rates. The screened voice wake-up models are models with higher reliability. For ease of calculation, the top 10 models with the highest voice wake-up success rates are screened out for model fusion. The voice wake-up success rates of these 10 voice wake-up models are 95%, 96%, 94%, 90%, 92%, 92%, 96%, 90%, 90%, and 90%, respectively.

[0203] Based on the 10 voice wake-up success rates, the corresponding weights are assigned to the 10 voice wake-up models. For the sake of simplicity in calculation, the weighted average is used as the weight of each voice wake-up model. First, calculate the sum of the 10 voice wake-up success rates, that is, 95% + 96% + 94% + 90% + 92% + 92% + 96% + 90% + 90% + 90% = 925%; then, respectively, compare the respective voice wake-up success rates with the sum, and use the ratio as the weight of the voice wake-up model, that is, 95 / 925, 96 / 925, 94 / 925, 90 / 925, 92 / 925, 92 / 925, 96 / 925, 90 / 925, 90 / 925 and 90 / 925. Finally, the weights of the 10 voice wake-up models are 0.103, 0.104, 0.102, 0.097, 0.099, 0.099, 0.104, 0.097, 0.097 and 0.097 respectively. So far, the 10 voice wake-up models are assigned corresponding weights. Subsequently, the 10 voice wake-up models can be weightedly fused according to the different assigned weights to obtain a new voice wake-up model.

[0204] After calculating the voice wake-up success rates of 1,000 voice wake-up models, in addition to the weight allocation method described above, you can also use all 1,000 voice wake-up models for model weighted fusion instead of screening models based on the voice wake-up success rates. At this time, you need to assign corresponding weights to each of the 1,000 voice wake-up models. The process of assigning weights to the 1,000 voice wake-up models is similar to the process of assigning weights to the 10 voice wake-up models mentioned above, that is, calculate the sum of the voice wake-up success rates of all models, and then compare the voice wake-up success rate of each model to the sum to get the corresponding weight. The specific process will not be repeated here.

[0205] After introducing how to determine the fusion weight of the terminal-side personalized model based only on the model cross-validation result, the following describes how to determine the fusion weight of the terminal-side personalized model based on the model cross-validation result and the statistical analysis result.

[0206] After the cloud obtains the end-side personalized model, it can perform statistical analysis on the end-side personalized model to obtain statistical analysis results. As an example and not a limitation, the number of training times and the amount of training sample data for each end-side personalized model are counted. According to the pre-set conditions for whether the model is fully trained, determine whether the end-side personalized model is fully trained. The conditions for sufficient training can be configured according to actual needs and actual application scenarios. For example, the conditions for sufficient training are: the number of training times is greater than or equal to 1000 times, and the amount of training sample data is greater than or equal to 100, that is, if the number of training times of a certain end-side personalized model is greater than or equal to 1000 times, and the amount of training sample data is greater than or equal to 100, then the end-side personalized model is considered to be fully trained, otherwise, it is considered not to be fully trained.

[0207] After the cloud cross-validation dataset is constructed in the cloud, the weight of the terminal-side personalized model can be calculated according to the process of determining the fusion weight of the model based on the model cross-validation results as shown above. However, the weight needs to be adjusted accordingly according to whether the model is fully trained to obtain the final fusion weight.

[0208] Adjusting the weights according to whether the model is fully trained may mean: if the model is fully trained, then adding a preset value to the fusion weight calculated based on the model cross-validation result; conversely, if the model is not fully trained, then subtracting a preset value from the fusion weight calculated based on the model cross-validation result.

[0209] For example, in the voice wake-up scenario, the cloud obtains 1,000 voice wake-up models, and divides the cloud cross-validation data set into 3 sub-validation data sets, each of which includes the user's voice data.

[0210] The 1000 voice wake-up models were verified using the three sub-validation datasets to obtain the voice wake-up success rate of the model, which is the model verification result. For each voice wake-up model, there are three voice wake-up success rates corresponding to the three sub-validation datasets, that is, each voice wake-up model has three corresponding voice wake-up success rates.

[0211] The three voice wake-up success rates of one of the voice wake-up models are 90%, 80% and 70% respectively. For the convenience of calculation, the average value of the three voice wake-up success rates is calculated here, that is, 90%+80%+70%=240%, 240% / 3=80%. The calculated average value is used as the voice wake-up success rate of the voice wake-up model, that is, the voice wake-up success rate of the voice wake-up model is 80%.

[0212] Similarly, the voice wake-up success rates of the 1000 voice wake-up models are calculated accordingly.

[0213] After calculating the voice wake-up success rates of 1,000 voice wake-up models, the voice wake-up models for model weighted fusion can be screened out from the 1,000 voice wake-up models according to the voice wake-up success rates. The screened voice wake-up models are models with higher reliability. For ease of calculation, the top 10 models with the highest voice wake-up success rates are screened for model fusion. The voice wake-up success rates of these 10 voice wake-up models are 95%, 96%, 94%, 90%, 92%, 92%, 96%, 90%, 90%, and 90%, respectively.

[0214] Based on the 10 voice wake-up success rates, the corresponding weights are assigned to the 10 voice wake-up models. For the sake of simplicity in calculation, the weighted average is used as the weight of each voice wake-up model. First, calculate the sum of the 10 voice wake-up success rates, that is, 95% + 96% + 94% + 90% + 92% + 92% + 96% + 90% + 90% + 90% = 925%; then, respectively, compare the respective voice wake-up success rates with the sum, and use the ratio as the weight of the voice wake-up model, that is, 95 / 925, 96 / 925, 94 / 925, 90 / 925, 92 / 925, 92 / 925, 96 / 925, 90 / 925, 90 / 925 and 90 / 925. Finally, the weights of the 10 voice wake-up models are obtained, which are 0.103, 0.104, 0.102, 0.097, 0.099, 0.099, 0.104, 0.097, 0.097 and 0.097 respectively.

[0215] Next, the results of statistical analysis are used to determine whether the 10 voice wake-up models are fully trained. Suppose that the conditions for sufficient training are: the number of training times is greater than or equal to 1000 times, and the amount of training sample data is greater than or equal to 100. The results of determining whether the 10 voice wake-up models are fully trained based on the sufficient training conditions are: 95%, 96%, 94%, 90% and 92% of the corresponding voice wake-up models are fully trained, and 92%, 96%, 90%, 90% and 90% of the corresponding voice wake-up models are not fully trained.

[0216] For fully trained voice wake-up models, the weight value is increased by 0.001, and for inadequately trained voice wake-up models, the weight value is decreased by 0.001. The weight values ​​adjusted according to whether they are fully trained or not are: 0.104, 0.105, 0.103, 0.098, 0.100, 0.098, 0.103, 0.096, 0.096, and 0.096. In this way, the final fusion weights of the 10 voice wake-up models can be obtained, and the adjusted fusion weights can be used for model weighted fusion later.

[0217] Of course, in specific applications, the weight adjustment value corresponding to whether the model is fully trained or not can be set according to the actual application scenario.

[0218] After calculating the voice wake-up success rates of 1000 voice wake-up models, in addition to the weight allocation method described above, you can also use all 1000 voice wake-up models for model weighted fusion instead of screening models based on the voice wake-up success rates. At this time, you need to assign corresponding weights to each of the 1000 voice wake-up models. The process of assigning weights to the 1000 voice wake-up models is similar to the process of assigning weights to the 10 voice wake-up models mentioned above, that is, calculate the sum of the voice wake-up success rates of all models, and then compare the voice wake-up success rate of each model with the sum to obtain the corresponding weight. Then, adjust the weight value accordingly based on the pre-set sufficient training conditions and statistical analysis results to obtain the final fusion weight. The specific process will not be repeated here.

[0219] After the cloud device determines the fusion weight of the terminal-side personalized model according to the fusion weight calculation method shown above, it can perform model weighted fusion according to the model fusion weight to obtain a new cloud model.

[0220] In some other embodiments, after cross-validating the reliability of the model, the cloud device can first determine whether the structure of the personalized model uploaded by the client device has changed. If the model structure has not changed, the model weighted fusion can be performed. If the model structure has changed, the model structure is analyzed, specifically the depth and width of the model can be analyzed, and then a cloud model search is performed based on the model structure analysis, and the searched model is used as a new cloud model. In this way, when the end-side model structure changes, a new cloud model can still be generated in combination with the cloud model search technology.

[0221] It should be pointed out that the above Figure 4 In the process, if the client device determines that a change is found in the model training task, it will use a multi-task incremental learning method to incrementally train the model, which may cause changes in the end-side model structure.

[0222] In some other embodiments, after obtaining the weighted fusion model, the cloud device can perform model verification on the weighted fusion model. If the verification passes, the model can be used as a new cloud model. If the verification fails, it can be re-determined whether the structure of the personalized model on the end side has changed.

[0223] As an example and not a limitation, the following Figure 5 The schematic diagram of the end-cloud collaboration process is shown for introduction.

[0224] like Figure 5As shown, it is divided into a cloud side 51 and a terminal side 52, wherein: Figure 5 The dotted line above is the cloud side 51, and the dotted line below is the end side 52. The cloud side 51 refers to the cloud device side, and the end side 52 refers to the client device side. Specifically, the cloud side generally includes a cloud software platform and cloud devices, and the cloud software platform and cloud devices work together to provide cloud services. For example, the cloud side can refer to the server side opposite to the client. The end side generally refers to the terminal device side, for example, the end side can be a user's mobile phone side and computer side, etc. The cloud side and the end side are connected through a communication network, which is generally the Internet. For example, a mobile phone as a client and a server side are connected through Internet communication.

[0225] The end-side includes the end-side model and cloud verification data. The end-side model refers to the model that can be uploaded to the cloud after the client device has trained the model. The end-side model can be a model that has been verified or a model that has not been verified. The cloud verification data can refer to the cloud verification data generated by the client device after simulating a small amount of sample data, and the generated cloud verification data is uploaded to the cloud.

[0226] Of course, in some other embodiments, if the client device does not perform the data simulation process, the client side will not include the cloud verification data. In this case, the cross-validation data set on the cloud side may be composed of cloud data generated by the cloud.

[0227] After the cloud side obtains the end-side model uploaded by the end-side, it can store it in the end-side model pool in the cloud. Cloud data is generated based on the feature information carried by the end-side model and the cloud storage data, and then a cross-validation dataset for model reliability verification is constructed based on the cloud data generated by the cloud and the cloud verification data uploaded by the end-side. The cloud storage data here refers to the data pre-stored in the cloud. Then, the cloud device uses the constructed cross-validation dataset to perform model reliability cross-validation. At the same time, the cloud device also performs statistical analysis of the end-side model training on the end-side model. The fusion weight of the end-side model is determined by combining the reliability cross-validation results and the statistical analysis results.

[0228] After the cloud performs reliability cross-validation on the end-side model, it can determine whether the structure of the end-side model has changed. If the end-side model structure has changed, the end-side model can be structurally analyzed, specifically analyzing the depth and width of the end-side model, for example, how many convolutional layers and fully connected layers the end-side model has. Then, a cloud-side model search is performed to find a model with better model effect as the new cloud-side model. If the end-side model structure has not changed, the model weighted fusion is performed by combining the end-side model training statistical analysis and model reliability cross-validation.

[0229] In the weighted fusion of models, the need to retrain the model is determined based on pre-set conditions. The judgment conditions for retraining are mainly set based on the training tasks and model characteristics. For some simple models, the model obtained by direct weighted fusion or weighted averaging is very effective, so retraining is not required, such as random forests; while for some models with complex structures, the model obtained by direct weighted fusion or weighted averaging is less effective, so retraining is required, such as deep learning models.

[0230] The cloud side determines whether the model needs to be retrained. If retraining is required, weighted federated meta-learning is used for model training. If retraining is not required, the determined model fusion weight is used for model weighted fusion or weighted averaging. Weighted federated meta-learning means: according to the fusion weight of the terminal-side personalized model, the model parameters of the terminal-side personalized model and the corresponding fusion weight are weightedly aggregated to obtain new model parameters, and the new model parameters are used as the initial model parameters of federated meta-learning. Then, training is performed based on the initial model parameters, and the trained model is used as the new cloud model.

[0231] After the cloud side obtains the model through weighted model fusion or through cloud model search, it can perform model verification on the obtained model. Only after the verification passes, the model is used as the new cloud model. If the verification fails, it can return to the step of determining whether the terminal model structure has changed. After the cloud side obtains the new cloud model, it can send the new cloud model to the terminal side, and the terminal side can continue to perform incremental training on the new cloud model.

[0232] It should be noted that multiple rounds of model training can be performed between the client device and the cloud device. In one round of model training, the client device can perform multiple end-side model trainings.

[0233] In order to better introduce the system architecture and corresponding processes that may be involved in the embodiments of the present application, Figure 6 A schematic diagram of the model training process is shown for introduction.

[0234] like Figure 6As shown, the cloud 61 sends the model to be trained 62 to the client device 64. The models to be trained 62 received by multiple client devices 64 are all the same. After receiving the model to be trained 62 sent by the cloud 61, the client device 64 can perform a terminal-side self-learning process to obtain a trained model, perform model effect verification on the trained model, and obtain a personalized model 65 that can be uploaded to the cloud 61. After each client device 64 uses local training sample data to train the model to be trained 62, a different personalized model 65 can be obtained, and the personalized model 65 is uploaded to the cloud 61. The cloud 61 performs model fusion update based on the uploaded personalized model 65 to obtain a fused and updated cloud model 63.

[0235] It should be noted that the end-side self-learning process can be the process mentioned above where the end-side device uses the local training sample data set to train the model sent from the cloud. It can include end-side data collection, end-side data simulation generation, data validity verification, end-side incremental training, and model effect verification. For the process of how the end-side trains the model, please refer to the above Figure 2 , Figure 3 and Figure 4 The corresponding content.

[0236] The process of obtaining a new model by cloud model fusion based on personalized model 65 in cloud 61 can be seen in the above Figure 5 The corresponding content will not be repeated here.

[0237] It should be noted that the client device 64 may need to undergo multiple trainings before obtaining the personalized model 65 uploaded to the cloud 61. The reason for the multiple trainings may be that the trained model fails to pass the model verification.

[0238] So far, the above introduces the system architecture that may be involved in the embodiments of the present application, as well as possible processes of the client device and the cloud device.

[0239] The following is an exemplary introduction to the solution processes that may be involved in the embodiments of the present application.

[0240] First combine Figure 7 The model training process is introduced as shown in FIG. Figure 7 As shown, it includes a cloud side 71 and an end side 72 .

[0241] The end side 72 refers to the client device side, which may include an end side data pool, end side data preparation, end side data validity verification, end side model training and end side model.

[0242] The end-side data pool refers to the place on the client device side used to store end-side data. The end-side data pool may include historical behavior data, tracking data, and privacy data recorded by the client device. The data in the end-side data pool may be data recorded by the end after obtaining legal and clear user authorization.

[0243] Device-side data preparation refers to obtaining device-side data based on sample data in the device-side data pool. In some embodiments, device-side data with a large amount of sample data can be generated by data simulation, that is, a small amount of sample data is obtained from the device-side data pool, the small amount of sample data is analyzed, and then the device-side behavior is modeled by reinforcement learning, and then the established model is used to perform AI simulation data generation to obtain training data and cloud verification data, and then obtain device-side data. The process of preparing device-side data by data simulation can be seen in Figure 2 In other embodiments, if the client device does not perform the data simulation process, it can obtain a small amount of sample data from the end-side data pool, and obtain the end-side data after performing data preprocessing on the small amount of sample data.

[0244] End-side data validity verification refers to the verification of the end-side data obtained from the end-side data preparation. The data validity verification process can be found in the above Figure 3 Of course, in some other embodiments, the end-side data validity verification process may not be performed. In this case, after the end-side data preparation is completed, a training data set can be formed based on the end-side data, and the end-side model training can be performed using the training data set.

[0245] The client side model training refers to the client side training the client side model sent from the cloud. The client side model training process can be as follows: Figure 4 The corresponding model training process will not be described here.

[0246] After the end side 72 obtains the end side model (ie, the user personalized model) through end side model training, it can coordinate each client device to upload the trained user personalized model to the cloud side through the end cloud control platform.

[0247] The cloud side 71 includes whether the end-side model is sufficient, end-side model cross-validation, and cloud-side model weighted fusion. Among them, whether the end-side model is sufficient means that the cloud side continuously receives the end-side models uploaded by each client device, and counts the number of end-side models received; compares the number of end-side models with the preset number threshold, and if the number of end-side models is greater than the preset number threshold, it is determined to be sufficient, otherwise, it is determined to be insufficient. The above-mentioned preset number threshold can be set according to actual application needs and is not limited here.

[0248] After determining that the end-side model is sufficient, the cloud side 71 can use the constructed cloud cross-validation data set to cross-validate the end-side model. Then, the cloud model is weighted and fused according to the different weights assigned to the end-side model to obtain a new cloud model, and the new cloud model is sent to the client device again. Among them, the end-side model cross-validation process, the model fusion weight calculation process and the model weighted fusion process can be referred to the corresponding content above, and will not be repeated here.

[0249] As can be seen from the above, in some embodiments, the client device side may or may not perform a data simulation process. Correspondingly, the cloud device side may construct a cross-validation data set based on the cloud verification data generated by the client device and the data generated by the cloud, or may only construct a cross-validation data set based on the data generated by the cloud.

[0250] The following will first introduce the solution for building a cross-validation data set based on the data generated by the cloud on the cloud device side without performing data simulation on the client device side.

[0251] See also Figure 8 Another schematic diagram of the model training process is shown, Figure 8 As shown, it includes a cloud side 81 and an end side 82 .

[0252] The process on the end side 82 may include:

[0253] The client device obtains the end-side data, which may include user behavior data, and may be obtained through data logging and other methods. The end-side data may be data collected by the end-side after obtaining legal and clear user authorization. After obtaining the end-side data, the client device may perform data preprocessing on the end-side data, which may include data cleaning and other operations. Then, the client device may perform end-side data validity verification on the preprocessed data. For the specific process of end-side data validity verification, please refer to Figure 3 The corresponding related content will not be repeated here. Of course, it is also possible to directly perform end-side model training without performing data validity verification.

[0254] After the client device verifies the data validity, it obtains the training data set, and performs incremental training on the model based on the training data set. The incremental training process of the model can be found in Figure 4 The corresponding related contents will not be repeated here. It should be pointed out that since no data simulation process is performed in this embodiment, the sample size of the training data set is small. Figure 4 The corresponding training process will determine that the model is difficult to update, and the few-shot learning in meta-learning will be used to perform incremental model training.

[0255] After the client device performs incremental model training, it can perform model verification on the trained model. After the model verification passes, a personalized model is obtained, and then the personalized model is uploaded to the end-side model pool on the cloud side. The model verification on the client device side may refer to: using a verification data set to verify the model obtained after incremental training to obtain a model verification result, which may be the accuracy of the model, for example, the wake-up success rate of the voice wake-up model; if the accuracy of the model is higher than a preset threshold, the model verification is considered to have passed, otherwise, if the accuracy of the model is lower than a preset threshold, the model verification is considered to have failed. The model that has passed the model verification is uploaded to the cloud, and the model that has not passed the verification is not uploaded to the cloud.

[0256] The process of cloud side 81 may include:

[0257] Since the client device does not perform a data simulation process in this embodiment and does not generate cloud verification data for constructing a cloud cross-validation data set, the cloud device can construct a cross-validation data set based on cloud storage data. The cloud storage data refers to the cloud pre-stored data mentioned above. Of course, the cross-validation data set can also be constructed directly based on the cloud pre-stored data.

[0258] Next, the cloud device can use the cross-validation data set to perform cross-validation on the reliability of the client-side model, and count the number of training times and the amount of training data for the personalized model. Combined with the cross-validation results and statistical analysis results, the fusion weights of each personalized model are calculated. Then, the cloud device performs a weighted average of the model parameters based on the fusion weights to obtain a new cloud model. The cloud-side process can be found in Figure 5 The corresponding related content will not be repeated here.

[0259] In this embodiment, before the cloud performs model fusion, the model uploaded on the client side is first cross-validated for reliability based on the cloud cross-validation data set, and different weights are assigned to different client side models based on the reliability cross-validation results and statistical analysis results, thereby improving the robustness of the new model obtained by cloud aggregation. Furthermore, on the client side, through processes such as client side data validity verification, small sample learning, and model verification, the effect of the model uploaded from the client side to the cloud can be guaranteed as much as possible, thereby further improving the robustness of the new model obtained by cloud fusion.

[0260] Of course, in some other embodiments, the client-side model training process may not include Figure 7 The end-side data validity verification process in the Figure 7 The model validation process on the client device side in .

[0261] In other embodiments, the client device side may perform a data simulation process, and the cross-validation data set on the cloud device side is composed of the cloud validation data generated by the client device and the data generated by the cloud. Fig. 9 Another schematic diagram of the model training process is shown to introduce the scheme.

[0262] like Fig. 9 As shown, it includes a cloud side 91 and an end side 92 .

[0263] The process on the end side 92 may include:

[0264] The client device obtains end-side data, which may include user behavior data. The end-side data is data collected by the end-side after obtaining legal and clear user authorization. Next, the client device performs data simulation operations based on the obtained end-side data to generate a large amount of sample data from a small amount of sample data. After the data simulation, the generated data is obtained, and the generated data can be used as training data or as cloud verification data, so that training data and cloud verification data can be obtained, and the cloud verification data is uploaded to the cloud, and data preprocessing operations are performed on the training data. The data simulation process can be found in the specific above. Figure 2 , I will not go into details here.

[0265] After the client device performs data preprocessing on the training sample data generated by data simulation, it can perform data validity verification on the preprocessed training data to obtain a training data set. The data validity verification process can be specifically referred to in the above Figure 3 , I will not go into details here.

[0266] After obtaining the training data set, the client device can perform incremental training on the model based on the training data set. The incremental training process of the model on the client side can be seen in the above Figure 4 It should be noted that this embodiment performs a data simulation process, so the sample size of the training data set is relatively large. Figure 4 In the corresponding model training process, if it can be determined that the model is easy to update, the model incremental training is performed using knowledge distillation and convolutional layer solidification.

[0267] After incremental model training, the client device can perform model verification on the trained model. After the model verification passes, a personalized model can be obtained and then uploaded to the terminal model pool on the cloud side.

[0268] The cloud-side process may include:

[0269] In this embodiment, since the end side performs a data simulation process, the cloud side will receive the cloud verification data uploaded by the end side, and the cloud verification data can be used to construct a cloud cross-validation data set. At this time, the cloud device can construct a cross-validation data set based on the cloud verification data and the cloud storage data, where the cloud storage data refers to the cloud pre-stored data mentioned above; it can also construct a cross-validation data set based on the cloud verification data and the cloud generated data, where the cloud generated data refers to the data generated by the cloud based on the characteristic parameter information of the end-side model and the cloud pre-stored data. Fig. 9 Only one case is shown.

[0270] After the cloud device constructs the cross-validation data set, it uses the cross-validation data set to perform reliability cross-validation on the client-side model, and counts the number of training times and the amount of training data of the client-side personalized model. It then calculates the fusion weight of the client-side model by combining the cross-validation and statistical analysis results. Then, the model parameters are weighted averaged according to the fusion weight to obtain the new cloud model. The fusion weight calculation process on the cloud side and the model weighted fusion process can be found in the above section. Figure 5 , I will not go into details here.

[0271] To better introduce Fig. 9 The corresponding embodiment is as follows Fig.10 Another schematic diagram of model training is shown for introduction.

[0272] like Fig.10 As shown, the end-side data generates a large amount of sample data through data simulation. Of the data generated by the data simulation, part of it is uploaded to the cloud for cloud verification of data composition, and the other part of it is used for end-side training data composition. After the end-side forms the training sample data through the data simulation process, the data validity of the training sample data set can be verified (not shown in the figure), and then the end-side model incremental training can be performed. During the process of incremental training of the end-side model, due to the data simulation process, the amount of training sample data is large, so it can be determined that the model is easy to update. If the model is easy to update, the convolution layer solidification and knowledge distillation method are used to perform incremental training of the end-side model. During the incremental training of the end-side model, the distillation loss (Distilation Loss) and the regression loss (Regression Loss) can be obtained. The weight of the distillation loss is λ, and the weight of the regression loss is 1-λ. Based on the distillation loss and the regression loss, and the corresponding weights, the total loss (Total loss) can be obtained.

[0273] After the client obtains the trained model, it can verify the model effect. After the model effect verification is passed, the trained model can be uploaded to the cloud.

[0274] After the cloud obtains the model uploaded by the client, it uses the cross-validation dataset to cross-validate the model and calculates the fusion weights of each model based on the cross-validation results. After the fusion weights of each model are obtained through weight calculation, weighted federated meta-learning is used to perform weighted fusion of the models to obtain the fused and updated cloud model.

[0275] It should be noted that this embodiment is different from the above Figure 8 The corresponding embodiments are different in that the present embodiment performs a terminal-side data simulation process, and the cross-validation data set in the cloud includes the cloud-side verification data generated by the terminal-side data simulation process.

[0276] Based on this, before model fusion is performed on the cloud side, this embodiment first performs reliability cross-validation on the model uploaded on the client side based on the cloud cross-validation data set, and assigns different weights to different client side models based on the reliability cross-validation results and statistical analysis results, thereby improving the robustness of the new model obtained by cloud aggregation. Furthermore, on the client side, through processes such as client side data validity verification and model verification, the effect of the model uploaded from the client side to the cloud can be guaranteed as much as possible, thereby further improving the robustness of the new model obtained by cloud fusion.

[0277] In addition, this embodiment expands the training data set on the end side through the end-side data simulation process, improves the training efficiency of the end-side model, and makes the trained end-side model more personalized. In addition, the cross-validation data set includes a cloud-based validation data set, which enriches the scenarios and complexity of model validation and can further improve the robustness of the new model obtained by cloud-based fusion.

[0278] It should be noted that the above Figure 8 and Fig. 9 The scheme shown in the corresponding embodiment is only an example. In a specific application, some steps can be added or reduced accordingly, or some steps can be replaced by other implementation methods to obtain a new scheme, but these schemes obtained by replacement or recombination all fall within the protection scope of the embodiments of the present application.

[0279] For example, Figure 8 and Fig. 9 The on-device model training process can be as follows: Figure 4 Corresponding incremental training process. In some other embodiments, Figure 8 and Fig. 9 The training process in can also use the client-side training process in existing federated learning. Figure 4 The corresponding model incremental training process selects the optimal model incremental training method based on the model training task and model type, which can further improve the end-side model training efficiency and model training effect. Figure 4The incremental model training process may not include the judgment process of whether the end-side training task has changed, etc.

[0280] For example, Figure 8 and Fig. 9 In the cloud-side process, it is not necessary to perform the model structure change judgment process, nor to verify the model after the weighted fusion of the model. However, adding the model structure change judgment process can still generate a new cloud model in combination with the cloud model search when the end-side model structure changes. Adding the model verification process after the weighted fusion of the model can ensure the effect of the new cloud model as much as possible.

[0281] In some other embodiments, the end-side process may be as follows: Figure 8 or Fig. 9 The process shown in the figure, but the cloud-side process can use the existing federated learning process instead of Figure 8 and Fig. 9 The corresponding cloud-side process. However, if only Figure 8 or Fig. 9 The end-side process is the end-side process in the existing federated learning, while the cloud-side process is the cloud-side process in the existing federated learning. At this time, the end-side performs the data validity verification process, the end-side model verification process, and Figure 4 The corresponding model incremental training process can ensure the effect of the end-side model uploaded to the cloud as much as possible, and can still improve the robustness of the new model obtained by cloud fusion. However, the cloud side performs reliability cross-validation on the end-side model, and assigns different fusion weights to different end-side models based on the reliability cross-validation results and statistical analysis results, which can further improve the robustness of the new model fused in the cloud.

[0282] After introducing the system architecture provided by the embodiments of the present application, and the related processes that may be involved in the client device and the cloud device, the application scenarios that may be involved in the embodiments of the present application are introduced below.

[0283] The application scenarios of the embodiments of the present application are wide-ranging, and there are no restrictions on specific fields and algorithms. As an example and not a limitation, the embodiments of the present application can be applied to biometric recognition, such as fingerprint recognition, voiceprint recognition, and Face ID unlocking; in some embodiments, combined with 5G (5th-Generation, fifth-generation mobile communication technology) and IoT (Internet of Things) technology, it can also be applied to multi-terminal data to build an ecosystem and improve AI capabilities. For example, scenarios such as power consumption control, charging control, and voice wake-up. The above shows the relevant content under the voice wake-up scenario, and the processes of other scenarios and the types of voice wake-up scenarios will not be repeated here.

[0284] In the embodiment of the present application, the model prediction accuracy will become higher and higher through continuous end-side self-learning. And through end-cloud collaboration, a more robust model can be obtained, so that the model can also achieve accurate prediction in some scenarios with less sample data. Fig.11 A schematic diagram of the regression prediction effect is shown for introduction. Fig.11 It can be based on the above Figure 8 The solution of the corresponding embodiment.

[0285] Figure 8 In the corresponding embodiment, the data simulation process is not performed on the client device side, and the amount of training sample data is small. At this time, the end-side self-learning process may include: end-side data collection, end-side data validity verification, end-side model incremental training and model effect verification. Since the amount of training sample data is small, it can be determined that the model is difficult to update. If the model is difficult to update, the model incremental training is performed using convolutional layer updates and meta-learning. The end-cloud collaborative process refers to the process in which the cloud and the end jointly train the model, that is, the cloud sends the model to be trained to the client, the client device trains the model to be trained through end-side self-learning, and uploads the trained model to the cloud. The cloud merges and updates the received models to obtain a new cloud model. Fig.11 This is a technical effect diagram using the regression prediction problem as an example. In the regression prediction problem, there is a difference between the model prediction value and the true value.

[0286] like Fig.11 As shown in the figure, the striped rectangle is the true value, and the blank rectangle and the dotted rectangle are the regression prediction values. Among them, the blank rectangle is the regression prediction value before the end-to-end collaboration, and the dotted rectangle is the regression prediction value after the end-to-cloud collaboration. The height of the bar represents the size of its value. The difference between the true value and the regression prediction value can be used to know the degree of difference between the regression prediction value and the true value. Fig.11 The horizontal axis is the time axis, and the figure shows time nodes such as July 1, August 1, September 1, October 1, November 1 and December 1.

[0287] On August 1 and September 1, two end-side self-learning processes were performed, that is, the client device performed two incremental trainings on the to-be-trained model sent from the cloud. An end-side self-learning process was also performed around October 1. It can be seen that three end-side self-learning processes were performed between August 1 and November 1. After each end-side self-learning, the difference in column height between the true value and the regression prediction value will gradually decrease, that is, the regression prediction value will get closer and closer to the true value. In other words, through continuous end-side self-learning, the regression prediction accuracy will become higher and higher. However, there are also times when the regression prediction value and the true value are quite different, that is, the model has low robustness.

[0288] On November 1, a terminal-cloud collaboration process was carried out, that is, multiple client devices were controlled to upload the trained model to the cloud. The cloud cross-validated the received model for reliability, assigned different weights according to the reliability cross-validation results, and finally performed weighted fusion of the models according to the assigned weights to obtain a new cloud model. The cloud then sends the fused new model to the terminal side. The client device uses the fused updated model for regression prediction. The difference between the true value and the regression prediction value has been relatively stable and will not suddenly become large. In other words, a more robust model can be obtained through the terminal-cloud collaboration process, and the model can also achieve accurate prediction in scenarios with small amounts of data.

[0289] The following will be combined Fig.12 The picture shown is a recommended schematic diagram for introduction. Fig.12 is based on Fig. 9 and Fig.10 The technical solution of the corresponding embodiment.

[0290] exist Fig. 9 and Fig.10 In the corresponding embodiment, the end side performs a data simulation process. At this time, the amount of training sample data on the end side is large. The end side self-learning process may include end side data collection, data simulation, data validity verification, end side model incremental training, and model effect verification.

[0291] like Fig.12 As shown in the figure, there are four pictures in the mobile phone on the far left and in the middle, namely, picture 221, picture 222, picture 223, and picture 224. Before the end-side self-learning is performed, the picture recommended by the system is picture 222, and the picture selected by the user is picture 221. At this time, the user's picture category label is obtained through the user's behavior of selecting pictures, that is, the user prefers the smiling picture of the category corresponding to picture 221. After obtaining the user's operation data, a large amount of end-side operation data is generated through the data simulation process. The end-side operation data is used to form training sample data, and then the training sample data is used to train the picture recommendation model.

[0292] After the end-side self-learning, the system recommends picture 221 to the user through the picture recommendation model. In other words, after the end-side self-learning, a picture recommendation model that can better reflect the user's personalization can be obtained.

[0293] The client uploads the trained model to the cloud, and the cloud aggregates a large number of client-side personalized models into a new image recommendation model. Use the image recommendation model after the client-cloud collaboration or the updated image recommendation model after the cloud fusion to recommend images. For example, the rightmost mobile phone in the figure includes four images, namely, image 225, image 226, image 227, and image 228. After the client-cloud collaboration, the system recommends image 225 to the user.

[0294] It should be noted that picture 225 is a smiling picture from a side angle, while picture 221 is a smiling picture from a front angle. Pictures 225 and 221 are intended to illustrate that the model obtained by end-cloud collaboration can also ensure the effect in complex scenarios. Fig.12 This indicates that the end-cloud collaborative model can recommend smiling pictures that meet user preferences in scenarios with changing angles.

[0295] The following will be combined Fig.13 The schematic diagram of the interaction between the terminal device and the cloud device is shown to introduce the model training process.

[0296] like Fig.13 As shown, the interaction process may include the following steps:

[0297] Step S1301: The cloud device sends the model to be trained to the client device.

[0298] Step S1302: The terminal-side device performs data simulation based on the small amount of collected samples to generate a large number of training samples.

[0299] It can be understood that the data simulation process of the terminal side device can refer to the corresponding content above and will not be repeated here.

[0300] Step S1303: The client device uploads the cloud verification data generated by data simulation to the cloud device.

[0301] Step S1304: The terminal-side device constructs a training sample data set according to the training samples generated by the data simulation, and performs data validity verification on the training sample data set to obtain a verified training sample data set.

[0302] It is understandable that the data validity verification process can refer to the corresponding content above and will not be repeated here.

[0303] Step S1305: The end-side device uses the verified training sample data set to train the model to be trained to obtain the end-side model, and performs model verification on the end-side model to obtain a verified end-side personalized model.

[0304] Step S1306: The terminal-side device uploads the terminal-side personalized model to the cloud device.

[0305] Step S1307: The cloud device constructs a cloud cross-validation data set based on the cloud verification data uploaded by the client device.

[0306] Step S1308: The cloud device uses the cross-validation data set to perform reliability cross-validation on the client-side personalized model to obtain a cross-validation result.

[0307] Step S1309: The cloud device performs statistical analysis on the terminal-side personalized model to obtain a statistical analysis result indicating whether the terminal-side personalized model is fully trained.

[0308] Step S1310: The cloud device assigns a fusion weight to each terminal-side personalized model based on the cross-validation result and the statistical analysis result.

[0309] Step S1311: The cloud device performs model weighted fusion according to the fusion weight and the terminal-side personalized model to obtain an updated cloud model.

[0310] It should be noted that the same or similar contents in this embodiment and the above embodiments can be referred to each other and will not be described in detail here.

[0311] The embodiment of the present application also provides an electronic device, which includes a memory and a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the model training method in any of the above embodiments can be implemented.

[0312] In the embodiment of the present application, the first electronic device is a cloud device, which is generally a server. The second electronic device is a terminal device, which can be but not limited to a mobile phone, a tablet computer, a laptop computer, a wearable device, a car machine or a car-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook personal digital assistant (PDA), etc.

[0313] By way of example and not limitation, Fig.14As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0314] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0315] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0316] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0317] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0318] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0319] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0320] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170.

[0321] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. Both the I2S interface and the PCM interface can be used for audio communication.

[0322] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function.

[0323] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0324] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0325] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0326] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0327] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0328] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0329] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0330] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0331] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0332] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0333] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0334] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0335] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0336] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0337] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0338] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0339] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0340] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0341] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0342] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0343] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0344] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0345] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0346] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0347] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0348] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0349] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0350] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0351] The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194.

[0352] There are many types, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. Capacitive pressure sensors can be at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0353] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0354] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0355] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0356] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0357] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0358] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0359] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0360] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0361] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0362] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0363] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0364] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0365] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0366] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0367] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0368] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.

[0369] As an example but not limitation, the processor 110 in the electronic device 100 can call and execute the model training program in the internal memory 121 to implement the steps of the model training method in any of the above embodiments.

[0370] For example, if the electronic device is a cloud server, the internal memory 121 of the cloud server stores a model training program and various models. At this time, the models stored in the internal memory 121 of the cloud storage server may include but are not limited to: models to be trained, personalized models uploaded by the terminal side, and models obtained by cloud fusion update.

[0371] In a specific application, after receiving the end-side personalized model uploaded by the end-side device and the cloud verification data uploaded by the end-side device, the cloud server can store both the end-side personalized model and the cloud verification data in the internal memory. In the subsequent process of constructing the cloud cross-validation data set, the processor 110 of the cloud server can execute the model training program to read the cloud verification data from the internal memory 121, and implement the step of constructing the cloud verification data according to the cloud verification data. In the process of cross-validation using the cross-validation data set, the processor 110 of the cloud server can execute the model training program stored in the internal memory 121 to cross-validate each end-side personalized model using the cross-validation data set, obtain the cross-validation result, and store the cross-validation result in the internal memory 121. Then, the processor 110 of the cloud server can also execute the model training program stored in the internal memory 121 to assign fusion weights to each different end-side personalized model according to the cross-validation result, and then perform weighted fusion according to the fusion weight of each end-side personalized model to obtain a cloud-based fusion-updated model, and store the cloud-based fusion-updated model in the internal memory 121.

[0372] Of course, in some other embodiments, the function of the internal memory 121 may also be implemented by an external memory. In this case, the processor 100 of the cloud server reads the data on the external memory through the external storage interface 120 .

[0373] For another example, if the electronic device is a terminal device, such as a mobile phone, a tablet computer, or a wearable device, the terminal device includes a processor 110 and an internal memory 121. The internal memory may store: a model training program, a model to be trained sent by a cloud server, collected training sample data, and a trained model, etc.

[0374] After receiving the model to be trained sent by the cloud server, the end-side device can write the model to be trained into the internal memory 121. Similarly, after the end-side device collects the training sample data by means of data dotting, etc., the collected training sample data can be written into the internal memory 121. The processor 100 of the end-side device executes the model training program stored in the internal memory 121 to implement the data simulation process, the data validity verification process, the end-side model incremental training process and the model verification process. The processes of data simulation, data validity verification, end-side model incremental training and model verification can all be referred to the corresponding embodiments above, which will not be repeated here.

[0375] Taking the data validity verification process as an example, the processor 100 of the terminal device executes a model training program to read the terminal data from the internal memory 121, and then classifies the terminal data to determine valid data, redundant data and noise data; stores the redundant data in the data cache area, and stores the noise data in the internal memory 121 after performing noise repair.

[0376] In addition, the embodiment of the present application further provides a model training device, which includes a processing module and a storage module. The storage module can be used to store the model training program and the model, and the processing module can be used to execute the model training program stored on the storage module to implement the model training method in any of the above embodiments.

[0377] As an example but not limitation, the storage module may be the internal memory 121 mentioned above, and the processing module may be the processor 110 mentioned above.

[0378] In some embodiments, the model training device can be a cloud server or a device integrated in a cloud server. At this time, the processing module of the model training device can execute the model training program on the storage module to implement the cloud cross-validation data set construction process, model cross-validation process, model fusion weight allocation process and model weighted fusion process, etc.

[0379] The storage module of the model training device stores a model training program and various models. The models stored in the storage module may include but are not limited to: a model to be trained, a personalized model uploaded by the terminal side, and a model obtained by fusion and update in the cloud.

[0380] For example, after receiving the end-side personalized model and the cloud verification data, the model training device can store both the end-side personalized model and the cloud verification data in the storage module. In the subsequent process of constructing the cloud cross-validation data set, the processing module of the model training device can execute the model training program to read the cloud verification data from the storage module to implement the step of constructing the cloud verification data according to the cloud verification data. In the process of cross-validation using the cross-validation data set, the processing module of the model training device can execute the model training program stored on the storage module to cross-validate each end-side personalized model using the cross-validation data set, obtain the cross-validation result, and store the cross-validation result in the internal storage module. In addition, the processing module of the model training device can also execute the model training program stored on the storage module to assign fusion weights to each different end-side personalized model according to the cross-validation result, and then perform weighted fusion according to the fusion weight of each end-side personalized model to obtain a cloud-based fusion-updated model, and store the cloud-based fusion-updated model in the storage module.

[0381] In other embodiments, the model training device may be a terminal device or a device integrated on the terminal device. In this case, the processing module of the model training device executes the model training program on the storage module to implement the above data simulation process, data validity verification process, terminal model incremental training process and model verification process.

[0382] The internal storage module of the model training device may include but is not limited to: a model training program, a model to be trained sent from the cloud, collected training sample data, and a trained model, etc.

[0383] The model training device can write the model to be trained into the storage module after receiving the model to be trained sent by the cloud server. Similarly, after the model training device collects the training sample data by means of data dotting, etc., the collected training sample data can be written into the storage module. The processing module of the model training device can implement the data simulation process, data validity verification process, end-side model incremental training process and model verification process by executing the model training program stored on the storage module. The processes of data simulation, data validity verification, end-side model incremental training and model verification can all be found in the corresponding embodiments above and will not be repeated here.

[0384] Taking the data validity verification process as an example, the processing module of the model training device executes the model training program to read the end-side data from the storage module, and then classifies the end-side data to determine the valid data, redundant data and noise data; stores the redundant data in the storage module, and stores the noise data in the storage module after performing noise repair.

[0385] It should be noted that, for the same or similar points here as those in the above embodiments, reference can be made to the corresponding contents of the above embodiments, which will not be repeated here.

[0386] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the model training method in the above-mentioned embodiments can be implemented.

[0387] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps of the model training method in the above-mentioned embodiments when executing the computer program product.

[0388] The embodiment of the present application also provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the model training method in any of the above embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0389] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0390] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. References to "one embodiment" or "some embodiments" etc. described in the present application specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0391] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0392] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method based on federated learning, characterized in that: The method comprises: The first electronic device sends a model to be trained to the second electronic device; The first electronic device receives one or more models to be fused sent by the second electronic device, where the model to be fused is a model obtained by the second electronic device training the model to be trained according to a local training sample data set; the model to be trained is a biometric recognition model, a voice wake-up model, a charging control model, a news recommendation model, a power consumption control model or a picture recommendation model; The first electronic device uses a cross-validation data set to cross-validate the model to be fused to obtain a cross-validation result; The first electronic device assigns a fusion weight to one or more of the models to be fused according to the cross-validation result; The first electronic device performs model weighted fusion on the models to be fused according to the fusion weight of each model to be fused to obtain an updated cloud model; After the first electronic device receives one or more models to be fused sent by the second electronic device, the method further includes: The first electronic device determines the training result of each of the models to be fused according to a preset sufficient training condition, wherein the training result includes sufficient training and insufficient training; The first electronic device assigns fusion weights to each model to be fused according to the cross-validation result, including: The first electronic device determines a first weight of each of the models to be fused according to the cross-validation result; The first electronic device adjusts the first weight according to the training result for each of the models to be fused to obtain a second weight; The first electronic device uses the second weight of each of the models to be fused as the fusion weight.

2. The method according to claim 1, characterized in that The first electronic device determines the training result of each of the to-be-fused models according to a preset sufficient training condition, including: If the number of terminal-side training times of the model to be fused is greater than a preset number threshold, and the amount of terminal-side data of the model to be fused is greater than a preset threshold, the first electronic device determines that the training result of the model to be fused is sufficient training; If the number of terminal-side trainings of the model to be fused is less than the preset number threshold, and / or the amount of terminal-side data of the model to be fused is less than the preset threshold, the first electronic device determines that the training result of the model to be fused is insufficient training; The number of terminal-side training times is the number of training times of the model to be trained on the second electronic device, and the amount of terminal-side data is the amount of training sample data locally on the second electronic device.

3. The method according to claim 1, characterized in that The first electronic device adjusts the first weight according to the training result to obtain a second weight, including: If the training result of the model to be fused is sufficient training, the first electronic device adds the first weight and a preset value to obtain the second weight; If the training result of the model to be fused is insufficient training, the first electronic device subtracts the first weight from the preset value to obtain the second weight.

4. The method according to claim 1, characterized in that The first electronic device determines a first weight of each of the to-be-fused models according to the cross-validation result, including: The first electronic device adds the cross-validation results of each of the models to be fused to obtain a sum of the cross-validation results; The first electronic device uses the ratio between the cross-validation result of each model to be fused and the sum of the cross-validation results as the first weight of the model to be fused.

5. The method according to claim 1, characterized in that The first electronic device uses a cross-validation data set to cross-validate the model to be fused to obtain a cross-validation result, including: The first electronic device divides the cross-validation data set into sub-cross-validation data sets; The first electronic device uses each of the sub-cross-validation data sets to validate each of the models to be fused, respectively, to obtain a sub-validation result, wherein one sub-cross-validation data set corresponds to one sub-validation result; The first electronic device obtains a cross-validation result of each of the models to be fused according to the sub-validation result of each of the models to be fused.

6. The method according to claim 1, characterized in that The method further comprises: The first electronic device obtains cloud verification data uploaded by the second electronic device, where the cloud verification data is data generated by the second electronic device through data simulation of training samples; The first electronic device constructs the cross-validation data set based on the cloud verification data and cloud data, where the cloud data is data stored locally in the cloud.

7. The method according to claim 1, characterized in that After the first electronic device uses the cross-validation data set to cross-validate the model to be fused and obtains a cross-validation result, the method further includes: If the model structure of the model to be fused is different from the model structure of the model to be fused in the previous round of training, the first electronic device determines a target cloud model from pre-stored cloud models according to the model structure of the model to be fused; The first electronic device performs model verification on the target cloud model; If the verification is successful, the first electronic device uses the target cloud model as an updated cloud model; If the model structure of the model to be fused is the same as the model structure of the model to be fused in the previous round of training, the first electronic device enters the step of assigning fusion weights to each model to be fused according to the cross-validation result.

8. The method according to any one of claims 1 to 7, characterized in that The first electronic device performs model weighted fusion on the models to be fused according to the fusion weight of each model to be fused to obtain an updated cloud model, including: The first electronic device determines whether the model to be fused needs to be retrained according to a preset retraining condition; If retraining is required, the first electronic device trains the model to be fused according to the fusion weight by weighted federated meta-learning to obtain a fusion model; performs model verification on the fusion model, and if the verification passes, uses the fusion model as the updated cloud model; If retraining is not required, the first electronic device performs weighted fusion on the models to be fused according to the fusion weights to obtain a fusion model; performs model verification on the fusion model, and if the verification passes, uses the fusion model as the updated cloud model.

9. A model training method based on federated learning, characterized in that: The method comprises: The second electronic device receives the model to be trained sent by the first electronic device; the model to be trained is a biometric recognition model, a voice wake-up model, a charging control model, a news recommendation model, a power consumption control model or a picture recommendation model; The second electronic device verifies the data validity of the training sample data set to obtain a verified training sample data set; The second electronic device performs incremental training on the model to be trained according to the verified training sample data set to obtain a terminal-side model; The second electronic device performs model verification on the terminal side model; The second electronic device transmits the terminal-side model that has passed the model verification to the first electronic device as a model to be fused; The second electronic device performs incremental training on the model to be trained according to the verified training sample data set to obtain a terminal-side model, including: If the type of the model training task is different from the type of the model training task in the previous round of training, the second electronic device performs incremental training on the model to be trained by multi-task incremental learning according to the verified training sample data set to obtain the terminal side model; If the type of the model training task is the same as the type of the model training task in the previous round of training, and the base model type of the model to be trained is a machine learning model, the second electronic device performs incremental training on the model to be trained by machine incremental learning according to the verified training sample data set to obtain the terminal side model; If the type of the model training task is the same as the type of the model training task in the previous round of training, and the base model type of the model to be trained is a non-machine learning model, the second electronic device performs incremental training on the model to be trained by means of neural network incremental learning according to the verified training sample data set to obtain the terminal side model.

10. The method according to claim 9, characterized in that The second electronic device verifies the data validity of the training sample data set to obtain a verified training sample data set, including: The second electronic device performs data classification on the training sample data set to determine first valid data, redundant data, and noise data in the training sample data; The second electronic device removes the redundant data; The second electronic device performs noise data repair on the noise data to obtain second valid data; The second electronic device composes the verified training sample data set based on the first valid data and the second valid data.

11. The method according to claim 9, characterized in that The second electronic device performs incremental training on the model to be trained by means of neural network incremental learning according to the verified training sample data set to obtain the terminal side model, including: If the model to be trained meets the first condition, the second electronic device performs incremental training on the model to be trained by small sample learning and convolution layer updating according to the verified training sample data set to obtain the terminal side model; If the model to be trained meets the second type of conditions, the second electronic device performs incremental training on the model to be trained according to the verified training sample data set by means of knowledge distillation and convolutional layer solidification to obtain the terminal side model.

12. The method according to claim 9, characterized in that The method further comprises: The second electronic device acquires terminal-side sample data; The second electronic device performs data simulation according to the terminal-side sample data to generate training data and cloud verification data for constructing a cloud cross-validation data set; The second electronic device uploads the cloud verification data to the first electronic device; The second electronic device constructs the training sample data set according to the terminal-side sample data and the training data.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 or 9 to 12 is implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 or 9 to 12 is implemented.

Images