A method, device and medium for distributed training of federated learning of IoT devices

Abstract

The present invention discloses a method, apparatus, and medium for distributed training of federated learning for IoT devices. The method comprises: sending a plurality of prediction models to a central server for aggregation to obtain a global model; wherein the plurality of prediction models are established by introducing a Laplace noise set and based on a decision tree set and model prediction functions of a plurality of IoT devices; obtaining the global model, and training the global model using a gradient descent algorithm so that the parameters of the global model fit the local data set. The present invention proposes a method, apparatus, and medium for distributed training of federated learning for IoT devices. By utilizing the device's own data for local model training, the method reduces dependence on the central server, improves model optimization efficiency, reduces data transmission requirements, effectively improves the overall efficiency of the system, and can solve the problem of difficulty in achieving local optimization processing of data in federated learning, thereby improving data processing efficiency and security.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a federated learning distributed training method, device, and medium for Internet of Things devices. Background Art

[0002] The rapid development and widespread application of information technology has led to an increasing amount of data being converted into digital form for processing and storage. Machine learning techniques can automatically mine the hidden treasures within data. Machine learning models trained on large amounts of data have been applied in various scenarios and are profoundly changing our world. Examples include precision medicine, clinical auxiliary diagnosis, new drug development, portrait recognition, voiceprint recognition, personalized recommendation algorithms, and multimodal learning using images, speech, and natural language. In these applications, model accuracy and generalization capabilities are crucial, and these rely on machines learning from large amounts of data. Federated learning is a distributed machine learning framework that allows multiple participants to collaboratively train a shared model while maintaining the privacy of their respective data.

[0003] However, federated learning faces severe communication challenges, especially for the transmission of model parameters, which involves large amounts of data and low transmission efficiency. Moreover, since the devices of participants may have different computing and storage capabilities, it is difficult to effectively compress the model while maintaining model performance. In addition, federated learning involves interactions between multiple participants, and the complexity of this multi-party interaction increases the risk of data leakage. Summary of the Invention

[0004] The present invention provides a federated learning distributed training method, apparatus, and medium for IoT devices to solve the problem of difficulty in achieving local optimized processing of data in federated learning and to improve data processing efficiency and security.

[0005] Sending multiple prediction models to a central server for aggregation to obtain a global model; wherein the multiple prediction models are established by introducing a Laplace noise set and based on a decision tree set and model prediction functions of the multiple IoT devices; the decision tree set is established by fitting corresponding decision trees with a residual set as the first target variable; the residual set is established based on the target variables of the multiple IoT devices and the model prediction results;

[0006] The global model is obtained and trained using a gradient descent algorithm so that the parameters of the global model fit the local data set.

[0007] By calculating and analyzing the residual set, the present invention can intuitively understand the accuracy of model predictions in multiple IoT devices and determine whether the model has deviations or deficiencies. By fitting the residual set as the target variable to a decision tree, the prediction error of the model on the original data can be further optimized, thereby compensating for the deficiencies of the original model. By integrating multiple such decision trees, the overall prediction error can be gradually reduced and the prediction accuracy of the model can be improved. Because Laplace noise is additive and scalable, adding Laplace noise to the parameter update amount of the model can effectively protect personal privacy. Therefore, the multiple prediction models established in this way have high flexibility and scalability. Since each IoT device trains the model locally, the direct transmission and centralized storage of the original data are avoided, thereby reducing the risk of data leakage. In addition, data localization processing means that only model updates, rather than the entire data set of multiple IoT devices, need to be transmitted to the central server, which greatly reduces the amount of data transmission, reduces the demand for network bandwidth, and accelerates the transmission speed of model updates.

[0008] Compared with the existing technology, the present invention enables IoT devices to use their own data for local model training while ensuring data security and privacy. This local model update method reduces dependence on central servers, which not only improves model optimization efficiency, but also reduces the demand for data transmission, effectively improving the overall efficiency and scalability of the system. Therefore, it can solve the problem of difficulty in achieving local optimization processing of data in federated learning and improve data processing efficiency and security.

[0009] As a preferred solution, several prediction models are sent to a central server for aggregation to obtain a global model, specifically:

[0010] sending the plurality of prediction models to the central server;

[0011] Controlling the central server to dynamically adjust the weights of the plurality of prediction models according to the data volume, model quality, and preset parameters of the plurality of IoT devices to obtain a plurality of weighted models;

[0012] The weighted models are aggregated into the global model by performing weighted average calculation on the weighted models.

[0013] This preferred solution integrates multiple prediction models, and the global model can combine the advantages of each model, reduce the deviation and overfitting problems that may occur in a single model, thereby improving the overall generalization ability of the model, and enabling the global model to make more accurate and stable predictions when faced with new, unseen data. Dynamically adjusting the weights of each model based on the data volume and model quality of the IoT device can ensure that resources can be more reasonably allocated to models with better performance. By performing a weighted average calculation on the weighted models, the global model can output a more comprehensive and accurate prediction result. This result integrates the prediction information of multiple models and can better reflect the actual situation of the data, thereby improving the prediction accuracy.

[0014] As a preferred solution, the residual set is established based on the target variables and model prediction results of the plurality of IoT devices, specifically:

[0015] Setting the initial model prediction function in the plurality of IoT devices to the average value or category proportion of the target variable;

[0016] Based on the model prediction result of the initial model prediction function, the residual set is calculated according to the target variable and the model prediction functions in the plurality of IoT devices.

[0017] This preferred solution sets the prediction result of the initial model prediction function to the mean value or category proportion of the target variable. This effectively establishes a baseline or starting point for the entire prediction process. This baseline reflects a basic estimate of the target variable without considering any specific device or data characteristics. Based on the initial prediction result, the target variable, and the model prediction function, a residual set is calculated. This residual set reflects the model's prediction error on the current data.

[0018] As a preferred solution, the several prediction models are established by introducing a Laplace noise set and based on a decision tree set and the model prediction functions of the several IoT devices, specifically:

[0019] Controlling the step size of the model parameter update by the learning rate, and establishing a plurality of initial prediction models by combining the decision tree set and the model prediction functions of the plurality of IoT devices;

[0020] The Laplace noise set is added to the parameter update amount of the several initial prediction models to obtain the several prediction models.

[0021] This preferred solution effectively controls the model's learning progress and convergence speed by setting an appropriate learning rate. Combining a set of decision trees with the model prediction function of IoT devices leverages the strengths of both to build a more accurate and robust prediction model, better adapted to the complex and ever-changing IoT environment. Furthermore, Laplace noise, a random noise with a specific probability distribution, can be added to the parameter updates of the prediction model to significantly improve the model's robustness and privacy protection capabilities.

[0022] As a preferred solution, the Laplace noise set is specifically:

[0023] Under the condition that structural parameters other than the data sets are ensured to remain consistent, using the preset data sets to update the first initial prediction model among the plurality of initial prediction models, and obtain updated model outputs on different data sets;

[0024] The maximum value of the difference in the output of the model is defined as sensitivity;

[0025] Calculate a scale parameter based on the sensitivity and a preset privacy budget;

[0026] Based on the scale parameter, according to the probability density function of the Laplace distribution, a random number generator is used to generate a noise value that conforms to the Laplace distribution to obtain a first Laplace noise;

[0027] The plurality of initial prediction models are traversed to obtain the Laplace noise set consisting of a plurality of first Laplace noises.

[0028] In this preferred solution, a greater sensitivity indicates a more sensitive model to changes in individual data points, and the risk of privacy leakage increases accordingly. Therefore, the maximum difference in model output is defined as sensitivity. This quantifies the model's sensitivity to data changes and provides a basis for subsequent privacy protection measures. The scale parameter is calculated based on sensitivity and privacy budget, and it determines the amount of noise added to the model parameter updates. Adding Laplace noise to the model parameter updates can smooth the parameter update process and reduce the risk of overfitting. Because Laplace noise is additive and scalable, it can also effectively protect individual privacy.

[0029] The present invention also provides a federated learning distributed training device for IoT devices, which is applicable to several IoT devices and includes an aggregation module and a fitting module;

[0030] The aggregation module is configured to send the plurality of prediction models to a central server for aggregation to obtain a global model; wherein the plurality of prediction models are established by introducing a Laplace noise set and based on a decision tree set and the model prediction functions of the plurality of IoT devices; the decision tree set is established by fitting corresponding decision trees with a residual set as the first target variable; and the residual set is established based on the target variables of the plurality of IoT devices and the model prediction results;

[0031] The fitting module is used to obtain the global model and train the global model through a gradient descent algorithm so that the parameters of the global model fit the local data set.

[0032] As a preferred solution, the aggregation module includes a sending unit, a weighting unit and an aggregation unit;

[0033] Wherein, the sending unit is used to send the plurality of prediction models to the central server;

[0034] The weighting unit is used to control the central server to dynamically adjust the weights of the plurality of prediction models according to the data volume, model quality and preset parameters of the plurality of IoT devices to obtain a plurality of weighted models;

[0035] The aggregation unit is used to aggregate the multiple weighted models into the global model by performing weighted average calculation on the multiple weighted models.

[0036] As a preferred solution, the aggregation module includes a setting unit and a residual unit;

[0037] The setting unit is configured to set the initial model prediction function in the plurality of IoT devices to the average value or category ratio of the target variable;

[0038] The residual unit is used to calculate the residual set based on the model prediction result of the initial model prediction function, according to the target variable and the model prediction functions in the multiple Internet of Things devices.

[0039] As a preferred solution, the aggregation module includes an initial unit and a model unit;

[0040] The initialization unit is used to control the step size of the model parameter update through the learning rate, and to establish a plurality of initial prediction models in combination with the decision tree set and the model prediction functions of the plurality of IoT devices;

[0041] The model unit is used to add the Laplace noise set to the parameter update amount of the multiple initial prediction models to obtain the multiple prediction models.

[0042] As a preferred solution, the Laplace noise set is specifically:

[0043] Under the condition that structural parameters other than the data sets are ensured to remain consistent, using the preset data sets to update the first initial prediction model among the plurality of initial prediction models, and obtain updated model outputs on different data sets;

[0044] The maximum value of the difference in the output of the model is defined as sensitivity;

[0045] Calculate a scale parameter based on the sensitivity and a preset privacy budget;

[0046] Based on the scale parameter, according to the probability density function of the Laplace distribution, a random number generator is used to generate a noise value that conforms to the Laplace distribution to obtain a first Laplace noise;

[0047] The plurality of initial prediction models are traversed to obtain the Laplace noise set consisting of a plurality of first Laplace noises.

[0048] The present application also provides a storage medium having a computer program stored thereon, which is called and executed by a computer to implement the above-mentioned federated learning distributed training method for IoT devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flowchart of a federated learning distributed training method for IoT devices provided in an embodiment of the present application;

[0050] Figure 2 This is a diagram of the overall system architecture provided by the embodiment of the present application;

[0051] Figure 3 This is a structural diagram of a federated learning distributed training device for an Internet of Things device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] In the description of this application, it should be understood that the term "first" is used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" may explicitly or implicitly include one or more of such features. In the description of this application, unless otherwise specified, "several" means two or more.

[0054] The embodiment of the present application provides a federated learning distributed training method for IoT devices, which is mainly used in situations where local model training needs to be performed on IoT devices, and model updates need to be aggregated through a central server, thereby realizing localized data processing and centralized optimization of models, significantly improving the efficiency and security of data processing.

[0055] Example 1:

[0056] See also Figure 1 The embodiment of the present application provides a federated learning distributed training method for IoT devices, which is applicable to several IoT devices, including S1 to S2. The specific implementation steps are as follows:

[0057] S1. Sending several prediction models to a central server for aggregation to obtain a global model; wherein, several prediction models are established by introducing a Laplace noise set and based on a decision tree set and model prediction functions of several IoT devices; the decision tree set is established by fitting corresponding decision trees with a residual set as the first target variable; the residual set is established based on the target variables and model prediction results of several IoT devices.

[0058] Step S1 of the embodiment of the present application includes S1.1 to S1.6; wherein S1.1 is the process of calculating the residual set, S1.2 is the process of establishing the decision tree set based on the residual set, S1.3 is the process of establishing the Laplace noise set, S1.4 is the process of calculating several prediction models based on the decision tree set and the Laplace noise set, S1.5 is the process of evaluating the model performance, and S1.6 is the process of establishing the global model, specifically:

[0059] S1.1. Obtain sample data from several IoT devices;

[0060] Based on the sample data, if the goal of several IoT devices is to predict one or more continuous-valued target variables (i.e., regression task), the initial model prediction function F0(x) in several IoT devices is set to the average value of the target variable; if the goal of several IoT devices is to predict the category of the target variable (i.e., classification task), the initial model prediction function F0(x) is set to the proportion of the category;

[0061] Based on the model prediction results of the initial model prediction function, a residual set is calculated according to the target variable and the model prediction functions in several IoT devices;

[0062] Among them, the first residual in the residual set can be expressed as:

[0063] r im =y i -F m-1 (x i )

[0064] Among them, y i is the target variable corresponding to sample i, which can be a continuous or categorical label; F m-1 (x i ) is the model prediction result of the m-1th decision tree corresponding to sample i.

[0065] In Example S1.1 of the present application, by setting the prediction result of the initial model prediction function to the mean value or category proportion of the target variable, this effectively establishes a baseline or starting point for the entire prediction process. This baseline reflects a basic estimate of the target variable without considering any specific device or data characteristics. A residual set is calculated based on the initial prediction result, the target variable, and the model prediction function. This residual set reflects the prediction error of the model on the current data.

[0066] S1.2, using the residual set as the new target variable, fitting the corresponding decision tree h m (x), get the decision tree set;

[0067] Furthermore, the capability limitations of IoT devices can be accommodated by limiting the depth and number of decision trees. Specifically:

[0068] Limiting the depth of each decision tree to no more than a first preset interval; wherein the first preset interval may be 3 to 5 layers;

[0069] Limiting the total number of decision trees to no more than a second preset interval; wherein the second preset interval may be 10 to 50 trees;

[0070] Only some features are used to train each decision tree to reduce the computational burden; a randomly sampled data subset is used to train each decision tree to reduce memory usage.

[0071] S1.3. While ensuring that all structural parameters other than the data sets remain consistent, update the first initial prediction model among the plurality of initial prediction models using the plurality of preset data sets to obtain updated model outputs on the different data sets;

[0072] The maximum value of the difference in model output is defined as sensitivity Δf;

[0073] According to the sensitivity Δf and the preset privacy budget ∈, the scale parameter λ is calculated;

[0074] Based on the scale parameter λ and the probability density function of the Laplace distribution, a random number generator is used to generate a noise value that conforms to the Laplace distribution to obtain the first Laplace noise Lap(λ);

[0075] Traversing a number of initial prediction models to obtain a Laplace noise set consisting of a number of first Laplace noises;

[0076] The scale parameter is:

[0077] λ=Δf / ∈

[0078] The probability density function of Laplace noise is:

[0079]

[0080] Where Δf is the sensitivity; ∈ is the privacy budget, which is used to control the strength of privacy protection. The smaller the value of ∈, the stronger the privacy protection, but the practicality of the data may be reduced; x is the noise value.

[0081] In Example S1.3 of the present application, the greater the sensitivity, the more sensitive the model is to changes in a single data point, and the risk of privacy leakage increases accordingly. Therefore, the maximum difference in the model output is defined as sensitivity, which can quantify the sensitivity of the model to data changes and provide a basis for subsequent privacy protection measures. The scale parameter is calculated based on the sensitivity and privacy budget, which determines the amount of noise added to the model parameter update amount. Adding Laplace noise to the parameter update amount of the model can smooth the parameter update process and reduce the risk of overfitting. Moreover, since Laplace noise is additive and scalable, it can also effectively protect personal privacy.

[0082] S1.4. Control the step size of model parameter updates through the learning rate v, and establish several initial prediction models by combining the decision tree set and the model prediction functions of several IoT devices;

[0083] Adding the Laplace noise set to the parameter updates of several initial prediction models to obtain several prediction models; wherein the several prediction models reflect the data characteristics and environments unique to the several IoT devices, and the prediction models can also be called local models;

[0084] Among them, the first initial prediction model among the several initial prediction models is:

[0085] F m (x) = F m-1 (x)+vh m (x)

[0086] The first prediction model among several prediction models is:

[0087] θ new =θ old +Δθ+Lap(λ)

[0088] Where m is the ordinal number of the decision tree in the decision tree set, m = 1, 2, ..., M, M is the total number of decision trees in the decision tree set; F m-1 (x) is the initial prediction model corresponding to the m-1th decision tree, v is the learning rate, h m (x) is the mth decision tree;

[0089] θ old is the model parameter of the first initial prediction model; Δθ is the change of the first initial prediction model in the current update, that is, the parameter update obtained after local model training; Lap(λ) is the Laplace noise corresponding to the first initial prediction model.

[0090] In the embodiment of the present application, the application process of Laplace noise is exemplified as follows:

[0091] Under the preset conditions, the maximum value of the model output difference of several initial prediction models is calculated to be 1, that is, the sensitivity is Δf = 1;

[0092] Setting the privacy budget ∈ = 0.1, we can calculate the scale parameter λ = 10;

[0093] Generate Plass noise based on scale parameter λ Lap(λ) = 0.5;

[0094] Introducing Plass noise Lap(λ) into the parameter update of the initial prediction model to obtain the prediction model;

[0095] Among them, applying noise to the model update can be expressed as:

[0096] W new =W old +ΔW+0.5

[0097] Among them, W old is the model parameter of the initial prediction model, and ΔW is the change of the initial prediction model in the current update, that is, the parameter update obtained after local model training.

[0098] By setting an appropriate learning rate, Example S1.4 of the present application can effectively control the learning progress and convergence speed of the model; by combining the decision tree set with the model prediction function of the IoT device, the advantages of both can be fully utilized to establish a more accurate and robust prediction model, making it better adaptable to the complex and changing IoT environment. In addition, Laplace noise, as a random noise with a specific probability distribution, can be added to the parameter update amount of the prediction model to greatly improve the robustness and privacy protection capabilities of the model. In this way, even if the model update is shared, the information of a single user is effectively protected, thereby enhancing the privacy of the data in the federated learning environment.

[0099] S1.5. If the goal of several IoT devices is to predict one or more continuous-valued target variables (i.e., regression tasks), optimize the model parameters of several prediction models using mean squared error (MSE), thereby helping the prediction models better fit the data during local training and evaluating model performance after global model aggregation.

[0100] If the goal of several IoT devices is to predict the category of the target variable (i.e., classification task), the model parameters of several prediction models are optimized using logarithmic loss, thereby helping the several prediction models to better fit the data during local training, and then used to evaluate model performance after global model aggregation;

[0101] Among them, the mean square error can be expressed as:

[0102]

[0103] Among them, n is the number of samples in the preset data set, y i is the true value or observed value of the i-th sample in the preset data set, F(x i ) is the predicted value of the i-th sample in the preset data set.

[0104] S1.6, sending several prediction models to a central server;

[0105] Controlling the central server to dynamically adjust the weights of several prediction models according to the data volume, model quality and preset parameters of several IoT devices to obtain several weighted models;

[0106] By performing weighted averaging calculations on several weighted models, several weighted models can be aggregated into a global model. In addition, several weighted models can be merged to obtain a global model through aggregation strategies such as stochastic gradient descent (SGD), differential privacy (Differential Privacy), federated stochastic gradient descent (FedSGD), and dynamic adjustment of learning rate.

[0107] By integrating multiple prediction models, Example S1.6 of the present application enables the global model to combine the advantages of each model, reduce the deviations and overfitting problems that may occur in a single model, thereby improving the overall generalization ability of the model, and enabling the global model to make more accurate and stable predictions when faced with new and unseen data. Dynamically adjusting the weights of each model based on the data volume and model quality of the IoT device can ensure that resources can be more reasonably allocated to models with better performance. By performing a weighted average calculation on the weighted model, the global model can output a more comprehensive and accurate prediction result that integrates the prediction information of multiple models and can better reflect the actual situation of the data, thereby improving the prediction accuracy.

[0108] S2. Obtain the global model and train it using the gradient descent algorithm so that the parameters of the global model fit the local data set.

[0109] Step S2 of the embodiment of the present application is specifically as follows:

[0110] Distribute the global model back to several IoT devices and ensure that each IoT device receives the correct version of the model update by managing the global model versions generated at different time points. In addition, reduce the size of the data when sending and receiving model updates among several IoT devices to reduce network bandwidth usage and improve data transmission efficiency.

[0111] Based on the sample data of several IoT devices, the global model is trained using a gradient descent algorithm, so that the parameters of the global model fit the local data sets of several IoT devices, thereby better adapting to the specific operating environment and data distribution of each IoT device;

[0112] Furthermore, during training, any device failure or data transmission error that occurs during the federated learning process is detected and a recovery process is automatically initiated to maintain the continuity and integrity of the training;

[0113] Among them, the learning rate η of the gradient descent algorithm is adjusted according to the following formula:

[0114]

[0115] Where δ is the preset decay coefficient; t is the number of iterations, which is used to dynamically adjust the learning rate based on the real-time performance feedback of the model training to optimize the training process and improve the convergence speed of the model; η t is the learning rate at the tth iteration.

[0116] In an embodiment of the present application, if, in a more distributed or decentralized learning environment, several prediction models not only communicate with a central server but also need to directly exchange model parameters with their neighboring IoT devices, the model parameters from the neighboring IoT devices can be merged in the following manner to improve the model performance and data efficiency of the entire system while protecting the privacy of user data:

[0117] Obtain model parameter sets for several IoT devices based on local data training;

[0118] According to the data volume n of each IoT device among several IoT devices i and data quality score q i , calculate the weight set;

[0119] According to the parameter average value and weight set of the model parameter set, the global model parameter W is calculated using the preset weighted average formula.

[0120] Distribute the aggregated global model parameters W back to each IoT device in the IoT devices for the next round of training and optimization;

[0121] The weights are:

[0122]

[0123] The weighted average formula is:

[0124]

[0125] Among them, n i is the data volume of the ith IoT device among several IoT devices, q i is the data quality score of the i-th IoT device; j is the total number of IoT devices, and the denominator of the weight is the sum of the weighted data volume and data quality of all participating devices;

[0126] α i is the weight of the ith IoT device, w i is the model parameter of the i-th IoT device.

[0127] Among them, the data quality score q i The solution optimization method is as follows:

[0128] According to the update frequency and latestness of the data, the data timeliness score t of the i-th IoT device is calculated i ;

[0129] According to the consistency of data between various sources, the data consistency score c of the i-th IoT device is calculated i ;

[0130] According to the integrity of the data, the data integrity score g of the i-th IoT device is calculated i ;

[0131] According to the data timeliness score t i , data consistency score c i and data integrity score g i Calculate the data quality score q i1 ;

[0132] The data quality scores are:

[0133] q i1 =w t ·t i +w c c i +w g ·g i +w q ·q i

[0134] Among them, t i is the data timeliness score of the i-th IoT device, ranging from 0 to 1; c i is the data consistency score of the ith IoT device, ranging from 0 to 1, including format and value; g i is the data integrity score of the ith IoT device, ranging from 0 to 1, including the proportion of missing values and erroneous values; q i is the original data quality score, which can be obtained based on the accuracy and representativeness of the data; t 、w c 、w g and w q They are the weight of the data timeliness score, the weight of the data consistency score, the weight of the data integrity score and the weight of the original data quality score, and their sum is 1.

[0135] In Example S2 of the present application, by distributing models, not only can the computing resources of IoT devices be effectively utilized, but also the overall performance and adaptability of the model can be improved in a federated learning environment while protecting data privacy. Furthermore, the algorithm used when merging model parameters from adjacent IoT devices not only ensures improved model performance, but also considers the impact of data volume and data quality through the design of weights, ensuring that high-quality data contributes more to the model. Furthermore, it also ensures that specific user data is not directly shared, thereby protecting data privacy.

[0136] In addition, for the optimized data quality score, the cross-device collaborative optimization algorithm not only takes into account the data volume and original data quality, but also comprehensively considers the timeliness, consistency and integrity of the data, thereby more comprehensively evaluating the value of each device data, ensuring that high-quality data makes a greater contribution to the model, and at the same time enhancing the accuracy and robustness of the model; this multi-dimensional data quality assessment method is particularly suitable for the Internet of Things environment where data sources are diverse and dynamically changing.

[0137] To apply this application example, please refer to Figure 2 , Figure 2 This is a diagram of the overall system architecture provided by an embodiment of the present application, which illustrates the general framework for implementing a federated learning distributed training method for IoT devices provided by this embodiment;

[0138] Among them, after the local models of IoT devices 1~n are updated locally, the updated local models 1~n are sent to the central server for aggregation processing, and the received local models are aggregated by the data processing module to form a global model update, and then the aggregated global model is sent back to IoT devices 1~n through the model distribution module.

[0139] Overall, the embodiments of the present application have the following beneficial effects:

[0140] By calculating and analyzing the residual set, the present invention can intuitively understand the accuracy of model predictions in multiple IoT devices and determine whether the model has deviations or deficiencies. By fitting the residual set as the target variable to a decision tree, the prediction error of the model on the original data can be further optimized, thereby compensating for the deficiencies of the original model. By integrating multiple such decision trees, the overall prediction error can be gradually reduced and the prediction accuracy of the model can be improved. Because Laplace noise is additive and scalable, adding Laplace noise to the parameter update amount of the model can effectively protect personal privacy. Therefore, the multiple prediction models established in this way have high flexibility and scalability. Since each IoT device trains the model locally, the direct transmission and centralized storage of the original data are avoided, thereby reducing the risk of data leakage. In addition, data localization processing means that only model updates, rather than the entire data set of multiple IoT devices, need to be transmitted to the central server, which greatly reduces the amount of data transmission, reduces the demand for network bandwidth, and accelerates the transmission speed of model updates.

[0141] Example 2:

[0142] See also Figure 3 , an embodiment of the present application provides a federated learning distributed training apparatus for an IoT device, applicable to a number of IoT devices, including an aggregation module 10 and a fitting module 20;

[0143] The aggregation module 10 is used to send the plurality of prediction models to the central server for aggregation to obtain a global model; wherein the plurality of prediction models are established by introducing a Laplace noise set and based on a decision tree set and a model prediction function of a plurality of IoT devices; the decision tree set is established by fitting a corresponding decision tree with a residual set as the first target variable; the residual set is established based on the target variables of the plurality of IoT devices and the model prediction results;

[0144] The fitting module 20 is used to obtain a global model and train the global model through a gradient descent algorithm so that the parameters of the global model fit the local data set.

[0145] In one embodiment, the aggregation module 10 includes a setting unit, a residual unit, a decision tree unit, a noise unit, an initial unit, a model unit, an evaluation unit, a sending unit, a weight unit, and an aggregation unit; wherein the setting unit and the residual unit are the process of calculating the residual set, the decision tree unit is the process of establishing a decision tree set based on the residual set, the noise unit is the process of establishing a Laplace noise set, the initial unit and the model unit are the process of calculating several prediction models based on the decision tree set and the Laplace noise set, the evaluation unit is the process of evaluating the model performance, and the sending unit, the weight unit, and the aggregation unit are the process of establishing a global model, specifically:

[0146] A setting unit, used to obtain sample data of several IoT devices;

[0147] The setting unit is further configured to, based on the sample data, set the initial model prediction function F0(x) in the plurality of IoT devices to the average value of the target variable if the goal of the plurality of IoT devices is to predict one or more continuous-valued target variables (i.e., a regression task); and set the initial model prediction function F0(x) in the plurality of IoT devices to the proportion of the category if the goal of the plurality of IoT devices is to predict the category of the target variable (i.e., a classification task);

[0148] The residual unit is used to calculate the residual set based on the model prediction results of the initial model prediction function according to the target variable and the model prediction functions in several IoT devices;

[0149] Among them, the first residual in the residual set can be expressed as:

[0150] r im =y i -F m-1 (x i )

[0151] Among them, y i is the target variable corresponding to sample i, which can be a continuous or categorical label; F m-1 (x i ) is the model prediction result of the m-1th decision tree corresponding to sample i.

[0152] In the embodiment of the present application, the setting unit and residual unit set the prediction result of the initial model prediction function to the average value or category proportion of the target variable. This actually establishes a baseline or starting point for the entire prediction process. This baseline reflects a basic estimate of the target variable without considering any specific device or data characteristics. Based on the initial prediction result, the target variable, and the model prediction function, a residual set is calculated. This residual set reflects the prediction error of the model on the current data.

[0153] The decision tree unit is used to fit the corresponding decision tree h using the residual set as the new target variable. m (x), get the decision tree set;

[0154] Furthermore, the capability limitations of IoT devices can be accommodated by limiting the depth and number of decision trees. Specifically:

[0155] Limiting the depth of each decision tree to no more than a first preset interval; wherein the first preset interval may be 3 to 5 layers;

[0156] Limiting the total number of decision trees to no more than a second preset interval; wherein the second preset interval may be 10 to 50 trees;

[0157] Only some features are used to train each decision tree to reduce the computational burden; a randomly sampled data subset is used to train each decision tree to reduce memory usage.

[0158] A noise unit is used to update a first initial prediction model among the plurality of initial prediction models using the plurality of preset data sets while ensuring that structural parameters other than the data sets remain consistent, to obtain updated model outputs on different data sets;

[0159] The noise unit is also used to define the maximum difference of the model output as sensitivity Δf;

[0160] The noise unit is also used to calculate the scale parameter λ according to the sensitivity Δf and the preset privacy budget ∈;

[0161] The noise unit is further configured to generate a noise value conforming to the Laplace distribution using a random number generator based on a scale parameter λ and a probability density function of the Laplace distribution, thereby obtaining a first Laplace noise Lap(λ);

[0162] The noise unit is further used to traverse a plurality of initial prediction models to obtain a Laplace noise set consisting of a plurality of first Laplace noises;

[0163] The scale parameter is:

[0164] λ=Δf / ∈

[0165] The probability density function of Laplace noise is:

[0166]

[0167] Where Δf is the sensitivity; ∈ is the privacy budget, which is used to control the strength of privacy protection. The smaller the value of ∈, the stronger the privacy protection, but the practicality of the data may be reduced; x is the noise value.

[0168] In the noise unit of the embodiment of the present application, the greater the sensitivity, the more sensitive the model is to changes in a single data point, and the risk of privacy leakage also increases accordingly. Therefore, the maximum difference of the model output is defined as sensitivity, which can quantify the sensitivity of the model to data changes and provide a basis for subsequent privacy protection measures. The scale parameter is calculated based on the sensitivity and privacy budget, which determines the amount of noise added to the model parameter update. Adding Laplace noise to the parameter update of the model can smooth the parameter update process and reduce the risk of overfitting. Moreover, since Laplace noise is additive and scalable, it can also effectively protect personal privacy.

[0169] The initial unit is used to control the step size of the model parameter update through the learning rate v, and to establish several initial prediction models by combining the decision tree set and the model prediction functions of several IoT devices;

[0170] A model unit is configured to add a Laplace noise set to parameter updates of a plurality of initial prediction models to obtain a plurality of prediction models; wherein the plurality of prediction models reflect data characteristics and environments unique to the plurality of IoT devices, and the prediction models may also be referred to as local models;

[0171] Among them, the first initial prediction model among the several initial prediction models is:

[0172] F m (x) = F m-1 (x)+vh m (x)

[0173] The first prediction model among several prediction models is:

[0174] θ new =θ old +Δθ+Lap(λ)

[0175] Where m is the ordinal number of the decision tree in the decision tree set, m = 1, 2, ..., M, M is the total number of decision trees in the decision tree set; F m-1 (x) is the initial prediction model corresponding to the m-1th decision tree, v is the learning rate, h m (x) is the mth decision tree;

[0176] θ old is the model parameter of the first initial prediction model; Δθ is the change of the first initial prediction model in the current update, that is, the parameter update obtained after local model training; Lap(λ) is the Laplace noise corresponding to the first initial prediction model.

[0177] In the embodiment of the present application, the application process of Laplace noise is exemplified as follows:

[0178] Under the preset conditions, the maximum value of the model output difference of several initial prediction models is calculated to be 1, that is, the sensitivity is Δf = 1;

[0179] Setting the privacy budget ∈ = 0.1, we can calculate the scale parameter λ = 10;

[0180] Generate Plass noise based on scale parameter λ Lap(λ) = 0.5;

[0181] Introducing Plass noise Lap(λ) into the parameter update of the initial prediction model to obtain the prediction model;

[0182] Among them, applying noise to the model update can be expressed as:

[0183] W new =W old +ΔW+0.5

[0184] Among them, W old is the model parameter of the initial prediction model, and ΔW is the change of the initial prediction model in the current update, that is, the parameter update obtained after local model training.

[0185] By setting appropriate learning rates for the initial unit and model unit in the embodiment of the present application, the learning progress and convergence speed of the model can be effectively controlled. Combining the decision tree set with the model prediction function of the IoT device can fully utilize the advantages of both to establish a more accurate and robust prediction model, making it better adaptable to the complex and changing IoT environment. In addition, Laplace noise, as a random noise with a specific probability distribution, can be added to the parameter update amount of the prediction model to greatly improve the robustness and privacy protection capabilities of the model. In this way, even if the model update is shared, the information of a single user is effectively protected, thereby enhancing the privacy of the data in the federated learning environment.

[0186] An evaluation unit, configured to optimize model parameters of the prediction models by using mean squared error (MSE) if the goal of the IoT devices is to predict one or more continuous-valued target variables (i.e., a regression task), thereby helping the prediction models better fit the data during local training and evaluating model performance after global model aggregation;

[0187] The evaluation unit is further used to optimize the model parameters of the multiple prediction models by using logarithmic loss if the goal of the multiple IoT devices is to predict the category of the target variable (i.e., classification task), thereby helping the multiple prediction models to better fit the data in local training, and to evaluate the model performance after global model aggregation;

[0188] Among them, the mean square error can be expressed as:

[0189]

[0190] Among them, n is the number of samples in the preset data set, y i is the true value or observed value of the i-th sample in the preset data set, F(x i ) is the predicted value of the i-th sample in the preset data set.

[0191] A sending unit, configured to send a plurality of prediction models to a central server;

[0192] A weighting unit is used to control the central server to dynamically adjust the weights of several prediction models according to the data volume, model quality and preset parameters of several IoT devices to obtain several weighted models;

[0193] The aggregation unit is used to aggregate several weighted models into a global model by performing weighted average calculation on them. In addition, it can also merge several weighted models into a global model through aggregation strategies such as stochastic gradient descent (SGD), differential privacy (Differential Privacy), federated stochastic gradient descent (FedSGD), and dynamic adjustment of learning rate.

[0194] The sending unit, weight unit and aggregation unit of the embodiment of the present application integrate multiple prediction models, and the global model can combine the advantages of each model, reduce the deviation and overfitting problems that may occur in a single model, thereby improving the overall generalization ability of the model, so that the global model can make more accurate and stable predictions when facing new and unseen data. Dynamically adjusting the weights of each model according to the data volume and model quality of the IoT device can ensure that resources can be more reasonably allocated to models with better performance. By performing weighted average calculation on the weighted models, the global model can output a more comprehensive and accurate prediction result, which integrates the prediction information of multiple models and can better reflect the actual situation of the data, thereby improving the prediction accuracy.

[0195] In one embodiment, the fitting module 20 is specifically:

[0196] Distribute the global model back to several IoT devices and ensure that each IoT device receives the correct version of the model update by managing the global model versions generated at different time points. In addition, reduce the size of the data when sending and receiving model updates among several IoT devices to reduce network bandwidth usage and improve data transmission efficiency.

[0197] Based on the sample data of several IoT devices, the global model is trained using a gradient descent algorithm, so that the parameters of the global model fit the local data sets of several IoT devices, thereby better adapting to the specific operating environment and data distribution of each IoT device;

[0198] Furthermore, during training, any device failure or data transmission error that occurs during the federated learning process is detected and a recovery process is automatically initiated to maintain the continuity and integrity of the training;

[0199] Among them, the learning rate η of the gradient descent algorithm is adjusted according to the following formula:

[0200]

[0201] Where δ is the preset decay coefficient; t is the number of iterations, which is used to dynamically adjust the learning rate based on the real-time performance feedback of the model training to optimize the training process and improve the convergence speed of the model; η t is the learning rate at the tth iteration.

[0202] In an embodiment of the present application, if, in a more distributed or decentralized learning environment, several prediction models not only communicate with a central server but also need to directly exchange model parameters with their neighboring IoT devices, the model parameters from the neighboring IoT devices can be merged in the following manner to improve the model performance and data efficiency of the entire system while protecting the privacy of user data:

[0203] Obtain model parameter sets for several IoT devices based on local data training;

[0204] According to the data volume n of each IoT device among several IoT devices i and data quality score q i , calculate the weight set;

[0205] According to the parameter average value and weight set of the model parameter set, the global model parameter W is calculated using the preset weighted average formula.

[0206] Distribute the aggregated global model parameters W back to each IoT device in the IoT devices for the next round of training and optimization;

[0207] The weights are:

[0208]

[0209] The weighted average formula is:

[0210]

[0211] Among them, n i is the data volume of the ith IoT device among several IoT devices, q i is the data quality score of the i-th IoT device; j is the total number of IoT devices, and the denominator of the weight is the sum of the weighted data volume and data quality of all participating devices;

[0212] α i is the weight of the ith IoT device, w i is the model parameter of the i-th IoT device.

[0213] Among them, the data quality score q i The solution optimization method is as follows:

[0214] According to the update frequency and latestness of the data, the data timeliness score t of the i-th IoT device is calculated i ;

[0215] According to the consistency of data between various sources, the data consistency score c of the i-th IoT device is calculated i ;

[0216] According to the integrity of the data, the data integrity score g of the i-th IoT device is calculated i ;

[0217] According to the data timeliness score t i , data consistency score c i and data integrity score g i Calculate the data quality score q i1 ;

[0218] The data quality scores are:

[0219] q i1 =w t ·t i +w c c i +w g ·g i +w q ·q i

[0220] Among them, t i is the data timeliness score of the i-th IoT device, ranging from 0 to 1; c i is the data consistency score of the ith IoT device, ranging from 0 to 1, including format and value; g i is the data integrity score of the ith IoT device, ranging from 0 to 1, including the proportion of missing values and erroneous values; q i is the original data quality score, which can be obtained based on the accuracy and representativeness of the data; t 、w c 、w g and w q They are the weight of the data timeliness score, the weight of the data consistency score, the weight of the data integrity score and the weight of the original data quality score, and their sum is 1.

[0221] The fitting module 20 of the embodiment of the present application not only effectively utilizes the computing resources of IoT devices through model distribution, but also improves the overall performance and adaptability of the model in a federated learning environment while protecting data privacy. Moreover, the algorithm used when merging model parameters from adjacent IoT devices not only ensures improved model performance, but also considers the impact of data volume and data quality through weight design, ensuring that high-quality data contributes more to the model. At the same time, it also ensures that specific user data is not directly shared, thereby protecting data privacy.

[0222] In addition, for the optimized data quality score, the cross-device collaborative optimization algorithm not only takes into account the data volume and original data quality, but also comprehensively considers the timeliness, consistency and integrity of the data, thereby more comprehensively evaluating the value of each device data, ensuring that high-quality data makes a greater contribution to the model, and at the same time enhancing the accuracy and robustness of the model; this multi-dimensional data quality assessment method is particularly suitable for the Internet of Things environment where data sources are diverse and dynamically changing.

[0223] To apply this application example, please refer to Figure 2 , Figure 2 This is a diagram of the overall system architecture provided by an embodiment of the present application, which illustrates the general framework of a federated learning distributed training apparatus for IoT devices provided by this embodiment;

[0224] Among them, after the local models of IoT devices 1~n are updated locally, the updated local models 1~n are sent to the central server for aggregation processing, and the received local models are aggregated by the data processing module to form a global model update, and then the aggregated global model is sent back to IoT devices 1~n through the model distribution module.

[0225] Overall, the embodiments of the present application have the following beneficial effects:

[0226] By calculating and analyzing the residual set, the present invention can intuitively understand the accuracy of model predictions in multiple IoT devices and determine whether the model has deviations or deficiencies. By fitting the residual set as the target variable to a decision tree, the prediction error of the model on the original data can be further optimized, thereby compensating for the deficiencies of the original model. By integrating multiple such decision trees, the overall prediction error can be gradually reduced and the prediction accuracy of the model can be improved. Because Laplace noise is additive and scalable, adding Laplace noise to the parameter update amount of the model can effectively protect personal privacy. Therefore, the multiple prediction models established in this way have high flexibility and scalability. Since each IoT device trains the model locally, the direct transmission and centralized storage of the original data are avoided, thereby reducing the risk of data leakage. In addition, data localization processing means that only model updates, rather than the entire data set of multiple IoT devices, need to be transmitted to the central server, which greatly reduces the amount of data transmission, reduces the demand for network bandwidth, and accelerates the transmission speed of model updates.

[0227] Example 3:

[0228] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the federated learning distributed training method for an Internet of Things device;

[0229] Among them, the federated learning distributed training method of an Internet of Things device, if implemented in the form of a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0230] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A federated learning distributed training method for IoT devices, characterized in that: Applicable to several IoT devices, the method includes: Sending multiple prediction models to a central server for aggregation to obtain a global model; wherein the multiple prediction models are established by introducing a Laplace noise set and based on a decision tree set and model prediction functions of the multiple IoT devices; the decision tree set is established by fitting corresponding decision trees with a residual set as the first target variable; the residual set is established based on the target variables of the multiple IoT devices and the model prediction results; Obtaining the global model and training the global model using a gradient descent algorithm so that the parameters of the global model fit the local data set; The plurality of prediction models are established by introducing a Laplace noise set and based on a decision tree set and the model prediction functions of the plurality of IoT devices. Specifically, the plurality of initial prediction models are established by controlling the step size of the model parameter update by a learning rate and combining the decision tree set and the model prediction functions of the plurality of IoT devices; and the plurality of prediction models are obtained by adding the Laplace noise set to the parameter update amount of the plurality of initial prediction models. The Laplace noise set is obtained by: ensuring that structural parameters other than the data set remain consistent, using several preset data sets to update the first initial prediction model among the several initial prediction models, and obtaining the updated model output on different data sets; defining the maximum difference between the model outputs as sensitivity; calculating a scale parameter based on the sensitivity and a preset privacy budget; based on the scale parameter, using a random number generator to generate a noise value that conforms to the Laplace distribution according to the probability density function of the Laplace distribution, to obtain the first Laplace noise; traversing the several initial prediction models to obtain the Laplace noise set composed of several first Laplace noises.

2. The method for distributed training of federated learning for IoT devices according to claim 1, wherein: Several prediction models are sent to the central server for aggregation to obtain a global model, specifically: sending the plurality of prediction models to the central server; Controlling the central server to dynamically adjust the weights of the plurality of prediction models according to the data volume, model quality, and preset parameters of the plurality of IoT devices to obtain a plurality of weighted models; The weighted models are aggregated into the global model by performing weighted average calculation on the weighted models.

3. The method for distributed training of federated learning for IoT devices according to claim 1, wherein: The residual set is established based on the target variables and model prediction results of the plurality of IoT devices, specifically: Setting the initial model prediction function in the plurality of IoT devices to the average value or category proportion of the target variable; Based on the model prediction result of the initial model prediction function, the residual set is calculated according to the target variable and the model prediction functions in the plurality of IoT devices.

4. A federated learning distributed training device for IoT devices, characterized in that: Applicable to several IoT devices, including aggregation modules and fitting modules; The aggregation module is configured to send the plurality of prediction models to a central server for aggregation to obtain a global model; wherein the plurality of prediction models are established by introducing a Laplace noise set and based on a decision tree set and the model prediction functions of the plurality of IoT devices; the decision tree set is established by fitting corresponding decision trees with a residual set as the first target variable; and the residual set is established based on the target variables of the plurality of IoT devices and the model prediction results; The fitting module is used to obtain the global model and train the global model through a gradient descent algorithm so that the parameters of the global model fit the local data set; The plurality of prediction models are established by introducing a Laplace noise set and based on a decision tree set and the model prediction functions of the plurality of IoT devices. Specifically, the plurality of initial prediction models are established by controlling the step size of the model parameter update by a learning rate and combining the decision tree set and the model prediction functions of the plurality of IoT devices; and the plurality of prediction models are obtained by adding the Laplace noise set to the parameter update amount of the plurality of initial prediction models. The Laplace noise set is obtained by: ensuring that structural parameters other than the data set remain consistent, using several preset data sets to update the first initial prediction model among the several initial prediction models, and obtaining the updated model output on different data sets; defining the maximum difference between the model outputs as sensitivity; calculating a scale parameter based on the sensitivity and a preset privacy budget; based on the scale parameter, using a random number generator to generate a noise value that conforms to the Laplace distribution according to the probability density function of the Laplace distribution, to obtain the first Laplace noise; traversing the several initial prediction models to obtain the Laplace noise set composed of several first Laplace noises.

5. The federated learning distributed training device for an Internet of Things device according to claim 4, characterized in that: The aggregation module includes a sending unit, a weighting unit and an aggregation unit; Wherein, the sending unit is used to send the plurality of prediction models to the central server; The weighting unit is used to control the central server to dynamically adjust the weights of the plurality of prediction models according to the data volume, model quality and preset parameters of the plurality of IoT devices to obtain a plurality of weighted models; The aggregation unit is used to aggregate the multiple weighted models into the global model by performing weighted average calculation on the multiple weighted models.

6. The federated learning distributed training device for an Internet of Things device according to claim 4, characterized in that: The aggregation module includes a setting unit and a residual unit; The setting unit is configured to set the initial model prediction function in the plurality of IoT devices to the average value or category ratio of the target variable; The residual unit is used to calculate the residual set based on the model prediction result of the initial model prediction function, according to the target variable and the model prediction functions in the multiple Internet of Things devices.

7. A storage medium, characterized in that: The storage medium stores a computer program, which is called and executed by a computer to implement the federated learning distributed training method for an IoT device as described in any one of claims 1 to 3 above.

Images