Heterogeneous industrial detection training method based on rolling extraction
Through rolling extraction and dynamic weighting heterogeneous industrial detection training methods, the problems of equipment resource inequality and data inconsistency are solved, low-resource equipment participation in training, optimize model accuracy and generalization capabilities, reduce model offsets, and improve federated learning compatibility.
Patent Information
- Application Number
- CN202510416138.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-03
- Publication Date
- 2025-07-04
AI Technical Summary
In the case of significant differences in device resources and data distribution in traditional federated learning methods, low-resource devices cannot participate in training, resulting in limited generalization of the model, and the problems of uneven and inconsistency in model parameter updates in existing heterogeneous methods have not been effectively solved.
The heterogeneous industrial detection training method of rolling extraction is adopted. Each client extracts the sub-model from the server's global model through each client, and uses a rolling window to update the parameter set, combining selective average aggregation and dynamic weighting to optimize the global model.
It realizes that on the premise of ensuring data privacy, low-resource devices are supported to participate in training, optimize model accuracy and generalization capabilities, reduce model offsets, and improve federated learning compatibility and generalization capabilities.
Smart Images
Figure CN120256966A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of federated learning and distributed machine learning, and particularly relates to a heterogeneous industrial detection training method based on rolling extraction. Background Art
[0002] With the popularization of the Internet of Things and intelligent devices, federated learning has become a key technology for protecting data privacy and realizing cross-device collaborative modeling. However, traditional federated learning methods mostly adopt homogeneous models, that is, the same model architecture is used on the server and client devices. This method faces many limitations in reality. Especially in the case of uneven device resources and significant differences in data distribution, low-resource devices often cannot participate in the training of large models, resulting in limited model generalization.
[0003] Some existing heterogeneous federated learning methods introduce partial model training and knowledge distillation techniques in an attempt to bridge the gap brought by different device resources. However, knowledge distillation methods usually rely on public data on the server side, and such data is not always available. In addition, partial model training methods mostly adopt random or static sub-model extraction strategies, which will lead to uneven parameter updates of the global model and further result in the inconsistency between the client model and the server model. Summary of the Invention
[0004] The purpose of the present invention is to provide a heterogeneous industrial detection training method based on rolling extraction for the above deficiencies in the prior art, so as to solve the problems of the traditional method under device heterogeneity and data inconsistency.
[0005] To achieve the above purpose, the technical solution adopted by the present invention is:
[0006] A heterogeneous industrial detection training method based on rolling extraction, which includes:
[0007] S1. Each client extracts a sub-model from the global model of the server and trains the sub-model;
[0008] S2. Each client extracts a sub-model parameter set from the global model of the server using a rolling window, and trains the sub-model parameter set using the trained client sub-model to complete the update of the sub-model;
[0009] S3. The server selectively averages and aggregates the updated sub-model parameters of each client, and then updates the sub-model parameters of the client;
[0010] S4. The server dynamically weights and aggregates according to the classification accuracy of the sub-model of each client, and then optimizes the global model;
[0011] S5. Repeat steps S2 to S4 until the global model of the server converges.
[0012] Further, S1 includes the following sub-steps:
[0013] S11. The client extracts a sub-model from the global model of the server according to its device resources;
[0014] S12. Train the extracted sub-model using the local dataset.
[0015] Further, in S11, the extracted sub-model is:
[0016]
[0017] Where is the sub-model extracted from the global model θ, n represents the client, j represents the communication round; β n is the computing resource of the client device; the ExtractSubModel(·) function selects a certain proportion of parameters from the global model θ according to β n from the global model θ.
[0018] Further, in S12, the loss function of the sub-model trained using the local dataset is:
[0019]
[0020] Where is the loss function of the sub-model; D n is the local dataset of the client; k is the sample index of the nth client dataset D n ; d n,k is the specific sample of the local dataset for a certain training of the client; l(·) is the cross-entropy loss.
[0021] Further, in S2, each client extracts a set of sub-model parameters from the global model of the server using a rolling window as:
[0022]
[0023] Where is the set of sub-model parameters extracted by client n from the i-th layer of the global model in the j-th round; mod is the modulo symbol; K i is the number of optional modules in the i-th layer of the global model.
[0024] Further, S3 specifically includes:
[0025] Each client sends the updated sub-model to the server, and the server processes the updated sub-models of each client Perform selective average aggregation on the parameters:
[0026]
[0027] where θ [i,k] is the k-th parameter θ of the i-th layer of the global model [i,k] ; M k is the set of clients for the updated k-th parameter of the i-th layer; θ m,[i,k] is the k-th parameter θ of the i-th layer of the local model of the m-th client [i,k] .
[0028] Furthermore, S4 includes the following sub-steps:
[0029] S41. Calculate the classification accuracy of the sub-model of the client;
[0030] S42. The server updates the global model using the dynamic weighted aggregation method according to the classification accuracy of the client sub-model.
[0031] Furthermore, in S41, the classification accuracy of the sub-model of the client is:
[0032]
[0033] where A n is the classification accuracy; is the validation data set of the sub-model of the client; is the predicted value of the sub-model; y k is the true label; 1 is the indicator function.
[0034] Furthermore, in S42, the server updates the global model using the dynamic weighted aggregation method according to the classification accuracy of the client sub-model, including:
[0035] The server performs dynamic weight allocation according to the classification accuracy of the client sub-model:
[0036]
[0037] where w n is the dynamic weight assigned by the server; ε is the smoothing parameter; A m is the classification accuracy of the m-th client; Aggregate the dynamic weights and then update the global model:
[0038]
[0039] where θ is the global model parameter; N is the number of clients participating in this federated learning training.
[0040] The heterogeneous industrial detection training method based on rolling extraction provided by the present invention has the following beneficial effects:
[0041] 1. By rolling and extracting sub-models between different client devices, the present invention ensures that every part of the global model can be fully trained, thus alleviating the problem of client drift. The method of the present invention not only supports the participation of low-resource devices, but also optimizes the accuracy and generalization ability of the model on the premise of ensuring data privacy.
[0042] 2. Through the rolling sub-model extraction and dynamic weighted aggregation mechanism, the present invention effectively adapts to the differences in client device resources, improves the compatibility of federated learning and the generalization ability of the model. Through the rolling window-based sub-model extraction method, all parameters of the global model are evenly trained, reducing the drift phenomenon of the model in heterogeneous environments. At the same time, it supports the training of large global server models, breaks through the limitations of client device resources, and improves the inclusiveness of federated learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a schematic diagram of the heterogeneous industrial detection training method based on rolling extraction of the present invention
[0044] Figure 2 Schematic diagram of random sub-model extraction in the embodiment of the present invention.
[0045] Figure 3 It is a schematic diagram of static sub-model extraction in the embodiment of the present invention.
[0046] Figure 4 It is a flowchart of the heterogeneous industrial detection training method based on rolling extraction. DETAILED DESCRIPTION OF THE INVENTION
[0047] The following describes the specific implementation manners of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation manners. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.
[0048] Example 1. The heterogeneous industrial detection training method based on rolling extraction in this example refers to Figure 1 and Figure 4 , and specifically includes the following content:
[0049] Step S1. Extraction and training of sub-models;
[0050] It specifically includes the following sub-steps:
[0051] Step S11. Extraction of sub-models;
[0052] Each client extracts a sub-model from the global model of the server according to the resources of its device:
[0053] The extracted sub-model is:
[0054]
[0055] Where, is the sub-model extracted from the global model θ, n represents the client, j represents the communication round; β n is the computing resource of the client device; the ExtractSubModel(·) function selects a certain proportion of parameters from the global model θ according to β n From the global model θ.
[0056] Based on the above, the present invention allows small-capacity devices to only extract partial parameters for training, while larger-capacity devices extract more parameters, so that all devices can participate in federated training.
[0057] Step S12: Training of the sub-model;
[0058] The client uses the local dataset to train the extracted sub-model. The training goal is to minimize the classification error of the model for the local data, and its loss function is:
[0059]
[0060] Where, is the loss function of the sub-model; D n is the local dataset of the client; k is the nth client dataset; d n,k is the sample of the local dataset; l(·) is the cross-entropy loss.
[0061] After completing the training of the sub-model on the client side, the client uploads the updated sub-model parameters to the server. Since only the sub-model parameters are uploaded, the present invention significantly reduces the communication overhead while ensuring data privacy.
[0062] Step S2: Extracting rolling sub-models;
[0063] Specifically, each client uses a rolling window to extract a sub-model parameter set from the global model of the server. In each round of communication, this window will advance one step to ensure that all parameters of the global model are evenly trained, and the trained client sub-model is used to train the sub-model parameter set to complete the update of the sub-model;
[0064] The sub-model parameter set extracted by each client using a rolling window from the global model of the server is:
[0065]
[0066] Among them, is the set of sub-model parameters extracted by client n from the i-th layer of the global model in the j-th round; mod is the remainder symbol in mathematical formulas; K i is the number of optional modules in the i-th layer of the global model.
[0067] Static sub-model extraction selects the same sub-model structure from the global model in each round of communication. Rolling sub-model extraction is a dynamic sub-model extraction. Their processes are respectively as Figure 3 and 2 shown.
[0068] Through the rolling window, the present invention enables each part of the global model to be evenly trained by the clients in multiple rounds of communication, effectively reducing the client drift caused by the inconsistency between the client sub-model and the server global model architecture.
[0069] Step S3, sub-model update and selective average aggregation;
[0070] After each round of communication, each client sends its updated sub-model to the server, and the server performs selective average aggregation on the updated sub-model parameters of each client, and then updates the sub-model parameters of the clients;
[0071] Specifically, the server performs selective average aggregation on the parameters of the updated sub-model of each client :
[0072]
[0073] In the formula, θ [i,k] is the k-th parameter θ [i,k] of the i-th layer of the global model; M k is the set of clients of the updated k-th parameter of the i-th layer; θ m,[i,k] is the k-th parameter θ [i,k] of the i-th layer of the local model of the m-th client.
[0074] The selective aggregation of the present invention ensures that only the updated parameters (models) are incorporated into the update of the global model, thus avoiding the uneven update problem of random or static model extraction.
[0075] Step S4, dynamic weighted aggregation;
[0076] In order to further improve the generalization ability of the global model, in each round of communication, the server dynamically adjusts the aggregation weight according to the classification accuracy of the sub-model of each client, and then optimizes the global model, which specifically includes the following sub-steps:
[0077] Step S41: Calculate the classification accuracy of the sub-model of the client;
[0078]
[0079] In the formula, A n is the classification accuracy; is the validation data set of the sub-model of the client; is the predicted value of the sub-model; y k is the true label; 1 is the indicator function.
[0080] Step S42: The server updates the global model by using the dynamic weighted aggregation method according to the classification accuracy of the client sub-model;
[0081] The server performs dynamic weight allocation according to the classification accuracy of the client sub-model:
[0082]
[0083] In the formula, w n is the dynamic weight assigned by the server; ε is the smoothing parameter; A m is the classification accuracy of the m-th client;
[0084] Perform aggregation processing on the dynamic weights, and then update the global model:
[0085]
[0086] In the formula, θ is the global model parameter; N is the number of clients participating in the federated learning training this time;
[0087] Through dynamic weighting, the present invention depends more on the model parameters of clients with high accuracy during the aggregation process, thereby optimizing the generalization of the global model.
[0088] Step S5: Repeat steps S2 to S4 until the global model of the server converges.
[0089] Iterate in a loop until the model converges. The present invention repeats the above steps until the loss function of the global model converges, reaching the desired accuracy or error threshold. At this time, the global model has integrated the data distribution characteristics of different clients and can exhibit good prediction performance in a heterogeneous data environment.
[0090] Through the rolling sub-model extraction and dynamic weighted aggregation mechanism, the present invention effectively adapts to the differences in client device resources, improves the compatibility of federated learning and the generalization ability of the model. Through the rolling window-based sub-model extraction method, all parameters of the global model are evenly trained, reducing the deviation phenomenon of the model in heterogeneous environments. It supports FL with model heterogeneity and can train a global server model larger than the largest client model, and reduces the gap between model heterogeneity and model isomorphism FL, especially in the regime of large models and large datasets.
[0091] Although the specific implementation manners of the invention have been described in detail in conjunction with the accompanying drawings, it should not be construed as a limitation on the protection scope of this patent. Within the scope described in the claims, various modifications and deformations that can be made by those skilled in the art without creative efforts still fall within the protection scope of this patent.
Claims
1. A heterogeneous industrial detection training method based on rolling extraction, characterized in that, It includes the following steps: S1. Each client extracts a sub-model from the global model of the server and trains the sub-model; S2. Each client extracts a set of sub-model parameters from the global model of the server using a rolling window, and trains the set of sub-model parameters using the trained client sub-model to complete the update of the sub-model; S3. The server selectively averages and aggregates the updated sub-model parameters of each client, and then updates the sub-model parameters of the client; S4. The server performs dynamic weighted aggregation based on the classification accuracy of the sub-model of each client, and then optimizes the global model; S5. Repeat steps S2 to S4 until the global model of the server converges.
2. The heterogeneous industrial detection training method based on rolling extraction according to claim 1, wherein The S1 includes the following sub-steps: S11. The client extracts a sub-model from the global model of the server according to its device resources; S12. Train the extracted sub-model using the local dataset.
3. The heterogeneous industrial detection training method based on rolling extraction according to claim 2, wherein In the S11, the extracted sub-model is: where is the sub-model extracted from the global model θ, n represents the client, and j represents the communication round; β n is the computing resource of the client device; the ExtractSubModel(·) function selects a certain proportion of parameters from the global model θ according to β n 4. The heterogeneous industrial detection training method based on rolling extraction according to claim 3, wherein, In the S12, the loss function of the sub-model trained using the local dataset is: Wherein, is the loss function of the sub-model; D n is the local dataset of the client; k is the sample index of the nth client dataset D n ; d n,k is the specific sample of the local dataset for a certain training of the client; l(·) is the cross-entropy loss.
5. The heterogeneous industrial detection training method based on rolling extraction according to claim 3, wherein In the S2, the set of sub-model parameters extracted by each client from the global model of the server using a rolling window is: Among them, is the set of sub-model parameters extracted by client n from the i-th layer of the global model in the j-th round; mod is the modulo symbol; K i is the number of optional modules in the i-th layer of the global model.
6. The heterogeneous industrial detection training method based on rolling extraction according to claim 1, wherein The S3 specifically includes: Each client will send the updated sub-model to the server, and the server will perform selective average aggregation on the parameters of the updated sub-model of each client as follows: where θ [i,k] is the k-th parameter θ of the i-th layer of the global model [i,k] ; M k is the set of clients for the updated k-th parameter of the i-th layer; θ m,[i,k] is the k-th parameter θ of the i-th layer of the local model of the m-th client i,k .
7. The heterogeneous industrial detection training method based on rolling extraction according to claim 6, wherein, The S4 includes the following sub-steps: S41. Calculate the classification accuracy of the sub-model of the client; S42. The server updates the global model using the dynamic weighted aggregation method according to the classification accuracy of the client sub-model.
8. The heterogeneous industrial detection training method based on rolling extraction according to claim 7, wherein In the S41, the classification accuracy of the sub-model of the client is: Where A n is the classification accuracy; is the validation dataset of the sub-model of the client; is the predicted value of the sub-model; y k is the true label; 1 is the indicator function.
9. The heterogeneous industrial detection training method based on rolling extraction according to claim 8, characterized in that, In the S42, the server updates the global model using the dynamic weighted aggregation method according to the classification accuracy of the client sub-model, including: The server performs dynamic weight allocation according to the classification accuracy of the client sub-model: where w n is the dynamic weight assigned by the server; ε is the smoothing parameter; A m is the classification accuracy of the m-th client; the dynamic weights are aggregated to update the global model: where θ is the global model parameter; N is the number of clients participating in this federated learning training.