A federated learning method and system for industrial internet edge sensor device training

Abstract

The present application relates to the field of industrial internet, and provides a federated learning method and system for training of an industrial internet edge sensor device. The method comprises: distributing global model parameters and aggregated momentum to a selected subset of edge sensor devices, and initializing to obtain local model parameters and local momentum; dividing the local model parameters into multiple parameter blocks, performing momentum updating on the current batch gradient through a momentum accumulation algorithm to obtain a momentum matrix; performing orthogonalization processing on the momentum matrix through a Newton-Schulz iteration algorithm to obtain an orthogonalization matrix; performing momentum gradient alignment scoring on the multiple parameter blocks to obtain a smooth alignment score; performing weighted updating on the orthogonalization matrix to obtain updated parameter blocks; and aggregating the parameter block increments and the updated momentum by the server to obtain updated global model parameters and updated global aggregated momentum. The present application reduces the communication frequency and improves the training efficiency of the edge sensor device.

Description

Technical Field

[0001] This invention relates to the field of industrial internet technology, and in particular to a federated learning method and system for training edge sensor devices in the industrial internet. Background Technology

[0002] The Industrial Internet deploys a large number of edge sensor devices (such as industrial sensors and intelligent controllers). These devices continuously collect data from the industrial site during production, possessing inherent characteristics of data localization and distributed deployment. Federated learning, as a distributed machine learning framework, allows edge sensor devices to complete model training locally without uploading raw data, enabling collaborative modeling across multiple devices while protecting data privacy. However, edge sensor devices generally suffer from limited memory and resource constraints, relying on ARM CPUs for computing power without dedicated GPUs. This results in slow training of large-scale models on edge sensor devices. Furthermore, the frequent full parameter transfers between edge sensor devices and servers under the federated learning framework consume significant communication bandwidth, further limiting the efficiency and practicality of distributed model training in industrial scenarios.

[0003] In existing technologies, traditional local optimizers (such as Adam and SGD) update model parameters element-wise, ignoring the geometric structure of the weight matrix, resulting in low convergence efficiency of local training. The Muon optimizer uses Newton-Schulz iteration to approximate orthogonalize the gradient momentum to optimize the geometric structure of the weight matrix, but its high computational cost makes it difficult to adapt to resource-constrained edge sensor devices. The MAGMA optimizer introduces structured randomness and uses cosine similarity to measure the directional consistency between momentum and gradient to suppress heavy-tailed noise interference, but it does not incorporate the communication optimization requirements of federated learning frameworks. Summary of the Invention

[0004] This invention provides a federated learning method for training edge sensor devices in the Industrial Internet, in order to address the shortcomings of existing technologies.

[0005] The first aspect of this invention provides a federated learning method for training edge sensor devices in the Industrial Internet, comprising: S1. Distribute the global model parameters and aggregate momentum to the selected subset of edge sensor devices, and have multiple edge sensor devices initialize and obtain local model parameters and local momentum respectively; S2. Divide the local model parameters into multiple parameter blocks, and update the momentum of the current batch gradient using the momentum accumulation algorithm for the multiple parameter blocks to obtain the momentum matrix; S3. The momentum matrix is ​​orthogonalized using the Newton-Schultz iterative algorithm to obtain the orthogonalized matrix; S4. Based on the current batch gradient and the momentum matrix, perform momentum gradient alignment scoring on multiple parameter blocks to obtain smooth alignment scores; S5. Based on the smooth alignment score and the generated random mask, perform a weighted update on the orthogonalization matrix to obtain the update parameter block; S6. Upload the parameter block increments and update momentum corresponding to the updated parameter blocks of multiple edge sensor devices to the server. The server then aggregates the parameter block increments and update momentum to obtain the updated global model parameters and the updated global aggregated momentum.

[0006] According to the federated learning method for training edge sensor devices in the Industrial Internet provided by the present invention, step S2 further includes: S21. Divide the local model parameters into multiple non-overlapping parameter blocks, each parameter block corresponding to a weight matrix in the model; S22. In the current local iteration, calculate the batch gradient of each edge sensor device on the current parameter block; S23. The calculated batch gradient and preceding momentum are weighted and accumulated using a momentum accumulation algorithm to obtain the momentum matrix; the expression for the momentum matrix is: in, Indexing edge sensor devices, For parameter block index, This serves as a global training round index. For the local iteration count index of edge sensor devices The momentum coefficient, For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper The gradient of the current batch is calculated during the next local iteration.

[0007] According to the federated learning method for training edge sensor devices in the Industrial Internet provided by the present invention, the expression for the orthogonalization matrix in step S3 is as follows: in, For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the left orthogonal factor matrix. For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the right orthogonal factor matrix. Indicates matrix transpose. This is the Newton-Schultz iteration function, used to approximate orthogonalize the input matrix using a 5th-order polynomial iteration method.

[0008] According to the federated learning method for training edge sensor devices in the Industrial Internet provided by the present invention, step S4 further includes: S41. Calculate the cosine similarity between the current batch gradient and the momentum matrix; the expression for the cosine similarity is: in, For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper The gradient of the current batch is calculated during the next local iteration. for and Cosine similarity between them Indicates matrix transpose; S42. The cosine similarity is scaled using a temperature parameter and mapped using the sigmoid function to obtain the original alignment score; the expression for the original alignment score is: in, For the first Parameter block in the next iteration The original alignment score, For temperature parameters; S43. The original alignment score is smoothed using an exponential moving average algorithm to obtain a smoothed alignment score.

[0009] According to the federated learning method for training edge sensor devices in the Industrial Internet provided by the present invention, in step S42, the relationship between the temperature parameter and the original alignment score is as follows: when the cosine similarity approaches +1, the original alignment score approaches a first preset value, and the parameter block update is retained; when the cosine similarity approaches 0, the original alignment score approaches a second preset value, and the parameter block update amplitude is halved; when the cosine similarity approaches -1, the alignment score approaches a third preset value, and the parameter block update is suppressed. Wherein, the first preset value is greater than the second preset value, which is greater than the third preset value, and the second preset value is 0.5.

[0010] According to the federated learning method for training edge sensor devices in the Industrial Internet provided by the present invention, step S5 further includes: S51. For each parameter block, a random mask is generated by sampling using a Bernoulli distribution; the expression for the random mask is: in, For the first Parameter block in the next iteration The corresponding random mask, For sampling probability, For The sampling probability follows a Bernoulli distribution; S52. Based on the random mask, the smooth alignment score is combined with the random mask to perform weighted scaling on the basic gradient, obtaining the actual update amount; the expression for the actual update amount is: in, For the first Parameter block in the next iteration The actual update volume, For the first Parameter block in the next iteration Smooth alignment score, For the first Parameter block in the next iteration The corresponding base optimizer updates the gradient; S53. The orthogonalized matrix is ​​fused with the weighted basic gradient, and the parameter block is updated by combining the smooth alignment score and the random mask to obtain the updated parameter block; the expression of the updated parameter block is: in, For learning rate, This is the balance coefficient between orthogonalization update and basic gradient. For the first During round training, edge sensor devices In the parameter block Above the first Local model parameters after the next local iteration For the first During round training, edge sensor devices In the parameter block Above the first Local model parameters after the next local iteration For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the left orthogonal factor matrix. For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the right orthogonal factor matrix. Indicates matrix transpose. This refers to the global gradient update issued by the server.

[0011] According to the federated learning method for training edge sensor devices in the Industrial Internet provided by the present invention, step S6 further includes: S61. Multiple edge sensor devices upload the parameter block increment and update momentum after local training to the server. The parameter block increment is the difference between the corresponding parameter blocks before and after local training, and the update momentum is the momentum matrix obtained from the last local iteration. S62. The mean of the parameter block increments uploaded by all edge sensor devices participating in this round of training is aggregated to obtain updated global model parameters; the expression for updating the global model parameters is: in, For the first After the round aggregation is completed, the server updates the obtained parameter block. The corresponding global model parameters are updated. For the first At the start of a training round, the server maintains a block of parameters. The corresponding global model parameters, Indexing edge sensor devices, The size of the subset of edge sensor devices participating in training in each round. For the first During round training, edge sensor devices In the parameter block Complete all The final local model parameters after the next local iteration. For the first At the start of the training round, the edge sensor device In the parameter block Initial local model parameters; S63. Calculate the global gradient update based on the parameter block increment; the expression for the global gradient update is: in, For the first After the round aggregation is completed, the server performs a parameter block analysis. The calculated global gradient update, This represents the total number of local iterations for the edge sensor device. The learning rate; S64. The updated momentum uploaded by each edge sensor device is averaged and aggregated to obtain the updated global aggregated momentum; the expression for the updated global aggregated momentum is: in, For the first After round aggregation is completed, the server performs mean aggregation on the updated momentum of each device to obtain the updated global aggregated momentum. For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration; S65. The server will update the global model parameters, update the global aggregate momentum, and update the global gradient, and uniformly distribute them to multiple edge sensor devices as the initial input for the next round of local training.

[0012] A second aspect of the present invention provides a federated learning system for training edge sensor devices in the Industrial Internet, comprising: a server and an edge sensor device. The server-side includes: The global parameter management module is used to store and maintain updated global model parameters and updated global aggregate momentum, and to send the updated global model parameters and the updated global aggregate momentum to the selected subset of edge sensor devices. The device scheduling module is used to select a subset of edge sensor devices participating in the current round of training from the registered edge sensor devices, and to manage the training status of each edge sensor device. The aggregation calculation module is used to receive parameter block increments and update momentum uploaded by multiple edge sensor devices, and perform mean aggregation operations on the parameter block increments and update momentum respectively to obtain updated global model parameters and updated global aggregate momentum. The server communication interface module is used to send global model parameters and aggregate momentum to a subset of edge sensor devices, and to receive parameter block increments and updated momentum uploaded by each edge sensor device. Edge sensor devices include: The local initialization module is used to receive global model parameters and aggregate momentum sent by the server, initialize the global model parameters as local model parameters, and initialize the aggregate momentum as local momentum; The parameter block partitioning module is used to divide the local model parameters into multiple parameter blocks according to the boundary of the weight matrix, providing input for the independent calculation of multiple parameter blocks in the future; The momentum accumulation module is used to update the momentum matrix in each local iteration based on the current batch gradient and the previous momentum using the momentum accumulation algorithm. The orthogonalization calculation module is used to orthogonalize the momentum matrix using the Newton-Schultz iterative algorithm to obtain an orthogonal matrix; The alignment scoring module is used to perform momentum gradient alignment scoring on multiple parameter blocks based on the current batch gradient and the momentum matrix to obtain a smooth alignment score. The random mask generation module is used to sample and generate a random mask for each parameter block according to the Bernoulli distribution; The parameter block update module is used to fuse the orthogonalization matrix with the basic gradient weighted fusion, and combine the smooth alignment score with the random mask to complete the weighted update of each parameter block and obtain the updated parameter block; The device communication interface module is used to upload the parameter block increments and update momentum corresponding to the updated parameter blocks of each parameter block after local training to the server, and to receive the updated global model parameters and updated global aggregate momentum sent by the server.

[0013] A third aspect of the present invention provides a federated learning device for training edge sensor devices in the Industrial Internet, comprising: a memory and at least one processor, wherein the memory stores instructions; the processor invokes the instructions in the memory to cause the federated learning device for training edge sensor devices in the Industrial Internet to execute a federated learning method for training edge sensor devices in the Industrial Internet as described in any of the preceding claims.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement a federated learning method for training edge sensor devices for the Industrial Internet as described in any of the preceding claims.

[0015] This invention divides local model parameters into multiple independent parameter blocks and introduces a random masking mechanism, ensuring that only an average of 50% of the parameter blocks participate in computation and updates in each local iteration. This significantly reduces the computational burden on resource-constrained edge sensor devices, enabling them to support the local training process of large-scale models and fundamentally overcome the resource bottleneck of deploying large models at the industrial edge. Simultaneously, the content uploaded by the edge sensor device to the server is only the parameter block increments and update momentum, not the full model parameters. The amount of communication data is compressed to less than 50% of the full parameter size, effectively alleviating the bandwidth consumption problem caused by frequent full parameter transmissions under the federated learning framework and reducing the communication overhead of multi-device collaborative training. Regarding model training quality, this invention optimizes the geometric structure of the weight matrix by orthogonalizing the momentum matrix using the Newton-Schultz iterative algorithm, making the parameter update direction more reasonable and accelerating the convergence speed of the global model. The momentum gradient alignment scoring mechanism quantifies the consistency between the gradient direction and momentum direction of each parameter block based on cosine similarity, and obtains a smooth alignment score through exponential moving average smoothing. This allows parameter blocks with high directional consistency to receive larger update amplitudes, while the updates of parameter blocks with opposite directions are suppressed. This effectively suppresses noise interference in heavy-tailed distribution scenarios of industrial data, reduces the false negative rate in tasks such as industrial defect detection, and improves the generalization ability and robustness of the global model in complex industrial scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1A schematic diagram of a federated learning method for training edge sensor devices in the Industrial Internet provided by this invention; Figure 2 This invention provides a schematic diagram of a federated learning system architecture for training edge sensor devices in the Industrial Internet. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, embodiments of this invention, and should not be construed as limiting the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. In the description of this invention, it should be understood that the terminology used is for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0019] The embodiments of the present invention are described below with reference to the figures.

[0020] like Figure 1 As shown, this invention provides a federated learning method for training edge sensor devices in the Industrial Internet, comprising: S1. Distribute the global model parameters and aggregate momentum to the selected subset of edge sensor devices, and have multiple edge sensor devices initialize and obtain local model parameters and local momentum respectively.

[0021] Furthermore, at the start of each training round, the server first selects a subset of all registered edge sensor devices to participate in the current training round. Specifically, the server synchronously distributes the global model parameters and aggregate momentum maintained after the previous round of aggregation to each edge sensor device within the subset via the communication interface. After receiving the above data, each edge sensor device directly assigns the global model parameters to the local model parameters of the current round and the aggregate momentum to the local momentum of the current round, completing the initialization before the local training round. It should be noted that the initialization of local momentum does not start from 0, but rather inherits the historical momentum information aggregated by the server. Each edge sensor device already has a certain gradient direction accumulation at the start of the local iteration, and the parameter block index and device index jointly identify the initial state of each parameter block on each device.

[0022] S2. Divide the local model parameters into multiple parameter blocks. For each parameter block, update the momentum of the current batch gradient using the momentum accumulation algorithm to obtain the momentum matrix.

[0023] Step S2 further includes: S21. Divide the local model parameters into multiple non-overlapping parameter blocks, each parameter block corresponding to a weight matrix in the model.

[0024] In step S21, after initialization, each edge sensor device immediately performs parameter block partitioning on its local model parameters. Specifically, this invention divides the local model parameters into several non-overlapping parameter blocks according to the natural boundaries of the weight matrix in the model. Each parameter block corresponds exactly to a weight matrix in the model, and the product of the dimensions of each parameter block equals the total dimension of the model parameters. Taking ResNet-50 in an industrial defect detection scenario as an example, its convolutional layer parameters are divided into 20 64×64 parameter blocks, each approximately 125KB in size. After partitioning, each parameter block is independent of the others, and subsequent gradient calculation, momentum update, and orthogonalization processing are all performed separately at the granularity of a single parameter block.

[0025] S22. In the current local iteration, calculate the batch gradient of each edge sensor device on the current parameter block.

[0026] In step S22, for each parameter block, the present invention retrieves the current batch data from the local dataset of the edge sensor device in each local iteration, calculates the batch gradient on the current parameter block based on the local data distribution of the device, and the batch gradient reflects the direction and magnitude of the loss function gradient of the current parameter block on the current data batch, which is the direct input for subsequent momentum updates.

[0027] S23. The calculated batch gradient and preceding momentum are weighted and accumulated using a momentum accumulation algorithm to obtain the momentum matrix; the expression for the momentum matrix is: in, Indexing edge sensor devices, For parameter block index, This serves as a global training round index. For the local iteration count index of edge sensor devices The momentum coefficient, For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper The gradient of the current batch is calculated during the next local iteration.

[0028] After obtaining the batch gradient, this invention uses a momentum accumulation algorithm to weight and accumulate the current batch gradient with the momentum retained from the previous iteration, thus obtaining the momentum matrix for the current iteration. Specifically, the momentum accumulation algorithm scales the previous momentum using a momentum coefficient and then adds it directly to the current batch gradient. The momentum coefficient is set to 0.98 by default, meaning that 98% of the historical momentum information is retained in each iteration, and the current batch gradient is superimposed on it with full weight.

[0029] The preceding momentum is directly taken from the local momentum issued and initialized by the server during the first local iteration. In each subsequent iteration, the momentum matrix output by the previous iteration is used as the preceding momentum input, forming a momentum accumulation chain of successive iterations. The final output momentum matrix has the same dimension as the weight matrix of the current parameter block. Each element of the matrix comprehensively reflects the weighted superposition result of the gradient information of the parameter block in the historical iterations and the current iteration.

[0030] S3. The momentum matrix is ​​orthogonalized using the Newton-Schultz iterative algorithm to obtain the orthogonalized matrix.

[0031] The expression for the orthogonalization matrix in step S3 is as follows: in, For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the left orthogonal factor matrix. For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the right orthogonal factor matrix. Indicates matrix transpose. This is the Newton-Schultz iteration function, used to approximate orthogonalize the input matrix using a 5th-order polynomial iteration method.

[0032] In step S3, after obtaining the momentum matrix, this invention performs orthogonalization on it. The goal of orthogonalization is to decompose the momentum matrix into an orthogonal matrix, making the parameter update directions orthogonal, thereby optimizing the geometric structure of the weight matrix. Traditional methods directly perform singular value decomposition on the momentum matrix, decomposing it into the product of a left singular vector matrix, a singular value diagonal matrix, and a right singular vector matrix, and then taking the product of the left and right singular vector matrices as the orthogonal matrix. However, singular value decomposition has high computational complexity, which is difficult for edge sensor devices with less than 2GB of memory and relying solely on an ARM CPU to handle.

[0033] To address this, this invention employs the Newton-Schulz iterative algorithm to approximate the orthogonality of the momentum matrix. Specifically, in the approximate orthogonality process, based on the Newton-Schulz iterative algorithm, the momentum matrix is ​​used as the initial input. An iterative formula in the form of a 5th-order polynomial is used to repeatedly update the intermediate matrix. Each iteration uses the output of the previous iteration as input. After a fixed number of iterations, the output matrix converges to an approximation of the singular value decomposition result of the original matrix, ultimately outputting a left orthogonal factor matrix and a right orthogonal factor matrix. Finally, this invention takes the transpose of the left and right orthogonal factor matrices to obtain the orthogonalized matrix. This matrix has the same dimensions as the original momentum matrix, and each element is obtained by gradually approximating the singular value decomposition result through the aforementioned polynomial iterative operation, serving as the direct input for the orthogonalization update direction in the subsequent parameter block update formula.

[0034] S4. Based on the current batch gradient and the momentum matrix, perform momentum gradient alignment scoring on multiple parameter blocks to obtain smooth alignment scores.

[0035] Step S4 further includes: S41. Calculate the cosine similarity between the current batch gradient and the momentum matrix; the expression for the cosine similarity is: in, For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper The gradient of the current batch is calculated during the next local iteration. for and Cosine similarity between them This indicates the matrix transpose.

[0036] After completing the orthogonalization process to obtain the orthogonalized matrix, this invention calculates the cosine similarity between the current batch gradient and momentum matrix for each parameter block, using the current batch gradient and momentum matrix as inputs. The cosine similarity is calculated by transposing the current batch gradient matrix and multiplying it by the momentum matrix, then dividing the result by the product of the norm of the current batch gradient and the norm of the momentum matrix, outputting a scalar value with a range of [-1, +1]. This scalar value reflects the degree of consistency between the current batch gradient direction and the momentum direction on the current parameter block. The closer the value is to +1, the more consistent the two directions are; the closer the value is to -1, the more opposite the two directions are.

[0037] S42. The cosine similarity is scaled using a temperature parameter and mapped using the sigmoid function to obtain the original alignment score; the expression for the original alignment score is: in, For the first Parameter block in the next iteration The original alignment score, This refers to the temperature parameter.

[0038] In step S42, the temperature parameter's control relationship with the original alignment score is as follows: when the cosine similarity approaches +1, the original alignment score approaches the first preset value, and the parameter block update is retained; when the cosine similarity approaches 0, the original alignment score approaches the second preset value, and the parameter block update amplitude is halved; when the cosine similarity approaches -1, the alignment score approaches the third preset value, and the parameter block update is suppressed; wherein, the first preset value is greater than the second preset value, which is greater than the third preset value, and the second preset value is 0.5.

[0039] In step S42, the present invention divides the cosine similarity scalar by the temperature parameter and then feeds it into the sigmoid function for nonlinear mapping to obtain the original alignment score of the parameter block in the current iteration. The temperature parameter is a hyperparameter greater than 0, and its function is to control the concentration of the sigmoid function output value: when the temperature parameter value is small, the sigmoid function is more sensitive to the sign of the cosine similarity, and the output value tends to be close to 0 or 1, forming a hard mask effect; when the temperature parameter value is large, the output value tends to be around 0.5, forming a soft mask effect. Specifically, when the cosine similarity approaches +1, the original alignment score approaches a first preset value, and the parameter block update is retained; when the cosine similarity approaches 0, the original alignment score approaches 0.5, and the parameter block update amplitude is halved; when the cosine similarity approaches -1, the original alignment score approaches a third preset value, and the parameter block update is suppressed.

[0040] S43. The original alignment score is smoothed using an exponential moving average algorithm to obtain a smoothed alignment score.

[0041] In step S43, after obtaining the original alignment score, this invention smooths it using an exponential moving average algorithm to obtain a smoothed alignment score. During the smoothing process, the exponential moving average algorithm retains the historical smoothed alignment score from the previous iteration with a fixed weight of 0.9, and incorporates the original alignment score from the current iteration with a weight of 0.1. The two are then weighted and summed to output the smoothed alignment score for the current iteration. In the first iteration, the historical smoothed alignment score is initialized. In subsequent iterations, the smoothed alignment score output from the previous iteration is used as the input for the historical term, forming a progressively accumulating smoothed alignment chain. This ensures the mask signal remains stable during iteration, avoiding drastic mask fluctuations caused by single-step gradient noise.

[0042] S5. Based on the smooth alignment score and the generated random mask, perform a weighted update on the orthogonalization matrix to obtain the update parameter block.

[0043] Step S5 further includes: S51. For each parameter block, a random mask is generated by sampling using a Bernoulli distribution; the expression for the random mask is: in, For the first Parameter block in the next iteration The corresponding random mask, For sampling probability, For Let be the Bernoulli distribution of the sampling probability.

[0044] In step S51, for each parameter block, a sample is taken from the Bernoulli distribution with a sampling probability of 0.5, and a random mask scalar with a value of 0 or 1 is output. The Bernoulli distribution is a binary probability distribution. Each sample outputs 1 with a probability of 0.5 and 0 with a probability of 0.5, which determines whether the parameter block should be updated in the current iteration.

[0045] S52. Based on the random mask, the smooth alignment score is combined with the random mask to perform weighted scaling on the basic gradient, obtaining the actual update amount; the expression for the actual update amount is: in, For the first Parameter block in the next iteration The actual update volume, For the first Parameter block in the next iteration Smooth alignment score, For the first Parameter block in the next iteration The corresponding base optimizer updates the gradient.

[0046] In step S52, this invention combines the smooth alignment score, random mask, and basic gradient to calculate the actual update amount. The calculation method is as follows: first, the random mask is unbiasedly scaled by dividing by the sampling probability of 0.5, then multiplied by the smooth alignment score, and finally multiplied element-wise by the basic gradient to output the actual update amount, as shown in the formula above. The basic gradient is the gradient update value calculated by the basic optimizer for the current parameter block in the current iteration. When the random mask is 0, the actual update amount is 0, and the parameter block is skipped in the current iteration; when the random mask is 1, the actual update amount is obtained by adjusting the magnitude of the basic gradient using the smooth alignment score.

[0047] S53. The orthogonalized matrix is ​​fused with the weighted basic gradient, and the parameter block is updated by combining the smooth alignment score and the random mask to obtain the updated parameter block; the expression of the updated parameter block is: in, For learning rate, This is the balance coefficient between orthogonalization update and basic gradient. For the first During round training, edge sensor devices In the parameter block Above the first Local model parameters after the next local iteration For the first During round training, edge sensor devices In the parameter block Above the first Local model parameters after the next local iteration For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the left orthogonal factor matrix. For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the right orthogonal factor matrix. Indicates matrix transpose. This refers to the global gradient update issued by the server.

[0048] Based on the actual update amount, the present invention further completes the formal update of the parameter block to obtain the updated parameter block. The specific update method is as follows: the present invention uses the product of the learning rate, the smooth alignment score and the random mask as the overall scaling factor to scale the weighted fusion result of the orthogonalization matrix and the global gradient update, and then subtracts the scaling result from the current parameter block to obtain the iterative local model parameters.

[0049] The fusion ratio between the orthogonalization matrix and the global gradient update is controlled by the balance coefficient, which ranges from 0 to 1. When the balance coefficient approaches zero, the update direction is mainly based on the orthogonalization matrix, and when the balance coefficient approaches 1, the update direction is mainly based on the global gradient update. In addition, the global gradient update is the data sent by the server after the previous round of aggregation, which is directly used as the input of the current parameter block update formula for calculation.

[0050] S6. Upload the parameter block increments and update momentum corresponding to the updated parameter blocks of multiple edge sensor devices to the server. The server then aggregates the parameter block increments and update momentum to obtain the updated global model parameters and the updated global aggregated momentum.

[0051] Step S6 further includes: S61. Multiple edge sensor devices upload the parameter block increment and update momentum after local training to the server. The parameter block increment is the difference between the corresponding parameter blocks before and after local training, and the update momentum is the momentum matrix obtained from the last local iteration.

[0052] In step S61, after each edge sensor device completes all local iterations, for each parameter block, the difference between the final local model parameters at the end of this training round and the initial local model parameters at the beginning of this training round is calculated to obtain the parameter block increment; simultaneously, the momentum matrix output by the last local iteration is used as the update momentum. Finally, the two sets of data are packaged and uploaded to the server. The uploaded content does not include all local model parameters, only the difference and momentum matrix.

[0053] S62. The mean of the parameter block increments uploaded by all edge sensor devices participating in this round of training is aggregated to obtain updated global model parameters; the expression for updating the global model parameters is: in, For the first After the round aggregation is completed, the server updates the obtained parameter block. The corresponding global model parameters are updated. For the first At the start of a training round, the server maintains a block of parameters. The corresponding global model parameters, Indexing edge sensor devices, The size of the subset of edge sensor devices participating in training in each round. For the first During round training, edge sensor devices In the parameter block Complete all The final local model parameters after the next local iteration. For the first At the start of the training round, the edge sensor device In the parameter block The initial local model parameters.

[0054] In step S62, after receiving the parameter block increments uploaded by all edge sensor devices within the subset, the server performs a mean aggregation operation on the device dimension for each parameter block. Specifically, the server calculates the arithmetic mean of the parameter block increments of all edge sensor devices within the subset on the same parameter block element by element to obtain the mean increment of that parameter block. Then, this mean increment is superimposed on the global model parameters maintained by the server in the current round, and the updated global model parameters corresponding to that parameter block are output. The above operation is performed independently for each parameter block. After all parameter blocks have been processed, the server completes the update of the global model parameters for this round.

[0055] S63. Calculate the global gradient update based on the parameter block increment; the expression for the global gradient update is: in, For the first After the round aggregation is completed, the server performs a parameter block analysis. The calculated global gradient update, This represents the total number of local iterations for the edge sensor device. This is the learning rate.

[0056] In step S63, after the server completes the aggregation of the parameter block increment mean, it further calculates the global gradient update based on the aforementioned parameter block increment. The calculation method is as follows: calculate the arithmetic mean of the parameter block increments of all edge sensor devices in the subset on the same parameter block, divide it by the product of the total number of local iterations and the learning rate, take the negative value, and output the global gradient update corresponding to that parameter block. The global gradient update has the same dimension as the parameter block increment, and each element reflects the average gradient direction and magnitude information of all participating devices in this round of training for that parameter block, serving as the direct input to the global gradient update term in the parameter block update formula for the next round.

[0057] S64. The updated momentum uploaded by each edge sensor device is averaged and aggregated to obtain the updated global aggregated momentum; the expression for the updated global aggregated momentum is: in, For the first After round aggregation is completed, the server performs mean aggregation on the updated momentum of each device to obtain the updated global aggregated momentum. For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration.

[0058] In step S64, after receiving the updated momentum uploaded by all edge sensor devices within the subset, the server performs a mean aggregation operation on the device dimension for each parameter block. Specifically, this invention calculates the arithmetic mean element-wise of the updated momentum of all edge sensor devices within the subset on the same parameter block, and outputs the updated global aggregated momentum corresponding to that parameter block. The updated momentum is the momentum matrix output by each edge sensor device in its last local iteration, with dimensions consistent with the weight matrix of the corresponding parameter block. The aggregated updated global aggregated momentum has the same dimension. The above operation is performed independently for each parameter block. After all parameter blocks have been processed, the server completes the update of the global aggregated momentum for this round.

[0059] S65. The server will update the global model parameters, update the global aggregate momentum, and update the global gradient, and uniformly distribute them to multiple edge sensor devices as the initial input for the next round of local training.

[0060] After completing all aggregation calculations in steps S62 to S64, this invention packages the updated global model parameters, updated global aggregate momentum, and global gradient update data corresponding to each parameter block into a single package and distributes it to each edge sensor device participating in the next round of training. Upon receiving these three data items, each edge sensor device assigns the updated global model parameters as the initial local model parameters for the next round, assigns the updated global aggregate momentum as the initial local momentum for the next round, and stores the global gradient update as the input value for the global gradient update term in the parameter block update formula for the next round. This completes one round of federated training iteration. The server then determines whether the training termination condition is met and decides whether to start the next round of training.

[0061] like Figure 2 As shown, the present invention provides a federated learning system for training edge sensor devices in the industrial internet, comprising: a server and an edge sensor device. The server-side includes: The global parameter management module is used to store and maintain updated global model parameters and updated global aggregate momentum, and to send the updated global model parameters and the updated global aggregate momentum to the selected subset of edge sensor devices. The device scheduling module is used to select a subset of edge sensor devices participating in the current round of training from the registered edge sensor devices, and to manage the training status of each edge sensor device. The aggregation calculation module is used to receive parameter block increments and update momentum uploaded by multiple edge sensor devices, and perform mean aggregation operations on the parameter block increments and update momentum respectively to obtain updated global model parameters and updated global aggregate momentum. The server communication interface module is used to send global model parameters and aggregate momentum to a subset of edge sensor devices, and to receive parameter block increments and updated momentum uploaded by each edge sensor device. Edge sensor devices include: The local initialization module is used to receive global model parameters and aggregate momentum sent by the server, initialize the global model parameters as local model parameters, and initialize the aggregate momentum as local momentum; The parameter block partitioning module is used to divide the local model parameters into multiple parameter blocks according to the boundary of the weight matrix, providing input for the independent calculation of multiple parameter blocks in the future; The momentum accumulation module is used to update the momentum matrix in each local iteration based on the current batch gradient and the previous momentum using the momentum accumulation algorithm. The orthogonalization calculation module is used to orthogonalize the momentum matrix using the Newton-Schultz iterative algorithm to obtain an orthogonal matrix; The alignment scoring module is used to perform momentum gradient alignment scoring on multiple parameter blocks based on the current batch gradient and the momentum matrix to obtain a smooth alignment score. The random mask generation module is used to sample and generate a random mask for each parameter block according to the Bernoulli distribution; The parameter block update module is used to fuse the orthogonalization matrix with the basic gradient weighted fusion, and combine the smooth alignment score with the random mask to complete the weighted update of each parameter block and obtain the updated parameter block; The device communication interface module is used to upload the parameter block increments and update momentum corresponding to the updated parameter blocks of each parameter block after local training to the server, and to receive the updated global model parameters and updated global aggregate momentum sent by the server.

[0062] The present invention also provides a federated learning device for training edge sensor devices in the Industrial Internet, comprising: a memory and at least one processor, wherein the memory stores instructions; the processor invokes the instructions in the memory to cause the federated learning device for training edge sensor devices in the Industrial Internet to execute a federated learning method for training edge sensor devices in the Industrial Internet as described in any of the preceding claims.

[0063] The present invention also provides a computer-readable storage medium storing instructions that, when executed by a processor, implement a federated learning method for training edge sensor devices for the Industrial Internet as described in any of the preceding claims.

[0064] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0065] In practice, this invention updates only 50% of the parameter blocks using a random mask, and reduces orthogonalization overhead through Newton-Schulz iteration, shortening the training time for a single edge sensor device and accelerating the training process. For transmitting parameter block increments, only 50% of the parameters change, reducing the amount of communication data. Furthermore, momentum gradient alignment suppresses heavy-tailed noise, and matrix orthogonalization enhances the rationality of the weight structure, improving model accuracy. The parameter block partitioning and lightweight computation of this invention enable large models to be deployed on edge sensor devices with limited memory and CPU resources.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated learning method for training edge sensor devices in the Industrial Internet, characterized in that, include: S1. Distribute the global model parameters and aggregate momentum to the selected subset of edge sensor devices, and have multiple edge sensor devices initialize and obtain local model parameters and local momentum respectively; S2. Divide the local model parameters into multiple parameter blocks, and update the momentum of the current batch gradient using the momentum accumulation algorithm for the multiple parameter blocks to obtain the momentum matrix; S3. The momentum matrix is ​​orthogonalized using the Newton-Schultz iterative algorithm to obtain the orthogonalized matrix; S4. Based on the current batch gradient and the momentum matrix, perform momentum gradient alignment scoring on multiple parameter blocks to obtain smooth alignment scores; S5. Based on the smooth alignment score and the generated random mask, perform a weighted update on the orthogonalization matrix to obtain the update parameter block; S6. Upload the parameter block increments and update momentum corresponding to the updated parameter blocks of multiple edge sensor devices to the server. The server then aggregates the parameter block increments and update momentum to obtain the updated global model parameters and the updated global aggregate momentum.

2. The federated learning method for training edge sensor devices in the Industrial Internet according to claim 1, characterized in that, Step S2 further includes: S21. Divide the local model parameters into multiple non-overlapping parameter blocks, each parameter block corresponding to a weight matrix in the model; S22. In the current local iteration, calculate the batch gradient of each edge sensor device on the current parameter block; S23. The calculated batch gradient and preceding momentum are weighted and accumulated using a momentum accumulation algorithm to obtain the momentum matrix; the expression for the momentum matrix is: in, Indexing edge sensor devices, For parameter block index, This serves as a global training round index. For the local iteration count index of edge sensor devices The momentum coefficient, For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper The gradient of the current batch is calculated during the next local iteration.

3. The federated learning method for training edge sensor devices in the Industrial Internet according to claim 1, characterized in that, The expression for the orthogonalization matrix in step S3 is: in, For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the left orthogonal factor matrix. For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the right orthogonal factor matrix. Indicates matrix transpose. This is the Newton-Schultz iteration function, used to approximate orthogonalize the input matrix using a 5th-order polynomial iteration method.

4. The federated learning method for training edge sensor devices in the Industrial Internet according to claim 1, characterized in that, Step S4 further includes: S41. Calculate the cosine similarity between the current batch gradient and the momentum matrix; the expression for the cosine similarity is: in, For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration For the first During round training, edge sensor devices In the parameter block Upper The gradient of the current batch is calculated during the next local iteration. for and Cosine similarity between them Indicates matrix transpose; S42. The cosine similarity is scaled using a temperature parameter and mapped using the sigmoid function to obtain the original alignment score; the expression for the original alignment score is: in, For the first Parameter block in the next iteration The original alignment score, For temperature parameters; S43. The original alignment score is smoothed by the exponential moving average algorithm to obtain a smoothed alignment score.

5. The federated learning method for training edge sensor devices in the Industrial Internet according to claim 1, characterized in that, In step S42, the temperature parameter controls the original alignment score as follows: when the cosine similarity approaches +1, the original alignment score approaches the first preset value, and the parameter block update is retained; when the cosine similarity approaches 0, the original alignment score approaches the second preset value, and the parameter block update amplitude is halved; when the cosine similarity approaches -1, the alignment score approaches the third preset value, and the parameter block update is suppressed. Wherein, the first preset value is greater than the second preset value, which is greater than the third preset value, and the second preset value is 0.

5.

6. The federated learning method for training edge sensor devices in the Industrial Internet according to claim 1, characterized in that, Step S5 further includes: S51. For each parameter block, a random mask is generated by sampling using a Bernoulli distribution; the expression for the random mask is: in, For the first Parameter block in the next iteration The corresponding random mask, For sampling probability, For The sampling probability follows a Bernoulli distribution; S52. Based on the random mask, the smooth alignment score is combined with the random mask to perform weighted scaling on the basic gradient, obtaining the actual update amount; the expression for the actual update amount is: in, For the first Parameter block in the next iteration The actual update volume, For the first Parameter block in the next iteration Smooth alignment score, For the first Parameter block in the next iteration The corresponding base optimizer updates the gradient; S53. The orthogonalized matrix is ​​fused with the weighted basic gradient, and the parameter block is updated by combining the smooth alignment score and the random mask to obtain the updated parameter block; the expression of the updated parameter block is: in, For learning rate, This is the balance coefficient between orthogonalization update and basic gradient. For the first During round training, edge sensor devices In the parameter block Above the first Local model parameters after the next local iteration For the first During round training, edge sensor devices In the parameter block Above the first Local model parameters after the next local iteration For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the left orthogonal factor matrix. For the first During round training, edge sensor devices In the parameter block Upper In the next iteration, the momentum matrix is ​​orthogonalized using Newton-Schulz iteration to obtain the right orthogonal factor matrix. Indicates matrix transpose. This refers to the global gradient update issued by the server.

7. The federated learning method for training edge sensor devices in the Industrial Internet according to claim 1, characterized in that, Step S6 further includes: S61. Multiple edge sensor devices upload the parameter block increment and update momentum after local training to the server. The parameter block increment is the difference between the corresponding parameter blocks before and after local training, and the update momentum is the momentum matrix obtained from the last local iteration. S62. The mean of the parameter block increments uploaded by all edge sensor devices participating in this round of training is aggregated to obtain updated global model parameters; the expression for updating the global model parameters is: in, For the first After the round aggregation is completed, the server updates the obtained parameter block. The corresponding global model parameters are updated. For the first At the start of a training round, the server maintains a block of parameters. The corresponding global model parameters, Indexing edge sensor devices, The size of the subset of edge sensor devices participating in training in each round. For the first During round training, edge sensor devices In the parameter block Complete all The final local model parameters after the next local iteration. For the first At the start of the training round, the edge sensor device In the parameter block Initial local model parameters; S63. Calculate the global gradient update based on the parameter block increment; the expression for the global gradient update is: in, For the first After the round aggregation is completed, the server performs a parameter block analysis. The calculated global gradient update, This represents the total number of local iterations for the edge sensor device. The learning rate; S64. The updated momentum uploaded by each edge sensor device is averaged and aggregated to obtain the updated global aggregated momentum; the expression for the updated global aggregated momentum is: in, For the first After round aggregation is completed, the server performs mean aggregation on the updated momentum of each device to obtain the updated global aggregated momentum. For the first During round training, edge sensor devices In the parameter block Above the first The momentum matrix accumulated after each local iteration; S65. The server will update the global model parameters, update the global aggregate momentum, and update the global gradient, and uniformly distribute them to multiple edge sensor devices as the initial input for the next round of local training.

8. A federated learning system for training edge sensor devices in the Industrial Internet, characterized in that, include: Server-side and edge sensor device-side; The server-side includes: The global parameter management module is used to store and maintain updated global model parameters and updated global aggregate momentum, and to send the updated global model parameters and the updated global aggregate momentum to the selected subset of edge sensor devices. The device scheduling module is used to select a subset of edge sensor devices participating in the current round of training from the registered edge sensor devices, and to manage the training status of each edge sensor device. The aggregation calculation module is used to receive parameter block increments and update momentum uploaded by multiple edge sensor devices, and perform mean aggregation operations on the parameter block increments and update momentum respectively to obtain updated global model parameters and updated global aggregate momentum. The server communication interface module is used to send global model parameters and aggregate momentum to a subset of edge sensor devices, and to receive parameter block increments and updated momentum uploaded by each edge sensor device. Edge sensor devices include: The local initialization module is used to receive global model parameters and aggregate momentum sent by the server, initialize the global model parameters as local model parameters, and initialize the aggregate momentum as local momentum; The parameter block partitioning module is used to divide the local model parameters into multiple parameter blocks according to the boundary of the weight matrix, providing input for the independent calculation of multiple parameter blocks in the future; The momentum accumulation module is used to update the momentum matrix in each local iteration based on the current batch gradient and the previous momentum using the momentum accumulation algorithm. The orthogonalization calculation module is used to orthogonalize the momentum matrix using the Newton-Schultz iterative algorithm to obtain an orthogonal matrix; The alignment scoring module is used to perform momentum gradient alignment scoring on multiple parameter blocks based on the current batch gradient and the momentum matrix to obtain a smooth alignment score. The random mask generation module is used to sample and generate a random mask for each parameter block according to the Bernoulli distribution; The parameter block update module is used to fuse the orthogonalization matrix with the basic gradient weighted fusion, and combine the smooth alignment score with the random mask to complete the weighted update of each parameter block and obtain the updated parameter block; The device communication interface module is used to upload the parameter block increments and update momentum corresponding to the updated parameter blocks of each parameter block after local training to the server, and to receive the updated global model parameters and updated global aggregate momentum sent by the server.

9. A federated learning device for training edge sensor devices in the Industrial Internet, characterized in that, include: A memory and at least one processor, wherein the memory stores instructions; The processor invokes the instructions in the memory to cause a federated learning device for training industrial internet edge sensor devices to execute a federated learning method for training industrial internet edge sensor devices as described in any of the above.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a processor, implement a federated learning method for training edge sensor devices in the Industrial Internet as described in any of the preceding claims.

Images