Industrial Internet of Things federal learning method and system based on principal component analysis

Abstract

The invention provides an industrial Internet of Things federated learning method and system based on principal component analysis, and relates to the technical field of intelligent industrial manufacturing and model valuation, and the method comprises the steps: obtaining model parameters uploaded by all factory participants, and constructing a model parameter matrix through the model parameters; performing data dimension reduction on the model parameter matrix by adopting a principal component analysis method, calculating a covariance matrix of the model parameter matrix, and performing eigenvalue decomposition on the covariance matrix to obtain a principal component of the model parameter matrix; calculating similarity and difference between the specific local model and other local models based on principal components of the model parameter matrix; according to the similarity and difference results obtained through calculation, the minimum value of the similarity and the difference serves as the model contribution degree; and based on the model contribution degree, using a SoftMax-like function to perform conversion and calculate a corresponding weight, and applying the weight obtained by conversion of the model contribution degree to the aggregated global model so as to obtain a global model under a new weight.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent industrial manufacturing and model valuation technology, and specifically to an industrial Internet of Things federated learning method and system based on principal component analysis. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the development of machine learning and the increasing demand for personal data, the scope of data collection is gradually expanding. However, the traditional machine learning framework that collects data first and then trains uniformly has been increasingly restricted as people's need for data privacy protection grows. Therefore, a distributed machine learning framework called federated learning came into being. Federated learning uses local data for distributed training in the local area of ​​the participants and uploads the trained model to the central server for aggregation. Finally, after multiple iterations, the global model is issued to the participants and rewards are given according to the participants' contributions. In traditional federated learning, the contribution of participants is determined by the amount of data reported by the participants. However, since the local training process of the participants is highly autonomous, the amount of data reported cannot be guaranteed to be true. How to determine the true contribution of the participants is a technical problem in the current application of federated learning.

[0004] Federated learning methods can be used in the Industrial Internet of Things to train machine learning models based on data from multiple factories to achieve predictive maintenance or identify common patterns and trends to optimize the operations of all factories. For factories within their organizations, these data may have privacy issues and are not suitable for direct transmission. Using the FL method to train machine learning models means that the data is trained locally on edge devices, so that the privacy and security of the data can be established. In this process, the contribution of the factory needs to be evaluated to incentivize the factory to participate in federated learning honestly and credibly.

[0005] However, in real scenarios, there are likely to be malicious actors who take strategic actions. These malicious actors participate in certain attack methods during the training process, such as backdoor attacks or fabricating model parameters. These malicious actors not only affect the performance of the global model, but also bring many security risks.

[0006] In the prior art, in order to solve the problem of measuring the contribution of different factories to the model in the federated learning of the Industrial Internet of Things, some studies use the Shapley value method, which accurately measures the contribution of the model uploaded by the participants in the presence of a standard public test data set. However, in real scenarios, it is difficult to obtain a standard public data set for testing; at the same time, the complexity of calculating the Shapley value grows exponentially with the amount of data, making this method difficult to play a role in practice. In addition, there are some studies that measure the contribution of different factories to the model and detect malicious participants by calculating the similarity between models based on model parameters, such as cosine similarity or Euclidean distance. These methods rely on the assumption that similar models are conducive to federated learning and that the models uploaded by malicious participants are quite different from other models. However, these assumptions may not hold true in practice, which also brings some obstacles to application. Summary of the invention

[0007] In order to solve the above problems, the present disclosure proposes an industrial Internet of Things federated learning method and system based on principal component analysis. Based on the principal component analysis method, the similarity and difference between a participant and other participants are calculated to obtain the contribution measurement of the federated learning model. Through the aggregation weight calculation method, the weight obtained by contribution conversion is applied to the aggregation global model to obtain the global model under the new weight, which can help the federated learning server reduce the adverse effects of malicious nodes and low-quality nodes and better improve the performance of federated learning.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions:

[0009] The industrial IoT federated learning method based on principal component analysis includes:

[0010] Obtain the model parameters uploaded by all factory participants and use the model parameters to construct a model parameter matrix;

[0011] The principal component analysis method is used to reduce the data dimension of the model parameter matrix, and the covariance matrix of the model parameter matrix is ​​calculated. The principal components of the model parameter matrix are obtained by using eigenvalue decomposition on the covariance matrix.

[0012] Calculate similarities and differences between a specific local model and other local models based on the principal components of the model parameter matrix;

[0013] According to the calculated similarity and difference results, the minimum value of the similarity and difference is taken as the model contribution;

[0014] Based on the model contribution, a SoftMax-like function is used to transform and calculate the corresponding weights. The weights obtained by transforming the model contribution are applied to the aggregated global model to obtain the global model under the new weights.

[0015] According to some embodiments, the present disclosure adopts the following technical solutions:

[0016] The industrial IoT federated learning system based on principal component analysis includes:

[0017] The parameter acquisition module is used to obtain the model parameters uploaded by all factory participants and construct the model parameter matrix using the model parameters;

[0018] The principal component analysis module is used to perform data dimension reduction on the model parameter matrix using the principal component analysis method, calculate the covariance matrix of the model parameter matrix, and use eigenvalue decomposition on the covariance matrix to obtain the principal components of the model parameter matrix;

[0019] A metric calculation module is used to calculate the similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix; according to the calculated similarity and difference results, the minimum value of the similarity and the difference is used as the model contribution;

[0020] The aggregation module is used to transform and calculate the corresponding weights based on the model contribution using a SoftMax-like function, and apply the weights obtained from the model contribution transformation to the aggregated global model to obtain the global model under the new weights.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] A computer program product includes a computer program, and when the computer program is executed by a processor, the industrial Internet of Things federated learning method based on principal component analysis is implemented.

[0023] According to some embodiments, the present disclosure adopts the following technical solutions:

[0024] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the industrial Internet of Things federated learning method based on principal component analysis is implemented.

[0025] According to some embodiments, the present disclosure adopts the following technical solutions:

[0026] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the industrial Internet of Things federated learning method based on principal component analysis.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The industrial Internet of Things federated learning method based on principal component analysis disclosed in the present invention is based on a method for calculating similarities and differences, which is assisted by principal component analysis. First, a model parameter matrix is ​​generated, and the principal component vectors and corresponding eigenvalues ​​of the model parameter matrix are obtained by performing eigenvalue decomposition on its covariance matrix. Then, the corresponding vectors of other model parameter matrices are calculated based on the principal component vectors to obtain the required parameters, so that the calculation of similarities and differences can be completed based on these parameter operations; an agency relationship is established between the model parameter matrix and the data matrix, so that the similarities and differences calculated based on the model parameters can accurately describe the similarities and differences between data distributions, so that the contribution of participants can accurately describe the contribution size of their data distributions.

[0029] The industrial Internet of Things federated learning method based on principal component analysis disclosed in the present invention converts model parameters into matrices based on eigenvalue decomposition, and then calculates eigenvalues ​​and eigenvectors as principal components; performs operations based on principal components to realize the proxy of model parameters for the distribution of original data; based on the vectors of principal components and other model parameters in the direction of principal components, a similarity and difference calculation method is proposed by the present invention to calculate the contribution of the model uploaded by the participant; based on the contribution, it is converted into aggregation weights and applied to the aggregation process to finally obtain a global model. Based on the contribution size, the server disclosed in the present invention can detect low-quality participants and malicious participants and give them lower weights, thereby reducing their negative impact.

[0030] The principal component analysis-based industrial Internet of Things federated learning method disclosed in the present invention trains a machine learning model based on data from multiple factories, evaluates the contribution of factories, realizes predictive maintenance or identifies common patterns and trends to optimize the operation of all factories, so as to incentivize factories to participate in federated learning honestly and credibly. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0032] Figure 1 A schematic diagram of a scene in which the similarity is smaller than the difference according to an embodiment of the present disclosure;

[0033] Figure 2 This is a schematic diagram of a scenario in which the difference is smaller than the similarity according to an embodiment of the present disclosure.

[0034] The direction of the vector is the direction of the two principal components, and the length of the vector represents the size of the eigenvalue in the direction of the principal component. The yellow rectangle represents the similarity, and its sides are the intersection of the vectors in the same direction; the gray rectangle represents the difference, and its sides are the difference of the vectors in the same direction. According to the size of the similarity and difference, the smaller value is regarded as the contribution, that is, the red shaded part. DETAILED DESCRIPTION

[0035] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] Example 1

[0039] In one embodiment of the present disclosure, a method for federated learning of industrial Internet of Things based on principal component analysis is provided, including:

[0040] Step 1: Obtain the model parameters uploaded by all factory participants and use the model parameters to build a model parameter matrix;

[0041] Step 2: Use principal component analysis to reduce the data dimension of the model parameter matrix, calculate the covariance matrix of the model parameter matrix, and use eigenvalue decomposition on the covariance matrix to obtain the principal components of the model parameter matrix;

[0042] Step 3: Calculate the similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix;

[0043] Step 4: Based on the calculated similarity and difference results, the minimum value of the similarity and difference is taken as the model contribution;

[0044] Step 5: Based on the model contribution, use a SoftMax-like function to transform and calculate the corresponding weights, and apply the weights obtained from the model contribution transformation to the aggregated global model to obtain the global model under the new weights.

[0045] As an embodiment, the present invention discloses an industrial Internet of Things federated learning method based on principal component analysis, a federated learning model contribution measurement technology calculation method based on principal component analysis, and an aggregation weight calculation method, which can help the federated learning server reduce the adverse effects of malicious nodes and low-quality nodes and better improve the performance of federated learning. The specific implementation process is as follows:

[0046] Step 1: Obtain the model parameters uploaded by all factory participants and use the model parameters to build a model parameter matrix;

[0047] Specifically, the model parameters are converted into matrices. First, consider some classic and common structures. For a two-dimensional fully connected neural network, it can be regarded as a matrix itself; for a four-dimensional convolutional neural network, the convolution kernel is unfolded as a data vector, so that each convolution kernel can be stacked to obtain a matrix in which each column is an unfolded convolution kernel. In addition, for some more complex structures, such as transformers, they can be regarded as a collection of matrices, from which matrices can be extracted. The composition includes a fully connected neural network, one of which can be selected as a representative, or each fully connected neural network can be unfolded as a vector and spliced ​​to form a matrix.

[0048] Step 2: Use principal component analysis to reduce the data dimension of the model parameter matrix, calculate the covariance matrix of the model parameter matrix, and use eigenvalue decomposition on the covariance matrix to obtain the principal components of the model parameter matrix;

[0049] Furthermore, the covariance matrix is ​​calculated based on the model parameter matrix, and the eigenvalue decomposition is performed on the covariance matrix to obtain its d eigenvectors and eigenvalue λ0 as principal components.

[0050] Specifically, for the model parameters uploaded by participant i, its covariance matrix is ​​first calculated:

[0051]

[0052] Among them, Cov i represents the covariance matrix of i, m is the dimension of the model parameter matrix, w i is the model parameter matrix.

[0053] Then perform eigenvalue decomposition (EVD) on the covariance matrix, calculate the characteristic equation, and obtain the eigenvalue. The characteristic equation is:

[0054] |Cov i -λE|=0

[0055] Then, each eigenvalue is substituted into the linear equation system to obtain the eigenvector and eigenvalue, and the d eigenvalues ​​with the largest eigenvalues ​​are selected as the principal components. Substituting each eigenvalue into the linear equation system is:

[0056] (Cov i -λE)x=0

[0057] The d components with the largest eigenvalues ​​are finally selected as the main components: {λ1,λ2,…,λ d},{v1,v2,…,v d}.

[0058] Furthermore, for the model parameter matrix constructed by the model parameters uploaded by other factory participants, the covariance matrix is ​​calculated in turn, and then the eigenvalue size of the covariance matrix in the direction of a specific principal component vector is calculated.

[0059] Specifically, for the uploaded model parameters of other selected factory participants, their covariance matrices are calculated in turn:

[0060]

[0061] Among them, Cov j represents the covariance matrix of j, m is the dimension of the model parameter matrix, w i ,w j is the model parameter matrix.

[0062] Then calculate the vectors and eigenvalues ​​of the covariance matrix of other model parameter matrices on the principal components of the specific model parameter matrix.

[0063] λ j,l ←||Cov j v l ||

[0064] Among them, λ j,l represents the eigenvalue size of the model parameter matrix of participant j on principal component l, Cov j is the model parameter matrix of participant j, v l is the principal component l of the parameter matrix of the compared participants.

[0065] Therefore, for each principal component, the eigenvalue size of the model parameter matrix in that direction is obtained.

[0066] Step 3: Calculate the similarity and difference between the specific local model and other local models based on the principal components of the model parameter matrix, including: comparing the eigenvalues ​​of the specific local model and other local models pairwise, calculating the similarity and difference, and taking the product of the intersection of the eigenvectors as the similarity and the product of the difference as the difference.

[0067] Specifically, based on the eigenvalue size of the model parameter matrix in the direction of a specific principal component vector, the eigenvalues ​​of participant i and other participants j are compared to calculate the similarity and difference. The product of the intersection of their eigenvectors is used as the similarity, and the product of the difference is used as the difference. The final product value is rooted to prevent its value from being too small, such as Figure 1 shown.

[0068] The product of the intersection of the feature vectors is used as the similarity, which is:

[0069]

[0070] Among them, Sim i,j represents the similarity value between participants i and j, d is the number of selected principal components, and λ l is the eigenvalue of the lth principal component of participant i, λ j,l Represents the eigenvalues ​​of the parameter matrix of participant j on principal component l.

[0071] The product of the difference parts is taken as the difference degree, which is:

[0072]

[0073] Among them, Div i,j represents the similarity value between participants i and j, d is the number of selected principal components, and λ l is the eigenvalue of the lth principal component of participant i, λ j,l Represents the eigenvalues ​​of the parameter matrix of participant j on principal component l.

[0074] Furthermore, according to the calculated similarity and difference results, the minimum value of the similarity and the difference is taken as the model contribution.

[0075] Step 4: Based on the model contribution, use a SoftMax-like function to transform and calculate the corresponding weights, and apply the weights obtained from the model contribution transformation to the aggregated global model to obtain the global model under the new weights.

[0076] Specifically, the smaller value of the similarity and the difference is used as the contribution degree to ensure that the similarity and the difference of participants with high contributions are not too small.

[0077] The calculated contribution is then applied to the aggregation process of federated learning and converted into weights using a softmax-like function:

[0078]

[0079] Among them, e irepresents the aggregation weight of participant i, α is the temperature coefficient used to adjust the variance, which is a public parameter, and C i It represents the contribution value of participant i, and k is the number of participants in the current aggregation round.

[0080] Then, a weighted aggregation process is performed based on the weights to obtain the global model: Model global =∑e i M i .

[0081] Example 2

[0082] In one embodiment of the present disclosure, an industrial Internet of Things federated learning system based on principal component analysis is provided, including:

[0083] The parameter acquisition module is used to obtain the model parameters uploaded by all factory participants and construct the model parameter matrix using the model parameters;

[0084] The principal component analysis module is used to perform data dimension reduction on the model parameter matrix using the principal component analysis method, calculate the covariance matrix of the model parameter matrix, and use eigenvalue decomposition on the covariance matrix to obtain the principal components of the model parameter matrix;

[0085] A metric calculation module is used to calculate the similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix; according to the calculated similarity and difference results, the minimum value of the similarity and the difference is used as the model contribution;

[0086] The aggregation module is used to transform and calculate the corresponding weights based on the model contribution using a SoftMax-like function, and apply the weights obtained from the model contribution transformation to the aggregated global model to obtain the global model under the new weights.

[0087] Example 3

[0088] In one embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the industrial Internet of Things federated learning method based on principal component analysis is implemented.

[0089] Example 4

[0090] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the industrial Internet of Things federated learning method based on principal component analysis is implemented.

[0091] Example 5

[0092] In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the industrial Internet of Things federated learning method based on principal component analysis.

[0093] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. The industrial Internet of Things federated learning method based on principal component analysis is characterized by: include: Obtain the model parameters uploaded by all factory participants and use the model parameters to build a model parameter matrix; The principal component analysis method is used to reduce the data dimension of the model parameter matrix, and the covariance matrix of the model parameter matrix is ​​calculated. The principal components of the model parameter matrix are obtained by using eigenvalue decomposition on the covariance matrix. Calculate similarities and differences between a specific local model and other local models based on the principal components of the model parameter matrix; According to the calculated similarity and difference results, the minimum value of the similarity and difference is taken as the model contribution; Based on the model contribution, a SoftMax-like function is used to transform and calculate the corresponding weights. The weights obtained by transforming the model contribution are applied to the aggregated global model to obtain the global model under the new weights.

2. The industrial Internet of Things federated learning method based on principal component analysis as claimed in claim 1, characterized in that: The principal component analysis method is used to reduce the data dimension of the specific model parameter matrix, and its covariance matrix is ​​calculated as: Among them, Cov i represents the covariance matrix of i, m is the dimension of the model parameter matrix, w i is the model parameter matrix; Then, the covariance matrix is ​​decomposed by SVD eigenvalue to obtain eigenvectors and eigenvalues, and the d eigenvectors with the largest eigenvalues ​​and eigenvalue λ0 are selected in turn as principal components.

3. The industrial Internet of Things federated learning method based on principal component analysis as claimed in claim 2, characterized in that: For the model parameter matrix constructed by the model parameters uploaded by other factory participants, the covariance matrix is ​​calculated in turn, and then the eigenvalue size of the covariance matrix in the direction of the specific principal component vector is calculated.

4. The industrial Internet of Things federated learning method based on principal component analysis as claimed in claim 1, characterized in that: The similarity and difference between the specific local model and other local models are calculated based on the principal components of the model parameter matrix, including: calculating the vector of the covariance matrix of the other model parameter matrix on each principal component of the specific model parameter matrix and its eigenvalue size, comparing the eigenvalues ​​of the specific local model and other local models pairwise, calculating the similarity and difference, and taking the product of the intersection of the eigenvectors as the similarity, and the product of the difference as the difference.

5. The industrial Internet of Things federated learning method based on principal component analysis as claimed in claim 4, characterized in that: The product of the intersection of the feature vectors is used as the similarity, which is: The product of the difference parts is taken as the difference degree, which is: Among them, Sim i,j and Div i,j denote the similarity and difference between participants i and j, d is the number of selected principal components, and λ l is the eigenvalue of the lth principal component of participant i, λ j,l Represents the eigenvalues ​​of the parameter matrix of participant j on principal component l.

6. The industrial Internet of Things federated learning method based on principal component analysis according to claim 1, characterized in that: The smaller value between the similarity and the difference is used as the contribution, and the calculated contribution is applied to the aggregation process of federated learning, and converted into a weight using a softmax-like function: Among them, e i represents the aggregation weight of participant i, α is the temperature coefficient used to adjust the variance, which is a public parameter, and C i It represents the contribution value of participant i, and k is the number of participants in the current aggregation round.

7. Industrial Internet of Things federated learning system based on principal component analysis, characterized by: include: The parameter acquisition module is used to obtain the model parameters uploaded by all factory participants and construct the model parameter matrix using the model parameters; The principal component analysis module is used to perform data dimension reduction on the model parameter matrix using the principal component analysis method, calculate the covariance matrix of the model parameter matrix, and use eigenvalue decomposition on the covariance matrix to obtain the principal components of the model parameter matrix; A metric calculation module for calculating similarities and differences between a specific local model and other local models based on the principal components of the model parameter matrix; According to the calculated similarity and difference results, the minimum value of the similarity and difference is taken as the model contribution; The aggregation module is used to transform and calculate the corresponding weights based on the model contribution using a SoftMax-like function, and apply the weights obtained from the model contribution transformation to the aggregated global model to obtain the global model under the new weights.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the industrial Internet of Things federated learning method based on principal component analysis described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the industrial Internet of Things federated learning method based on principal component analysis as described in any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the industrial Internet of Things federated learning method based on principal component analysis as described in any one of claims 1 to 6.

Images