Industrial internet of things federated learning method and system based on principal component analysis

Abstract

The disclosure provides an industrial internet of things federated learning method and system based on principal component analysis, relates to the technical field of intelligent industrial manufacturing and model evaluation, and comprises the following steps: acquiring model parameters uploaded by all factory participants, and constructing a model parameter matrix by using the model parameters; performing data dimension reduction on the model parameter matrix by using a principal component analysis method, calculating a covariance matrix of the model parameter matrix, and obtaining principal components of the model parameter matrix by using eigenvalue decomposition on the covariance matrix; calculating similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix; taking the minimum value in the similarity and difference as model contribution degree according to the calculated similarity and difference results; transforming and calculating corresponding weights by using a SoftMax function based on the model contribution degree, and applying the weights obtained by transforming the model contribution degree to an aggregated global model, so as to obtain a global model under new weights.

Description

Technical Field

[0001] This disclosure relates to the fields of intelligent industrial manufacturing and model valuation technology, specifically to an industrial IoT federated learning method and system based on principal component analysis. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the development of machine learning and the increasing demand for personal data, the scope of data collection is gradually expanding. However, traditional machine learning frameworks that collect data first and then train it uniformly are facing increasing limitations due to growing concerns about data privacy. Therefore, federated learning, a distributed machine learning framework, has emerged. Federated learning involves participants training locally using local data, uploading the trained models, and then aggregating them on a central server. After multiple iterations, a global model is distributed to the participants, and rewards are given based on their contributions. In traditional federated learning, a participant's contribution is determined by the amount of data they report. However, because the local training process is highly autonomous, the amount of data reported cannot be guaranteed to be genuine. Determining the true contribution of a participant remains a technical challenge in current federated learning applications.

[0004] Federated learning methods can be used in the Industrial Internet of Things (IIoT) to train machine learning models based on data from multiple factories, enabling predictive maintenance or identifying common patterns and trends to optimize operations across all factories. For factories within an organization, this data may pose privacy concerns and is not suitable for direct transmission. Training machine learning models using federated learning methods means that the data is trained locally on edge devices, thus establishing data privacy and security. During this process, the contributions of each factory need to be evaluated to incentivize their honest and trustworthy participation in federated learning.

[0005] However, in real-world scenarios, malicious actors may engage in strategic actions. These actors may use attack methods during the training process, such as backdoor attacks or manipulating model parameters. These malicious actors not only affect the overall model performance but also introduce numerous security vulnerabilities.

[0006] In existing technologies, to address the problem of measuring the contribution of different factories to the model in federated learning of the Industrial Internet of Things (IIoT), some studies have used the Shapley value method. This method accurately measures the contribution of the model uploaded by participants when a standard public test dataset exists. However, in real-world scenarios, it is difficult to obtain a standard public dataset for testing; moreover, the complexity of calculating the Shapley value increases exponentially with the amount of data, making this method impractical. In addition, some studies have started from model parameters, measuring the contribution of different factories to the model and detecting malicious participants by calculating the similarity between models, such as cosine similarity or Euclidean distance. These methods rely on assumptions such as similar models being beneficial to federated learning and malicious participants uploading models that are significantly different from other models. However, these assumptions may not hold true in practice, leading to some obstacles in application. Summary of the Invention

[0007] To address the aforementioned issues, this disclosure proposes an industrial IoT federated learning method and system based on principal component analysis. Using principal component analysis, the similarity and difference between a participant and other participants are calculated to obtain a contribution metric for the federated learning model. Through an aggregation weight calculation method, the weights derived from the contribution are applied to the aggregated global model, resulting in a global model with the new weights. This approach helps the federated learning server mitigate the adverse effects of malicious and low-quality nodes, thereby improving federated learning performance.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions:

[0009] Federated learning methods for the Industrial Internet of Things based on principal component analysis include:

[0010] Obtain the model parameters uploaded by all factory participants, and construct a model parameter matrix using the model parameters;

[0011] Principal component analysis is used to reduce the dimensionality of the model parameter matrix and calculate the covariance matrix of the model parameter matrix. Eigenvalue decomposition is then used to obtain the principal components of the model parameter matrix from the covariance matrix.

[0012] Calculate the similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix;

[0013] Based on the calculated similarity and difference results, the minimum value between the similarity and difference is taken as the model contribution.

[0014] Based on the model contribution, a SoftMax-like function is used to transform and calculate the corresponding weights. The weights obtained from the transformation of the model contribution are then applied to the aggregated global model to obtain the global model with the new weights.

[0015] According to some embodiments, the present disclosure adopts the following technical solutions:

[0016] A federated learning system for industrial IoT based on principal component analysis includes:

[0017] The parameter acquisition module is used to acquire the model parameters uploaded by all factory participants and construct a model parameter matrix using the model parameters;

[0018] The principal component analysis module is used to perform dimensionality reduction on the model parameter matrix using the principal component analysis method, calculate the covariance matrix of the model parameter matrix, and obtain the principal components of the model parameter matrix by eigenvalue decomposition on the covariance matrix.

[0019] The metrics calculation module is used to calculate the similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix; based on the calculated similarity and difference results, the minimum value of the similarity and difference is taken as the model contribution.

[0020] The aggregation module is used to transform and calculate the corresponding weights based on the model contribution using a SoftMax-like function. The weights obtained from the transformation of the model contribution are then applied to the aggregated global model to obtain the global model with the new weights.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] A computer program product includes a computer program that, when executed by a processor, implements the aforementioned principal component analysis-based federated learning method for industrial IoT.

[0023] According to some embodiments, the present disclosure adopts the following technical solutions:

[0024] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned industrial IoT federated learning method based on principal component analysis.

[0025] According to some embodiments, the present disclosure adopts the following technical solutions:

[0026] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the industrial IoT federated learning method based on principal component analysis.

[0027] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0028] This disclosed industrial IoT federated learning method, based on principal component analysis (PCA), utilizes PCA to calculate similarity and dissimilarity. First, it generates the model parameter matrix. Then, by performing eigenvalue decomposition on the covariance matrix, it obtains the principal component vectors and corresponding eigenvalues ​​of the model parameter matrix. Next, based on the principal component vectors, it calculates the corresponding vectors of other model parameter matrices to obtain the required parameters. This allows the calculation of similarity and dissimilarity to be performed based on these parameters. A surrogate relationship is established between the model parameter matrix and the data matrix, ensuring that the similarity and dissimilarity calculated based on the model parameters accurately describe the similarity and dissimilarity between data distributions, and that the participants' contributions accurately describe the magnitude of their contributions to the data distribution.

[0029] This disclosure presents an industrial IoT federated learning method based on principal component analysis. It utilizes eigenvalue decomposition to transform model parameters into matrices, calculating eigenvalues ​​and eigenvectors as principal components. Operations are performed based on these principal components to proxy the distribution of the original data. A similarity and difference calculation method is proposed based on the principal components and the vectors of other model parameters along the principal component directions to calculate the contribution of each participant's uploaded model. Based on this contribution, the weights are converted into aggregation weights and applied to the aggregation process, ultimately yielding a global model. This disclosure allows the server to detect low-quality and malicious participants based on their contribution size, assigning them lower weights to reduce their negative impact.

[0030] This disclosure presents an industrial IoT federated learning method based on principal component analysis, which trains machine learning models based on data from multiple factories, evaluates the contributions of each factory, enables predictive maintenance or identifies common patterns and trends to optimize the operation of all factories, and incentivizes factories to participate in federated learning honestly and credibly. Attached Figure Description

[0031] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0032] Figure 1 This is a schematic diagram of a scenario where the similarity is less than the difference in an embodiment of this disclosure.

[0033] Figure 2 This is a schematic diagram of a scenario where the degree of difference is less than the degree of similarity according to an embodiment of this disclosure.

[0034] In this diagram, the direction of the vector represents the direction of the two principal components, and the length of the vector represents the magnitude of the eigenvalues ​​along the principal component direction. The yellow rectangle represents the similarity, with its sides being the intersection of vectors in the same direction; the gray rectangle represents the difference, with its sides being the difference between vectors in the same direction. Based on the magnitude of similarity and difference, the smaller value is taken as the contribution, i.e., the red shaded area. Detailed Implementation

[0035] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0037] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0038] Example 1

[0039] One embodiment of this disclosure provides an industrial IoT federated learning method based on principal component analysis, including:

[0040] Step 1: Obtain the model parameters uploaded by all factory participants, and construct a model parameter matrix using the model parameters;

[0041] Step 2: Use principal component analysis to reduce the dimensionality of the model parameter matrix and calculate the covariance matrix of the model parameter matrix. Use eigenvalue decomposition on the covariance matrix to obtain the principal components of the model parameter matrix.

[0042] Step 3: Calculate the similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix;

[0043] Step 4: Based on the calculated similarity and difference results, the minimum value between the similarity and difference is taken as the model contribution.

[0044] Step 5: Based on the model contribution, use a SoftMax-like function to transform and calculate the corresponding weights. Apply the weights obtained from the transformation of the model contribution to the aggregated global model to obtain the global model with the new weights.

[0045] As one embodiment, this disclosure presents an industrial IoT federated learning method based on principal component analysis (PCA). The PCA-based federated learning model contribution metric calculation and aggregation weight calculation method can help the federated learning server mitigate the adverse effects of malicious and low-quality nodes, thereby improving federated learning performance. The specific implementation process is as follows:

[0046] Step 1: Obtain the model parameters uploaded by all factory participants, and construct a model parameter matrix using the model parameters;

[0047] Specifically, the model parameters are transformed into matrices. First, consider some classic and common structures. For a two-dimensional fully connected neural network, it can be viewed as a matrix itself. For a four-dimensional convolutional neural network, the convolutional kernels are unfolded into data vectors, allowing the kernels to be stacked to form a matrix where each column is an unfolded kernel. For more complex structures, such as the transformer, it can be viewed as a collection of matrices from which matrices can be extracted. These matrices contain fully connected neural networks; one can be chosen as a representative, or the fully connected neurons can be unfolded into vectors and concatenated to form a matrix.

[0048] Step 2: Use principal component analysis to reduce the dimensionality of the model parameter matrix and calculate the covariance matrix of the model parameter matrix. Use eigenvalue decomposition on the covariance matrix to obtain the principal components of the model parameter matrix.

[0049] Furthermore, the covariance matrix is ​​calculated based on the model parameter matrix, and eigenvalue decomposition is performed on the covariance matrix to obtain its d eigenvectors and eigenvalues ​​λ0 as principal components.

[0050] Specifically, for the model parameters uploaded by participant i, its covariance matrix is ​​first calculated:

[0051]

[0052] Among them, Cov i Let m be the covariance matrix of i, m be the dimension of the model parameter matrix, and w be the covariance matrix of i. i This is the model parameter matrix.

[0053] Then, eigenvalue decomposition (EVD) is performed on its covariance matrix to calculate the characteristic equation and obtain the eigenvalues. The characteristic equation is as follows:

[0054] |Cov i -λE|=0

[0055] Then, each eigenvalue is substituted into the system of linear equations to obtain the eigenvectors and eigenvalues. The d largest eigenvalues ​​are selected as principal components. Substituting each eigenvalue into the system of linear equations yields:

[0056] (Cov i -λE)x=0

[0057] The d eigenvalues ​​with the largest values ​​selected as principal components include: {λ1, λ2, ..., λ...} d},{v1,v2,…,v d}

[0058] Furthermore, for the model parameter matrix constructed from the model parameters uploaded by other factory participants, the covariance matrix is ​​calculated sequentially, and then the eigenvalues ​​of the covariance matrix in the direction of a specific principal component vector are calculated.

[0059] Specifically, for the uploaded model parameters of the other selected factory participants, their covariance matrices are calculated sequentially:

[0060]

[0061] Among them, Cov j Let w represent the covariance matrix of j, m be the dimension of the model parameter matrix, and w be the dimension of the model parameter matrix. i ,w j This is the model parameter matrix.

[0062] Then, calculate the vector of the covariance matrix of other model parameter matrices on the principal components of the specific model parameter matrix, and the magnitude of its eigenvalues.

[0063] λ j,l ←||Cov j v l ||

[0064] Where, λ j,l Cov represents the magnitude of the eigenvalues ​​of the model parameter matrix of participant j on the principal component l. j Let v be the model parameter matrix for participant j. l l is the principal component of the parameter matrix of the participants being compared.

[0065] Thus, for each principal component, the magnitude of the eigenvalues ​​of the model parameter matrix in that direction is obtained.

[0066] Step 3: Calculate the similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix. This includes: comparing the feature values ​​of the specific local model with those of other local models pairwise, calculating the similarity and difference, and using the product of the intersection parts of the feature vectors as the similarity and the product of the difference parts as the difference.

[0067] Specifically, based on the magnitude of the eigenvalues ​​of the model parameter matrix along a specific principal component vector direction, the eigenvalues ​​of participant i are compared pairwise with those of other participants j to calculate similarity and difference. The product of the intersections of their eigenvectors is used as the similarity, and the product of the differences is used as the difference. The square root of the final product is then taken to prevent it from being too small. Figure 1 As shown.

[0068] The product of the intersecting parts of the feature vectors is used as the similarity score, as follows:

[0069]

[0070] Among them, Sim i,j Let λ represent the similarity value between participants i and j, d be the number of principal components selected, and λ be the similarity value between participants i and j. l Let λ be the eigenvalue of participant i on the l-th principal component. j,l Let represent the eigenvalues ​​of the parameter matrix of participant j on the principal component l.

[0071] The product of the differing portions, taken as the degree of difference, is:

[0072]

[0073] Among them, Div i,j Let λ represent the similarity value between participants i and j, d be the number of principal components selected, and λ be the similarity value between participants i and j. l Let λ be the eigenvalue of participant i on the l-th principal component. j,l Let represent the eigenvalues ​​of the parameter matrix of participant j on the principal component l.

[0074] Furthermore, based on the calculated similarity and difference results, the minimum value between the similarity and difference is taken as the model contribution.

[0075] Step 4: Based on the model contribution, use a SoftMax-like function to transform and calculate the corresponding weights. Apply the weights obtained from the transformation of the model contribution to the aggregated global model to obtain the global model with the new weights.

[0076] Specifically, the smaller of the similarity and difference values ​​is used as the contribution level to ensure that the similarity and difference values ​​of participants with high contributions are not too low.

[0077] The calculated contributions are then applied to the aggregation process of federated learning, transforming them into weights using a softmax-like function:

[0078]

[0079] Among them, e iThis represents the aggregate weight of participant i, α is the temperature coefficient used to adjust the variance, and is a publicly disclosed parameter. i This represents the contribution value of participant i, and k is the number of participants in the current aggregation round.

[0080] Then, a weighted aggregation process is performed based on the weights to obtain the global model: Model global =∑e i M i .

[0081] Example 2

[0082] One embodiment of this disclosure provides an industrial IoT federated learning system based on principal component analysis, comprising:

[0083] The parameter acquisition module is used to acquire the model parameters uploaded by all factory participants and construct a model parameter matrix using the model parameters;

[0084] The principal component analysis module is used to perform dimensionality reduction on the model parameter matrix using the principal component analysis method, calculate the covariance matrix of the model parameter matrix, and obtain the principal components of the model parameter matrix by eigenvalue decomposition on the covariance matrix.

[0085] The metrics calculation module is used to calculate the similarity and difference between a specific local model and other local models based on the principal components of the model parameter matrix; based on the calculated similarity and difference results, the minimum value of the similarity and difference is taken as the model contribution.

[0086] The aggregation module is used to transform and calculate the corresponding weights based on the model contribution using a SoftMax-like function. The weights obtained from the transformation of the model contribution are then applied to the aggregated global model to obtain the global model with the new weights.

[0087] Example 3

[0088] One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned industrial IoT federated learning method based on principal component analysis.

[0089] Example 4

[0090] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the aforementioned industrial IoT federated learning method based on principal component analysis.

[0091] Example 5

[0092] One embodiment of this disclosure provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the industrial IoT federated learning method based on principal component analysis.

[0093] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. An industrial internet of things federated learning method based on principal component analysis, characterized in that, Comprise: Obtain all factory participants upload model parameters, use model parameters to build model parameter matrix; Adopt principal component analysis method to model parameter matrix data dimension reduction, and calculate the covariance matrix of model parameter matrix, use eigenvalue decomposition to obtain the principal component of model parameter matrix; Based on the principal component of the model parameter matrix, the similarity and difference between the specific local model and other local models are calculated, including: calculating the covariance matrix of other model parameter matrix on each principal component of the specific model parameter matrix and its eigenvalue size, comparing the eigenvalues of the specific local model and other local models two by two, calculating the similarity and difference, and the product of the intersection of the eigenvectors as the similarity, for: The product of the difference part is the difference, for: wherein, and represent similarity and difference between participants and , is the number of selected principal components, is the eigenvalue of the participant on the th principal component, represents the eigenvalue of the parameter matrix of the participant on the principal component ; According to the calculated similarity and difference results, the minimum value of the similarity and difference is taken as the model contribution degree; Based on the model contribution degree, use the SoftMax function to transform and calculate the corresponding weight, and apply the weight obtained by transforming the model contribution degree to the aggregated global model, so as to obtain the global model under the new weight.

2. The principal component analysis based federated learning method for industrial internet of things according to claim 1, wherein, Use principal component analysis method to reduce the data dimension of specific model parameter matrix, calculate its covariance matrix as: wherein denotes the covariance matrix of is the dimension of the model parameter matrix, is the model parameter matrix; Then the SVD eigenvalue decomposition is carried out on the covariance matrix, the eigenvectors and eigenvalues are obtained, and the d largest eigenvalues and eigenvectors are selected as the principal components.

3. The principal component analysis based federated learning method for industrial internet of things according to claim 2, wherein, For the model parameter matrix constructed by the model parameters uploaded by other factory participants, the covariance matrix is calculated in turn, and then the eigenvalue size of the covariance matrix in the direction of the specific principal component vector is calculated.

4. The principal component analysis based federated learning method for industrial internet of things of claim 1, wherein, The small value in the similarity and difference is taken as the contribution degree, and the calculated contribution degree is applied to the aggregation process of federated learning, and a softmax function is used to transform it into weight: wherein, denotes the aggregated weight of the participants , is a temperature coefficient for adjusting the variance, is a public parameter, denotes the contribution value size of the participants , is the number of participants in the current aggregation round.

5. An industrial internet of things federated learning system based on principal component analysis, characterized in that, Comprise: Parameter acquisition module, used for obtaining all factory participants upload model parameters, use model parameters to build model parameter matrix; Principal component analysis module, for adopting principal component analysis method to model parameter matrix data dimension reduction, and calculate the covariance matrix of model parameter matrix, use eigenvalue decomposition to obtain the principal component of model parameter matrix; Degree calculation module, for calculating the similarity and difference between the specific local model and other local models based on the principal component of the model parameter matrix, including: calculating the covariance matrix of other model parameter matrix on each principal component of the specific model parameter matrix and its eigenvalue size, comparing the eigenvalues of the specific local model and other local models two by two, calculating the similarity and difference, and the product of the intersection of the eigenvectors as the similarity, for: The product of the difference part is the difference, for: wherein, and represent similarity and difference between participants and , is the number of selected principal components, is the eigenvalue of the participant on the th principal component, represents the eigenvalue of the parameter matrix of the participant on the principal component ; According to the calculated similarity and difference results, the minimum value of the similarity and difference is taken as the model contribution degree; Aggregation module, for using SoftMax function to transform and calculate the corresponding weight based on the model contribution degree, and applying the weight obtained by transforming the model contribution degree to the aggregated global model, so as to obtain the global model under the new weight.

6. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the principal component analysis-based industrial Internet of Things federated learning method in any one of claims 1-4.

7. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium is used to store computer instructions, and the computer instructions are executed by a processor to implement the principal component analysis-based industrial Internet of Things federated learning method in any one of claims 1-4.

8. An electronic device, comprising: Comprise: A processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the principal component analysis-based industrial Internet of Things federated learning method in any one of claims 1-4.

Images