Principal component analysis multi-bias interaction-based longitudinal federated learning optimization method, electronic device and storage medium

Abstract

The application discloses a principal component analysis multi-bias interaction-based longitudinal federal learning optimization method, an electronic device and a storage medium, and belongs to the technical field of privacy calculation. In order to improve the efficiency of a longitudinal federal neural network model in the case that the capacity of a data set is small, the application trains participating parties including a training initiator and a training assistant, trains data of the training initiator and data of the training assistant by using a forward propagation method, obtains a training result of the forward propagation method, trains the training result of the forward propagation method by using a backward propagation method, and the training assistant and the training initiator respectively update model parameters, so that one round of training is completed. The training is repeated until the training result reaches a precision requirement or a stop condition, and neural network training in a longitudinal federal scene based on multi-bias interaction is completed. The principal component analysis data dimension reduction method is used, so that the filtering function of features is guided by multi-party data information, and the result is more persuasive.

Description

Longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions, electronic devices and storage media Technical Field

[0001] This invention belongs to the field of privacy computing technology, specifically relating to a longitudinal federated learning optimization method based on principal component analysis with multiple bias interactions, an electronic device, and a storage medium. Background Technology

[0002] Federated machine learning, also known as federated learning, is a machine learning framework that effectively helps multiple organizations use data and perform machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations. It can effectively solve the data silo problem. Federated learning is divided into horizontal federated learning, vertical federated learning, and transfer federated learning based on the different patterns of data held in the scenario. Vertical federated learning is used in scenarios where the data held by the training assisting party has some overlap in the real-world individuals corresponding to the entries, but the data features are almost completely different.

[0003] Currently, the most widely used vertical federated learning algorithm is the vertical federated neural network algorithm in the federated AI framework developed by WeBank. It supports multi-party collaborative execution of neural network algorithms in vertical federated scenarios, ensuring that assisting parties can provide the same level of accuracy as non-privacy-protected schemes without compromising data privacy. This algorithm is currently mainly used for prediction and classification. During training, assisting parties are divided into training initiators (who possess the classification guidance labels) and training assistants (who do not). The vertical federated neural network model structure is divided into three layers: the bottom layer model, the interaction layer model, and the top layer model. Training assistants only possess the bottom layer model, while the training initiator possesses all models. The three types of models have different functions: the bottom layer model is mainly responsible for refining and training the model using local data from each party to obtain intermediate results; the interaction layer model is responsible for data interaction between the two parties, integrating the intermediate results from the bottom layer and outputting the intermediate results as input data for the top layer; the top layer is unique to the training initiator, which trains the top layer model based on the output of the interaction layer model to obtain the final classification result.

[0004] In existing model structures, the training assistant essentially only performs feature engineering on local data and transmits the results to the training initiator, without any other involvement. This has a significant impact when the dataset size is small. The dataset size is determined by objective realities such as the device conditions of the training parties, the application scenario of the trained model, the frequency of data usage by each party, and the amount of data involved. With a small dataset, model training relies more heavily on the information and implicit features contained in the original data. Simplifying the training model to reduce interaction and improve efficiency makes the model training more dependent on chance results, and its interpretability and robustness against interference will correspondingly decrease. Summary of the Invention

[0005] The problem this invention aims to solve is to improve the efficiency of longitudinal federated neural network models when the dataset size is small. It proposes a longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions, as well as an electronic device and storage medium.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions includes the following steps:

[0008] S1. The training participants include the training initiator and the training assistant.

[0009] Set the data characteristics of the training initiator as follows: This indicates that the training initiator has m-dimensional data, where y is the label attribute, and the data has been aligned.

[0010] The data characteristics of the training assistant are This indicates that the training assistant has n-dimensional data, and the data has been aligned.

[0011] S2. Use the data from the training initiator and the training assistant in step S1 to train the forward propagation method and obtain the training results of the forward propagation method.

[0012] The specific implementation method of step S2 includes the following steps:

[0013] S2.1, Underlying Model Training Method: This includes training participants calculating the intermediate results of each layer of the underlying model, performing principal component analysis to reduce the dimensionality of the intermediate results, exchanging and transmitting the dimensionality-reduced data results among the training participants, updating the bias of the underlying model based on the exchange data, and then training each participant to update the underlying model with the updated bias, obtaining the output results of the underlying model, and proceeding to the next step.

[0014] S2.2 Interaction Layer Model Training Method: Based on the output results of the bottom-level model in step S2.1, the training participants interact and pass the data to the interaction layer model using Paillier homomorphic encryption and differential noise to obtain the overall intermediate results of the interaction layer model as the input data of the activation function of the interaction layer model. The output results of the activation function of the interaction layer model are then used to proceed to the next step.

[0015] S2.3 Top-level model training method: The training initiator uses the output of the activation function of the interaction layer model as the input of the top-level model to train the final classification model and obtain the training result of the forward propagation method.

[0016] S3. Backpropagation method: The training results of the forward propagation method obtained in step S2 are used for backpropagation training. The training assistant and the training initiator update the model parameters respectively to complete one round of training.

[0017] S4. Repeat steps S2 and S3 until the training results meet the accuracy requirements or the stopping condition is met, thus completing the training of the longitudinal federated scene neural network based on multi-bias interaction.

[0018] Furthermore, the model structure of each training participant consists of fully connected layers and Dropout layers, and the underlying models of each training participant have the same number of layers.

[0019] Furthermore, the specific implementation method of step S2.1 includes the following steps:

[0020] S2.1.1, Settings The parameters of the t-th layer of the underlying model used to train the initiator. For the parameters of the t-th layer of the underlying model used to train the assistant:

[0021]

[0022]

[0023] Among them, the training initiator's underlying model has i neurons in the t-th layer and j neurons in the (t+1)-th layer, while the training assistant's underlying model has l neurons in the t-th layer and k neurons in the (t+1)-th layer.

[0024] S2.1.2 The training initiator and training assistant calculate the intermediate results of each layer of the underlying model, except for the last layer. The formula for calculating the intermediate results of the first layer is as follows:

[0025]

[0026]

[0027] The formulas for calculating the intermediate results of the remaining layers are as follows:

[0028]

[0029]

[0030] in and ReLU is a common instance of the activation function for fully connected layers and Dropout layers in neural networks, representing the intermediate computation results of the training initiator and training assistant at layer t.

[0031] S2.1.3. Perform principal component analysis to reduce the dimensionality of the intermediate results obtained from the calculation. The processing formula is as follows:

[0032] (All intermediate results except the last layer are calculated)

[0033] (All intermediate results except the last layer are calculated)

[0034] Among them, Z B To train the propagation bias of the initiator, Z A To train the transmission bias of the assisting party;

[0035] The expression for the sum_row function is:

[0036]

[0037] in,

[0038] The expression for the trans function is:

[0039] teans(X) = PCA(X) T ,1) T ;

[0040] After principal component analysis, the matrix is ​​compressed and transposed into a one-dimensional row vector.

[0041] S2.1.4, The training initiator will Z B Send to the training assistant, and simultaneously receive Z from them. A The training initiator will Z A Each term is used as a weighting coefficient, and the model bias B = {B} is synchronously updated to the underlying model of the training initiator. i|i=1,2,…(all layers except the last one)}, Z A Save as local result

[0042] The training assistance party will ZA 发 Send it to the training initiator, and simultaneously receive the Z sent by them. B The training assistance party will Z B Each term is used as a weighting coefficient, and the model bias A = {A} is synchronously updated to the underlying model of the training assistant. i |i=1,2,…(all layers except the last one)}, Z B Save as local result

[0043] The update formula for the first round of training is:

[0044] B i =B i ·z ai

[0045] A i =A i ·z bi

[0046] Among them, z ai For Z A The i-th term, z bi For Z B The i-th term;

[0047] The update formula for non-first round training is:

[0048]

[0049]

[0050] in, for The i-th term, for The i-th term;

[0051] S2.1.5. Each participating party trains and updates its underlying model to update the bias, obtaining the output α of the training assistant's underlying model. A The underlying model output α of the training initiator B .

[0052] Furthermore, the specific implementation method of step S2.2 includes the following steps:

[0053] S2.2.1 The training assistant uses Paillier homomorphic encryption α A, thus obtaining α A Ciphertext data [α] A ], sent to the training initiator;

[0054] S2.2.2, The training initiator receives [α] A After that, perform the calculation. The calculation formula is:

[0055] [z A ]=[α A ]·W A

[0056] Among them, [z A W represents the intermediate result computed by the training assistant at the interaction layer in encrypted conditions. A The weights for the training assistant part of the interaction layer model;

[0057] S2.2.3, The training initiator simultaneously generates noise ε based on the dimension. B , will [z A ]+ε B The calculation formula sent to the training assistant for local computation is as follows:

[0058] z B =α B ·W B

[0059] Among them, z B W represents the intermediate result computed by the training initiator at the interaction layer in the unencrypted case. B These are the weights of the training initiator part of the interaction layer model;

[0060] S2.2.4 Training assistant calculates cumulative noise ε acc ·α A And decrypt the received ciphertext to obtain z A +ε B Then, the results are summed and sent to the training initiator. The calculation formula is:

[0061]

[0062] Where, ε acc The noise ε generated by the training assistant during each round of backpropagation. A The sum of This is an intermediate result of the training assistant interaction layer containing noise;

[0063] S2.2.5 Training Initiator Receives Then, calculate the overall intermediate result z of the interaction layer using the following formula:

[0064]

[0065] Using z as the input data to the activation function of the interaction layer, we obtain the output result g(z) of the activation function of the interaction layer model.

[0066] Furthermore, the specific implementation method of step S3 includes the following steps:

[0067] S3.1 Top-level model training method: The training initiator trains the model based on the training results of the forward propagation method. Update the top-level model parameters for label y, calculate the loss δ based on the top-level model, and proceed to the next step;

[0068] S3.2 Interaction Layer Model Training Method: The training initiator calculates the loss of the interaction layer activation function based on the loss and the interaction layer model activation function. The training initiator calculates the loss to update its own underlying model parameters, and then updates the parameters of the interaction layer model corresponding to the training initiator. Then, the training initiator and the training assistant perform interactive calculations based on Paillier homomorphic encryption and differential noise to obtain the updated loss of the underlying model parameters of the training assistant. Then, the training initiator updates the parameters of the interaction layer model corresponding to the training assistant and sends it to the training assistant. The training assistant decrypts the received updated loss of the underlying model parameters of the training assistant and proceeds to the next step.

[0069] S3.3, Underlying Model Training Method: The training assistant decrypts the loss received from updating the underlying model parameters and transmits it to the underlying model. The training initiator and the training assistant then update the underlying model parameters respectively.

[0070] Furthermore, the specific implementation method of step S3.2 includes the following steps:

[0071] S3.2.1 The training initiator calculates the loss δ of the interaction layer activation function based on the loss δ and the interaction layer model activation function. act ;

[0072] S3.2.2, The training initiator first calculates the loss δ of some parameters of the underlying model. bottom_B This is used to update the parameters of the underlying model, and the calculation formula is:

[0073] δ bottom_B =δ act ·W B

[0074] S3.2.3, Based on δ act Update the parameters of the interaction layer model corresponding to the training initiator, calculated using the following formula:

[0075] W B -=η*δ act·α B

[0076] Where η is the set model learning rate;

[0077] S3.2.4, Training initiator generates noise ε B The calculation is based on Paillier homomorphic encryption and differential noise, and the result δ is... act ·[α A ]+ε B Send to the training support provider;

[0078] S3.2.5, The training assistant will ε acc Encryption yields [ε] acc Then, noise ε is generated. A Update ε acc +=ε A After decrypting the calculation result of step S3.2.4, δ is obtained. act ·α A +ε B The training assistance party will [ε acc Transmitted to the training initiator;

[0079] S3.2.6 After receiving the result of step S3.2.5, the training initiator first sets δ... act ·(W A +[ε acc The loss of the underlying model of the training assistant is fed back to update the interaction layer model, which then updates the parameters W of the training assistant. A The calculation formula is:

[0080]

[0081] S3.2.7, Training Assistant Decrypts δ act ·(W A +[ε acc The data is passed to the underlying model to update the underlying model parameters.

[0082] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions.

[0083] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the described longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions.

[0084] The beneficial effects of this invention are:

[0085] The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions described in this invention utilizes Principal Component Analysis (PCA) for dimensionality reduction. In the field of machine learning, dimensionality reduction broadly refers to using a mapping method to map data points from a high-dimensional space to a low-dimensional space, while preserving as much important information as possible. Dimensionality reduction can be understood as learning a mapping, where the input is considered as existing high-dimensional data, and the output is lower-dimensional data retaining more features of the input. It uses orthogonal transformations as the mapping matrix to map high-dimensional data samples to a low-dimensional space using matrix multiplication, thereby alleviating the curse of dimensionality and enabling better data processing.

[0086] The longitudinal federated learning optimization method based on principal component analysis and multi-bias interaction described in this invention is an innovative strategy that modifies the model structure by adding intermediate computational result interaction to the underlying model, thereby increasing information interaction among the training parties. Simultaneously, introducing the interaction of intermediate computational results from each party into the underlying model training stage—which primarily performs functions such as feature selection and dimensionality reduction—allows the feature filtering function to be guided by multi-party data information, resulting in more convincing results. However, evaluating the feasibility and practicality of introducing this interaction requires consideration of protocol security and communication costs.

[0087] The longitudinal federated learning optimization method based on principal component analysis and multi-bias interaction described in this invention introduces a new security protocol for the bias interaction and update of the underlying model. This scheme considers converting the input information of the activation function of each layer calculated by each party's underlying model into an intermediate computational result, which is then transmitted as a bias value to the other party. The participating parties also treat the bias value as a combination of parameters and inputs rather than a constant; its new input form is the product of the originally generated constant bias value and the transmitted bias value. This invention compresses the input information into values ​​by performing principal component analysis on the intermediate input information of each data point for dimensionality reduction and obfuscation to obtain a transmitted intermediate value vector. A related secure transmission protocol is designed to ensure data privacy and security, as well as transmission efficiency and feasibility. This allows the underlying model to incorporate information from other participating parties' data during data processing and training, which is valuable for enhancing data utilization. Attached Figure Description

[0088] Figure 1 is a flowchart of the longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions described in this invention;

[0089] Figure 2 is a schematic diagram of the training method of the forward propagation method described in this invention;

[0090] Figure 3 is a schematic diagram of the training method of the backpropagation method described in this invention. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0092] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0093] To further understand the invention's content, features, and effects, the following specific embodiments are provided, and detailed descriptions are given in conjunction with Figures 1-3:

[0094] Specific implementation method one:

[0095] The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions includes the following steps:

[0096] S1. The training participants include the training initiator and the training assistant.

[0097] Set the data characteristics of the training initiator as follows: This indicates that the training initiator has m-dimensional data, where y is the label attribute, and the data has been aligned.

[0098] The data characteristics of the training assistant are This indicates that the training assistant has n-dimensional data, and the data has been aligned.

[0099] Furthermore, the model structure of each training participant consists of fully connected layers and Dropout layers, and the underlying models of each training participant have the same number of layers.

[0100] S2. Use the data from the training initiator and the training assistant in step S1 to train the forward propagation method and obtain the training results of the forward propagation method.

[0101] The specific implementation method of step S2 includes the following steps:

[0102] S2.1, Underlying Model Training Method: This includes training participants calculating the intermediate results of each layer of the underlying model, performing principal component analysis to reduce the dimensionality of the intermediate results, exchanging and transmitting the dimensionality-reduced data results among the training participants, updating the bias of the underlying model based on the exchange data, and then training each participant to update the underlying model with the updated bias, obtaining the output results of the underlying model, and proceeding to the next step.

[0103] Furthermore, the specific implementation method of step S2.1 includes the following steps:

[0104] S2.1.1, Settings The parameters of the t-th layer of the underlying model used to train the initiator. For the parameters of the t-th layer of the underlying model used to train the assistant:

[0105]

[0106]

[0107] Among them, the training initiator's underlying model has i neurons in the t-th layer and j neurons in the (t+1)-th layer, while the training assistant's underlying model has l neurons in the t-th layer and k neurons in the (t+1)-th layer.

[0108] S2.1.2 The training initiator and training assistant calculate the intermediate results of each layer of the underlying model, except for the last layer. The formula for calculating the intermediate results of the first layer is as follows:

[0109]

[0110]

[0111] The formulas for calculating the intermediate results of the remaining layers are as follows:

[0112]

[0113]

[0114] in and ReLU is a common instance of the activation function for fully connected layers and Dropout layers in neural networks, representing the intermediate computation results of the training initiator and training assistant at layer t.

[0115] S2.1.3. Perform principal component analysis to reduce the dimensionality of the intermediate results obtained from the calculation. The processing formula is as follows:

[0116] (All intermediate results except the last layer are calculated)

[0117] (All intermediate results except the last layer are calculated)

[0118] Among them, Z B To train the propagation bias of the initiator, Z A To train the transmission bias of the assisting party;

[0119] The expression for the sum_row function is:

[0120]

[0121] in,

[0122] The expression for the trans function is:

[0123] trnas(X)=PCA(X T ,1) T ;

[0124] After principal component analysis, the matrix is ​​compressed and transposed into a one-dimensional row vector.

[0125] S2.1.4, The training initiator will Z B Send to the training assistant, and simultaneously receive Z from them. A The training initiator will Z A Each term is used as a weighting coefficient, and the model bias B = {B} is synchronously updated to the underlying model of the training initiator. i |i=1,2,…(all layers except the last one)}, Z A Save as local result

[0126] The training assistant will Z A Send to the training initiator, and simultaneously receive the Z signal sent from them. B The training assistance party will Z B Each term is used as a weighting coefficient, and the model bias A = {A} is synchronously updated to the underlying model of the training assistant. i |i=1,2,…(all layers except the last one)}, Z B Save as local result

[0127] The update formula for the first round of training is:

[0128] B i =B i ·z ai

[0129] A i =A i ·z bi

[0130] Among them, z ai For Z A The i-th term, z bi For Z B The i-th term;

[0131] The update formula for non-first round training is:

[0132]

[0133]

[0134] in, for The i-th term, for The i-th term;

[0135] S2.1.5. Each participating party trains and updates its underlying model to update the bias, obtaining the output α of the training assistant's underlying model. A The underlying model output α of the training initiator B ;

[0136] Furthermore, the principal component analysis method is as follows:

[0137] Set the input dataset Required target dimension k; output dataset

[0138] Principal component analysis includes the following steps:

[0139] i)

[0140] ii) Calculate X T X;

[0141] iii) Calculate X T The eigenvalues ​​and eigenvectors of X are sorted by taking the square root of the eigenvalues ​​and then forming Σ.

[0142] iv) After normalizing the eigenvector matrix, sort it into Q according to the size of Σ;

[0143] Shaped like Each column of this matrix corresponds to the covariance matrix X. T The normalized eigenvectors of the eigenvalues ​​of X;

[0144] v) Take the transpose of the first k column vectors of Q to form the eigenvector matrix P, and output Y = XP;

[0145] Here, the above algorithm with k=1 is used to calculate, and the summed matrix is ​​compressed and transposed into a one-dimensional row vector;

[0146] S2.2 Interaction Layer Model Training Method: Based on the output results of the bottom-level model in step S2.1, the training participants interact and pass the data to the interaction layer model using Paillier homomorphic encryption and differential noise to obtain the overall intermediate results of the interaction layer model as the input data of the activation function of the interaction layer model. The output results of the activation function of the interaction layer model are then used to proceed to the next step.

[0147] Furthermore, the specific implementation method of step S2.2 includes the following steps:

[0148] S2.2.1 The training assistant uses Paillier homomorphic encryption α A , thus obtaining α A Ciphertext data [α] A ], sent to the training initiator;

[0149] S2.2.2, The training initiator receives [α] A After that, perform the calculation. The calculation formula is:

[0150] [z A ]=[α A ]·W A

[0151] Among them, [z A W represents the intermediate result computed by the training assistant at the interaction layer in encrypted conditions. A The weights for the training assistant part of the interaction layer model;

[0152] S2.2.3, The training initiator simultaneously generates noise ε based on the dimension. B , will [z A ]+ε B The calculation formula sent to the training assistant for local computation is as follows:

[0153] z B =α B ·W B

[0154] Among them, z B W represents the intermediate result computed by the training initiator at the interaction layer in the unencrypted case. BThese are the weights of the training initiator part of the interaction layer model;

[0155] S2.2.4 Training assistant calculates cumulative noise ε acc ·α A And decrypt the received ciphertext to obtain z A +ε B Then, the results are summed and sent to the training initiator. The calculation formula is:

[0156]

[0157] Where, ε acc The noise ε generated by the training assistant during each round of backpropagation. A The sum of This is an intermediate result of the training assistant interaction layer containing noise;

[0158] S2.2.5 Training Initiator Receives Then, calculate the overall intermediate result z of the interaction layer using the following formula:

[0159]

[0160] Using z as the input data to the activation function of the interaction layer, we obtain the output result g(z) of the activation function of the interaction layer model.

[0161] S2.3 Top-level model training method: The training initiator uses the output of the activation function of the interaction layer model as the input of the top-level model to train the final classification model and obtain the training result of the forward propagation method.

[0162] S3. Backpropagation method: The training results of the forward propagation method obtained in step S2 are used for backpropagation training. The training assistant and the training initiator update the model parameters respectively to complete one round of training.

[0163] Furthermore, the specific implementation method of step S3 includes the following steps:

[0164] S3.1 Top-level model training method: The training initiator trains the model based on the training results of the forward propagation method. Update the top-level model parameters for label y, calculate the loss δ based on the top-level model, and proceed to the next step;

[0165] S3.2 Interaction Layer Model Training Method: The training initiator calculates the loss of the interaction layer activation function based on the loss and the interaction layer model activation function. The training initiator calculates the loss to update its own underlying model parameters, and then updates the parameters of the interaction layer model corresponding to the training initiator. Then, the training initiator and the training assistant perform interactive calculations based on Paillier homomorphic encryption and differential noise to obtain the updated loss of the underlying model parameters of the training assistant. Then, the training initiator updates the parameters of the interaction layer model corresponding to the training assistant and sends it to the training assistant. The training assistant decrypts the received updated loss of the underlying model parameters of the training assistant and proceeds to the next step.

[0166] Furthermore, the specific implementation method of step S3.2 includes the following steps:

[0167] S3.2.1 The training initiator calculates the loss δ of the interaction layer activation function based on the loss δ and the interaction layer model activation function. act ;

[0168] S3.2.2, The training initiator first calculates the loss δ of some parameters of the underlying model. bottom_B This is used to update the parameters of the underlying model, and the calculation formula is:

[0169] δ bottom_B =δ act ·W B

[0170] S3.2.3, Based on δ act Update the parameters of the interaction layer model corresponding to the training initiator, calculated using the following formula:

[0171] W B -=η*δ act ·α B

[0172] Where η is the set model learning rate;

[0173] S3.2.4, Training initiator generates noise ε B The calculation is based on Paillier homomorphic encryption and differential noise, and the result δ is... act ·[α A ]+ε B Send to the training support provider;

[0174] S3.2.5, The training assistant will ε acc Encryption yields [ε] acc Then, noise ε is generated. A Update ε acc +=ε A After decrypting the calculation result of step S3.2.4, δ is obtained. act ·αA +ε B The training assistance party will [ε acc Transmitted to the training initiator;

[0175] S3.2.6 After receiving the result of step S3.2.5, the training initiator first sets δ... act ·(W A +[ε acc The loss of the underlying model of the training assistant is fed back to update the interaction layer model, which then updates the parameters W of the training assistant. A The calculation formula is:

[0176]

[0177] S3.2.7, Training Assistant Decrypts δ act ·(W A +[ε acc The data is passed to the underlying model to update its parameters.

[0178] S3.3, Underlying Model Training Method: The training assistant decrypts the loss received from updating the underlying model parameters and transmits it to the underlying model. The training initiator and the training assistant update the underlying model parameters respectively.

[0179] S4. Repeat steps S2 and S3 until the training results meet the accuracy requirements or the stopping condition is met, thus completing the training of the longitudinal federated scene neural network based on multi-bias interaction.

[0180] The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions described in this embodiment is verified by the following experimental results:

[0181] To demonstrate the security of this scheme, we first define the data targets to be protected and the adversary's role. The non-leaking data includes the raw data used in model training (i.e., the model's input dataset), denoted as X; and the model parameters of the participant's local bottom model, denoted as W. In addition, plaintext intermediate training results (gradients, updated parameters such as WX, etc.) also provide valuable information to the adversary and therefore must be protected; this intermediate result is denoted as α. The adversary is envisioned as a curious participant A, who always attempts to obtain data information from other participants through received data values. To address application scenarios with insufficient data entries, a scheme using PCA dimensionality reduction for intermediate data results is introduced. In this scheme, instead of averaging each data point in the intermediate data results, dimensionality reduction is performed on the intermediate data result matrix.

[0182] Let R be the intermediate result matrix, and b be the transmission bias vector calculated by the algorithm. In the PCA algorithm, the low-dimensional orthogonal mapping matrix of data R is actually the eigenvector of its highest eigenvalue. Therefore, if an adversary wants to reconstruct the intermediate data set R based on the obtained bias value information b, it needs to obtain the eigenvector of R. However, without knowing any information about matrix R, it is obviously impossible to solve for its eigenvalues ​​and eigenvectors. Therefore, this scheme can ensure good information protection even with limited data.

[0183] The decision regarding the limited data needs to be made based on the specific circumstances. First, the parties involved in executing the algorithm must assess their acceptable range of guesses based on their computing power, time resources, and the sensitivity of the data (for example, the protection priority for age and gender information among participants will certainly be lower than that for bank account and balance data). Secondly, the data dimensionality reduction algorithm needs to calculate (X... T ) T X T The size of this matrix is ​​a square matrix equal to the number of data entries. This is also affected by performance (for example, if the computing power and memory used are small, it may not support the calculation of matrices with tens of thousands of dimensions). The selection of the dataset capacity threshold needs to be determined through negotiation between the training parties.

[0184] This paper analyzes the increased time overhead in this implementation method. Communication overhead mainly consists of pre-send processing and the transmission process. The communication overhead analysis should assume that data alignment, key and model parameter generation, and plaintext computation costs are ignored. These time consumptions can be completed beforehand or are negligible compared to ciphertext computation. However, the newly added biased interactive communication process in the innovative underlying model uses principal component analysis for data dimensionality reduction before data transmission, obfuscating a set of data entries into a single value. This value cannot be easily reversed, and its security is not solved by encryption. In other words, the introduced communication is plaintext communication, which has an order-of-magnitude difference compared to the ciphertext communication overhead in the interaction layer. Therefore, the data transmission process in the newly introduced interaction cost can be ignored.

[0185] Specific Implementation Method Two:

[0186] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions.

[0187] The computer device of the present invention may include a processor and a memory, such as a microcontroller containing a central processing unit. Furthermore, when the processor executes the computer program stored in the memory, it implements the steps of the aforementioned recommendation method for modifyable relationship-driven recommendation data based on CREO software.

[0188] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0189] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0190] Specific implementation method three:

[0191] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the described longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions.

[0192] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by the processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. The computer-readable storage medium stores a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-described modeling method for modifyable relation-driven modeling data based on CREO software can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0193] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0194] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions, characterized by: The steps include: S1, training participants include the training initiator and the training assistant; Set the data characteristics of the training initiator as follows: , indicating that the training initiator has m-dimensional data, y is the label attribute, and the data has been aligned; The data characteristics of the training assistant are This indicates that the training assistant has n-dimensional data, and the data has been aligned. First, the data target to be protected by the scheme and the adversary setting of the scheme are given. The data that cannot be leaked includes the original data in the training process, that is, the input dataset of the model, denoted as X; the model parameters of the bottom model part of the participant's local data, denoted as W; and the plaintext intermediate training results, denoted as α. The adversary is proposed as a curious participant A, which always tries to obtain the data information of other participants through the received data values. In order to deal with the application scenario of insufficient data entries, a scheme of using PCA dimensionality reduction for intermediate data results is introduced. S2. Train the data from the training initiator and the training assistant in step S1 using the forward propagation method to obtain the training results of the forward propagation method. The specific implementation method of step S2 includes the following steps: S2.

1. Low-level model training method: This includes each training participant calculating the intermediate results of each layer of the low-level model, performing principal component analysis to reduce the dimensionality of the intermediate results, and then exchanging and transmitting the dimensionality-reduced data results among the training participants. Each training participant updates the bias of the low-level model based on the exchange data, and then each training participant trains the low-level model with updated biases to obtain the output results of the low-level model and proceed to the next step. S2.2, Interaction Layer Model Training Method: Based on the output results of the underlying model in step S2.1, the training participants train the interaction layer model... Homomorphic encryption and differential noise are passed together to obtain the overall intermediate result of the interaction layer model, which is used as the input data of the activation function of the interaction layer model. The output result of the activation function of the interaction layer model is then used to proceed to the next step. S2.3 Top-level model training method: The training initiator uses the output of the activation function of the interaction layer model as the input of the top-level model to train the final classification model and obtain the training result of the forward propagation method. ； S3. Backpropagation method: The training results of the forward propagation method obtained in step S2 are used for backpropagation training. The training assistant and the training initiator update the model parameters respectively to complete one round of training. S4. Repeat steps S2 and S3 until the training results reach the accuracy requirements or the stopping condition is met, thus completing the training of the longitudinal federated scene neural network based on multi-bias interaction. First, the participants executing the algorithm need to assess the acceptable range of guesses based on the other party's computing power, time resources, and data sensitivity. The protection priority of age and gender information in the participant's information will definitely be lower than that of data information such as bank accounts and balances.

2. The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions according to claim 1, characterized in that: The model structures of the training participants are fully connected layers and Dropout layers, and the underlying models of the training participants have the same number of layers.

3. The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions according to claim 1 or 2, characterized in that: The specific implementation method of step S2.1 includes the following steps: S2.1.1, setting The parameters of the t-th layer of the underlying model used to train the initiator. For the parameters of the t-th layer of the underlying model used to train the assistant: ； In this model, the training initiator's underlying model has i neurons in layer t and j neurons in layer t+1, while the training assistant's underlying model has l neurons in layer t and k neurons in layer t+1. S2.1.2 The training initiator and training assistant calculate the intermediate results for each layer of the underlying model, omitting calculations for the last layer. The formula for calculating the intermediate results for the first layer is: ； The formulas for calculating the intermediate results of the remaining layers are as follows: ； ;in and This represents the intermediate computation results at layer t for both the training initiator and the training assistant. This is a common example of an activation function for fully connected layers and Dropout layers in a neural network; S2.1.3, perform principal component analysis to reduce the dimensionality of the intermediate results obtained through calculation. The processing formula is as follows: ； ;in, To train the transmission bias of the initiator, To train the transmission bias of the assisting party; The function expression is: ;in, ； The function expression is: After principal component analysis, the matrix is ​​compressed and transposed into a one-dimensional row vector; S2.1.4, the training initiator will... Send to the training assistant, and simultaneously receive messages from them. The training organizer will Each term is used as a weighting coefficient, and the model bias is synchronously updated to the underlying model of the training initiator. ,Will Save as local result The training assistance party will Send to the training initiator, and simultaneously receive messages from them. The training assistance party will Each term is used as a weighting coefficient, and the model bias is synchronously updated to the underlying model of the training assistant. ,Will Save as local result The update formula for the first round of training is: ； ;in, for The i-th term, for The update formula for the i-th term (not in the first round of training) is: ； ;in, for The i-th term, for The i-th term; S2.1.5, each participating party trains and updates the underlying model of the bias, and obtains the output result of the training assistant's underlying model. The output of the underlying model of the training initiator 。 4. The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions according to claim 3, characterized in that: The specific implementation method of step S2.2 includes the following steps: S2.2.1, training the assistant to use Homomorphic encryption ,get encrypted data Send it to the training initiator; S2.2.2, The training initiator receives... Then, calculations are performed using the following formula: ;in, This refers to the intermediate results calculated by the training assistant at the interaction layer under encrypted conditions. S2.2.3, The training initiator simultaneously generates noise based on the dimension. ,Will The calculation formula sent to the training assistant for local computation is as follows: ;in, This represents the intermediate result computed by the training initiator at the interaction layer in the unencrypted case. These are the weights of the training initiator part of the interaction layer model; S2.2.4, the training assistant calculates the cumulative noise. And decrypt the received ciphertext to obtain Then, the results are summed and sent to the training initiator. The calculation formula is: ;in, The noise generated by the training assistant during each round of backpropagation. The sum of This is the intermediate result of the training assistant interaction layer containing noise; S2.2.5, the training initiator receives... Then, calculate the overall intermediate results of the interaction layer. The calculation formula is: ;Will The data is used as input to the activation function of the interaction layer to obtain the output of the activation function of the interaction layer model. 。 5. The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions according to claim 4, characterized in that: The specific implementation method of step S3 includes the following steps: S3.1, Top-level model training method: The training initiator trains the model based on the training results of the forward propagation method. ,Label Update the top-level model parameters and calculate the loss based on the top-level model. Proceed to the next step; S3.2, Interaction Layer Model Training Method: The training initiator calculates the loss of the interaction layer activation function based on the loss and the interaction layer model activation function. The training initiator calculates the loss for updating its own underlying model parameters, and then updates the parameters of the interaction layer model corresponding to the training initiator. Then, the training initiator and training assistant... Homomorphic encryption and differential noise are used for interactive computation to obtain the loss of the updated training assistant's underlying model parameters. Then, the training initiator updates the parameters of the interaction layer model corresponding to the training assistant and sends it to the training assistant. The training assistant decrypts the received loss of the updated training assistant's underlying model parameters and proceeds to the next step. S3.3, Underlying model training method: The training assistant decrypts the received loss of the updated training assistant's underlying model parameters and passes it to the underlying model. The training initiator and the training assistant update the underlying model parameters respectively.

6. The longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions according to claim 5, characterized in that: The specific implementation method of step S3.2 includes the following steps: S3.2.1, the training initiator based on loss 1. Activation function of the interaction layer model, and calculate the loss of the activation function of the interaction layer. S3.2.2, The training initiator first calculates the loss of some parameters of the underlying model. This is used to update the parameters of the underlying model, and the calculation formula is: ； S3.2.3, based on Update the parameters of the interaction layer model corresponding to the training initiator, calculated using the following formula: ;in, The set model learning rate; S3.2.4, noise generated by the training initiator. ,based on Homomorphic encryption and differential noise are calculated, and the results are... S3.2.5, The training assistant will send it to the training assistant. Encryption obtained Then noise is generated. ,renew After decrypting the calculation result of step S3.2.4, we obtain... ； The training assistance party will 、 The result is transmitted to the training initiator; S3.2.

6. After receiving the result from step S3.2.5, the training initiator first transmits it to the training initiator. The loss is fed back to the training assistant as the loss of the underlying model of the training assistant, and then the parameters of the interaction layer model corresponding to the training assistant are updated. The calculation formula is: S3.2.7, Training Assistance Decryption It is then passed to the underlying model to update the underlying model parameters.

7. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the longitudinal federated learning optimization method based on principal component analysis and multiple bias interactions as described in any one of claims 1-6.

Images