Longitudinal federated scenario neural network training method based on multi-bias interaction, electronic device and storage medium

Abstract

The application discloses a method for training a neural network in a longitudinal federated scenario based on multi-bias interaction, an electronic device and a storage medium, and belongs to the technical field of privacy computing. In order to improve the efficiency under the premise of safe transmission, the model structure of each training party is a full connection layer and a Dropout layer, including training of data of a training initiator and data of a training assistant by a forward propagation method to obtain a training result of the forward propagation method; the training result of the forward propagation method is trained by a backward propagation method, model parameters of the training initiator and the training assistant are updated respectively, one round of training is completed, and the next round is entered until the training result reaches a precision requirement or a stop condition, and the neural network training in the longitudinal federated scenario based on the multi-bias interaction is completed. The application increases information interaction between training parties, and the filtering function of features is guided by multi-party data information, so that the result is more convincing.

Description

Technical Field

[0001] This invention belongs to the field of privacy computing technology, specifically relating to a training method for a vertical federated neural network based on multi-bias interactions, an electronic device, and a storage medium. Background Technology

[0002] Federated machine learning, also known as federated learning, is a machine learning framework that effectively helps multiple organizations use data and perform machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations. It can effectively solve the data silo problem. Federated learning is divided into horizontal federated learning, vertical federated learning, and transfer federated learning based on the different patterns of data held in the scenario. Vertical federated learning is used in scenarios where the data held by the training assisting party has some overlap in the real-world individuals corresponding to the entries, but the data features are almost completely different.

[0003] Currently, the most widely used vertical federated learning algorithm is the vertical federated neural network algorithm in the federated AI framework developed by WeBank. It supports multi-party collaborative execution of neural network algorithms in vertical federated scenarios, ensuring that assisting parties can provide the same level of accuracy as non-privacy-protected schemes without compromising data privacy. This algorithm is currently mainly used for prediction and classification. During training, assisting parties are divided into training initiators (who possess the classification guidance labels) and training assistants (who do not). The vertical federated neural network model structure is divided into three layers: the bottom layer model, the interaction layer model, and the top layer model. Training assistants only possess the bottom layer model, while the training initiator possesses all models. The three types of models have different functions: the bottom layer model is mainly responsible for refining and training the model using local data from each party to obtain intermediate results; the interaction layer model is responsible for data interaction between the two parties, integrating the intermediate results from the bottom layer and outputting the intermediate results as input data for the top layer; the top layer is unique to the training initiator, which trains the top layer model based on the output of the interaction layer model to obtain the final classification result.

[0004] In existing model structures, the training assistant merely performs feature engineering on local data and transmits the results to the training initiator. It has no other involvement. After sending the training results of the underlying model to the training initiator via a protocol, it only needs to wait for updates. As a result, in order to improve efficiency, some studies have simplified the underlying structure to a single-layer perceptron structure when optimizing the model in the implementation of the system instantiation scenario. Although this can achieve the purpose, it does not effectively realize the original intention of federated learning and is not attractive enough for the training assistant to participate. Summary of the Invention

[0005] The problem this invention aims to solve is to introduce the timing nodes of data interaction and the secure transmission protocol for interaction, in order to achieve the same or even better efficiency with a smaller communication cost under the premise of secure transmission. It proposes a training method, electronic device and storage medium for a longitudinal federated scene neural network based on multi-bias interaction.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A training method for a vertical federated scene neural network based on multi-bias interaction includes the following steps:

[0008] S1. Define the model structure of each training participant as a fully connected layer and a Dropout layer. At the same time, the underlying model of each training participant has the same number of layers. The training participants include the training initiator and the training assistant.

[0009] Set the data characteristics of the training initiator as follows: This indicates that the training initiator has m-dimensional data, where y is the label attribute, and the data has been aligned.

[0010] The data characteristics of the training assistant are This indicates that the training assistant has n-dimensional data, and the data has been aligned.

[0011] S2. Use the data from the training initiator and the training assistant in step S1 to train the forward propagation method and obtain the training results of the forward propagation method.

[0012] The specific implementation method of step S2 includes the following steps:

[0013] S2.1, Underlying Model Training Method: This includes training participants calculating the intermediate results of each layer of the underlying model, processing the resulting transmission bias, exchanging and transmitting the biases among the training participants, updating the biases of the underlying model based on the interaction data, and then training the underlying model with updated biases to obtain the output results of the underlying model and proceeding to the next step.

[0014] S2.2 Interaction Layer Model Training Method: Based on the output results of the bottom-level model in step S2.1, the training participants interact and pass the data to the interaction layer model using Paillier homomorphic encryption and differential noise to obtain the overall intermediate results of the interaction layer model as the input data of the activation function of the interaction layer model. The output results of the activation function of the interaction layer model are then used to proceed to the next step.

[0015] S2.3 Top-level model training method: The training initiator uses the output of the activation function of the interaction layer model as the input of the top-level model to train the final classification model and obtain the training result of the forward propagation method.

[0016] S3. Backpropagation method: The training results of the forward propagation method obtained in step S2 are used for backpropagation training. The training assistant and the training initiator update the model parameters respectively to complete one round of training.

[0017] S4. Repeat steps S2 and S3 until the training results meet the accuracy requirements or the stopping condition is met, thus completing the training of the longitudinal federated scene neural network based on multi-bias interaction.

[0018] Furthermore, the specific implementation method of step S2.1 includes the following steps:

[0019] S2.1.1, Settings The parameters of the t-th layer of the underlying model used to train the initiator. For the parameters of the t-th layer of the underlying model used to train the assistive party:

[0020]

[0021]

[0022] Among them, the training initiator's underlying model has i neurons in the t-th layer and j neurons in the (t+1)-th layer, while the training assistant's underlying model has l neurons in the t-th layer and k neurons in the (t+1)-th layer.

[0023] S2.1.2 The training initiator and training assistant calculate the intermediate results of each layer of the underlying model, except for the last layer. The formula for calculating the intermediate results of the first layer is as follows:

[0024]

[0025]

[0026] The formulas for calculating the intermediate results of the remaining layers are as follows:

[0027]

[0028]

[0029] in and ReLU is a common instance of the activation function for fully connected layers and Dropout layers in neural networks, representing the intermediate computation results of the t-th layer for both the training initiator and the training assistant.

[0030] S2.1.3. Process the intermediate results obtained from the calculation. The processing formula is as follows:

[0031] (All intermediate results except the last layer are calculated)

[0032] (All intermediate results except the last layer are calculated)

[0033] Among them, Z B To train the propagation bias of the initiator, Z A To train the transmission bias of the assistant, the sum_row function takes matrix data as input and sums each row of the input data; the trans function takes matrix data as input and averages each column of the input data.

[0034] S2.1.4, The training initiator will Z B Send to the training assistant, and simultaneously receive Z from them. A The training initiator will Z A Each term is used as a weighting coefficient, and the model bias B = {B} is synchronously updated to the underlying model of the training initiator. i |i=1,2,…(all layers except the last one)}, Z A Save as local result

[0035] The training assistant will Z A Send to the training initiator, and simultaneously receive the Z signal sent from them. B The training assistance party will Z B Each term is used as a weighting coefficient, and the model bias A = {A} is synchronously updated to the underlying model of the training assistant. i |i=1,2,…(all layers except the last one)}, Z B Save as local result

[0036] The update formula for the first round of training is:

[0037] B i =B i ·z ai

[0038] A i =A i ·z bi

[0039] Among them, z ai For Z A The i-th term, z bi For Z B The i-th term;

[0040] The update formula for non-first round training is:

[0041]

[0042]

[0043] in, for The i-th term, for The i-th term;

[0044] S2.1.5. Each participating party trains and updates its underlying model to update the bias, obtaining the output α of the training assistant's underlying model. A The underlying model output α of the training initiator B .

[0045] Furthermore, the specific implementation method of step S2.2 includes the following steps:

[0046] S2.2.1 The training assistant uses Paillier homomorphic encryption α A , thus obtaining α A Ciphertext data [α] A ], sent to the training initiator;

[0047] S2.2.2, The training initiator receives [α] A After that, perform the calculation. The calculation formula is:

[0048] [z A ]=[αA ] ·W A

[0049] Among them, [z A W represents the intermediate result computed by the training assistant at the interaction layer in encrypted conditions. A The weights for the training assistant part of the interaction layer model;

[0050] S2.2.3, The training initiator simultaneously generates noise ε based on the dimension. B , will [z A ]+ε B The calculation formula sent to the training assistant for local computation is as follows:

[0051] z B =α B ·W B

[0052] Among them, z B W represents the intermediate result computed by the training initiator at the interaction layer in the unencrypted case. B These are the weights of the training initiator part of the interaction layer model;

[0053] S2.2.4 Training assistant calculates cumulative noise εacc ·α A And decrypt the received ciphertext to obtain z A +ε B Then, the results are summed and sent to the training initiator. The calculation formula is:

[0054]

[0055] Where, ε acc The noise ε generated by the training assistant during each round of backpropagation. A The sum of This is an intermediate result of the training assistant interaction layer containing noise;

[0056] S2.2.5, Training Initiator Receives Then, calculate the overall intermediate result z of the interaction layer using the following formula:

[0057]

[0058] Using z as the input data to the activation function of the interaction layer, we obtain the output result g(z) of the activation function of the interaction layer model.

[0059] Furthermore, the specific implementation method of step S3 includes the following steps:

[0060] S3.1 Top-level model training method: The training initiator trains the model based on the training results of the forward propagation method. Update the top-level model parameters for label y, calculate the loss δ based on the top-level model, and proceed to the next step;

[0061] S3.2 Interaction Layer Model Training Method: The training initiator calculates the loss of the interaction layer activation function based on the loss and the interaction layer model activation function. The training initiator calculates the loss to update its own underlying model parameters, and then updates the parameters of the interaction layer model corresponding to the training initiator. Then, the training initiator and the training assistant perform interactive calculations based on Paillier homomorphic encryption and differential noise to obtain the updated loss of the underlying model parameters of the training assistant. Then, the training initiator updates the parameters of the interaction layer model corresponding to the training assistant and sends it to the training assistant. The training assistant decrypts the received updated loss of the underlying model parameters of the training assistant and proceeds to the next step.

[0062] S3.3, Underlying Model Training Method: The training assistant decrypts the loss received from updating the underlying model parameters and transmits it to the underlying model. The training initiator and the training assistant then update the underlying model parameters respectively.

[0063] Furthermore, the specific implementation method of step S3.2 includes the following steps:

[0064] S3.2.1 The training initiator calculates the loss δ of the interaction layer activation function based on the loss δ and the interaction layer model activation function. act ;

[0065] S3.2.2, The training initiator first calculates the loss δ of some parameters of the underlying model. bottom_B This is used to update the parameters of the underlying model, and the calculation formula is:

[0066] δ bottom_B =δ act ·W B

[0067] S3.2.3, Based on δ act Update the parameters of the interaction layer model corresponding to the training initiator, calculated using the following formula:

[0068] W B -=η*δ act ·α B

[0069] Where η is the set model learning rate;

[0070] S3.2.4, Training initiator generates noise ε B The calculation is based on Paillier homomorphic encryption and differential noise, and the result δ is... act ·[α A ]+ε B Send to the training support provider;

[0071] S3.2.5, The training assistant will ε acc Encryption yields [ε] acc Then, noise ε is generated. A Update ε acc +=ε A After decrypting the calculation result of step S3.2.4, δ is obtained. act ·α A +ε B The training assistance party will [ε acc Transmitted to the training initiator;

[0072] S3.2.6 After receiving the result of step S3.2.5, the training initiator first sets δ... act ·(W A +[ε acc The loss of the underlying model of the training assistant is fed back to update the interaction layer model, which then updates the parameters W of the training assistant. A The calculation formula is:

[0073]

[0074] S3.2.7, Training Assistance Decryption δ act ·(W A +[ε acc The data is passed to the underlying model to update the underlying model parameters.

[0075] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the described longitudinal federated scene neural network training method based on multi-bias interaction.

[0076] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the described method for training a longitudinal federated scene neural network based on multi-bias interactions.

[0077] The beneficial effects of this invention are:

[0078] The present invention describes a longitudinal federated neural network training method based on multi-bias interaction, which innovatively modifies the model structure by adding intermediate computation result interaction to the underlying model, thereby increasing information interaction among the training parties. Simultaneously, the interaction of intermediate computation results from each party is introduced into the underlying model training stage—the stage where feature engineering functions, primarily relying on fully connected layer model structures for linear computation, feature selection, and cleaning, are performed. This allows the feature filtering function to be guided by multi-party data information, making the results more convincing. To evaluate the feasibility and practicality of introducing this interaction, it is necessary to consider protocol security and communication costs. The bias interaction and update of the underlying model is a newly introduced security protocol. This scheme considers converting the input information of the activation function of each layer calculated by each party's underlying model into an intermediate computation result, which is then transmitted as a bias value to the other party. The assisting party also treats the bias value as a combination of parameters and inputs rather than a constant; its new input form is the product of the originally generated constant bias value and the transmitted bias value. The scheme compresses the input information into values ​​by averaging the intermediate input information for each data point.

[0079] The longitudinal federated scene neural network training method based on multi-bias interaction described in this invention adds data information interaction to the bottom model part before each round of model training. For each layer of the bottom model, the intermediate result input is simulated and processed and obfuscated to obtain the intermediate value vector for transmission. Related secure transmission protocols are designed to ensure data privacy and security, as well as transmission efficiency and feasibility during information transmission. This allows the bottom model to carry information from other assisting parties during data processing and training, which is valuable for enhancing the utilization of data. Attached Figure Description

[0080] Figure 1This is a flowchart of the vertical federated scene neural network training method based on multi-bias interaction described in this invention;

[0081] Figure 2 This is a schematic diagram of the forward propagation method described in this invention;

[0082] Figure 3 This is a schematic diagram of the backpropagation method described in this invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0084] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0085] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1-3 Detailed explanation is as follows: Specific implementation method one:

[0087] A training method for a vertical federated scene neural network based on multi-bias interaction includes the following steps:

[0088] S1. Define the model structure of each training participant as a fully connected layer and a Dropout layer. At the same time, the underlying model of each training participant has the same number of layers. The training participants include the training initiator and the training assistant.

[0089] Set the data characteristics of the training initiator as follows: This indicates that the training initiator has m-dimensional data, where y is the label attribute, and the data has been aligned.

[0090] The data characteristics of the training assistant are This indicates that the training assistant has n-dimensional data, and the data has been aligned.

[0091] S2. Use the data from the training initiator and the training assistant in step S1 to train the forward propagation method and obtain the training results of the forward propagation method.

[0092] Furthermore, the specific implementation method of step S2 includes the following steps:

[0093] S2.1, Underlying Model Training Method: This includes training participants calculating the intermediate results of each layer of the underlying model, processing the resulting transmission bias, exchanging and transmitting the biases among the training participants, updating the biases of the underlying model based on the interaction data, and then training the underlying model with updated biases to obtain the output results of the underlying model and proceeding to the next step.

[0094] Furthermore, the specific implementation method of step S2.1 includes the following steps:

[0095] S2.1.1, Settings The parameters of the t-th layer of the underlying model used to train the initiator. For the parameters of the t-th layer of the underlying model used to train the assistive party:

[0096]

[0097]

[0098] Among them, the training initiator's underlying model has i neurons in the t-th layer and j neurons in the (t+1)-th layer, while the training assistant's underlying model has l neurons in the t-th layer and k neurons in the (t+1)-th layer.

[0099] S2.1.2 The training initiator and training assistant calculate the intermediate results of each layer of the underlying model, except for the last layer. The formula for calculating the intermediate results of the first layer is as follows:

[0100]

[0101]

[0102] The formulas for calculating the intermediate results of the remaining layers are as follows:

[0103]

[0104]

[0105] in and ReLU is a common instance of the activation function for fully connected layers and Dropout layers in neural networks, representing the intermediate computation results of the t-th layer for both the training initiator and the training assistant.

[0106] S2.1.3. Process the intermediate results obtained from the calculation. The processing formula is as follows:

[0107] (All intermediate results except the last layer are calculated)

[0108] (All intermediate results except the last layer are calculated)

[0109] Among them, Z B To train the propagation bias of the initiator, Z A To train the transmission bias of the assistant, the sum_row function takes matrix data as input and sums each row of the input data; the trans function takes matrix data as input and averages each column of the input data.

[0110] S2.1.4, The training initiator will Z B Send to the training assistant, and simultaneously receive Z from them. A The training initiator will Z A Each term is used as a weighting coefficient, and the model bias B = {B} is synchronously updated to the underlying model of the training initiator. i |i=1,2,…(all layers except the last one)}, Z A Save as local result

[0111] The training assistant will Z A Send to the training initiator, and simultaneously receive the Z signal sent from them. B The training assistance party will Z B Each term is used as a weighting coefficient, and the model bias A = {A} is synchronously updated to the underlying model of the training assistant. i |i=1,2,…(all layers except the last one)}, Z B Save as local result

[0112] The update formula for the first round of training is:

[0113] B i =B i ·z ai

[0114] A i =A i ·z bi

[0115] Among them, z ai For Z A The i-th term, z bi For Z B The i-th term;

[0116] The update formula for non-first round training is:

[0117]

[0118]

[0119] in, for The i-th term, for The i-th term;

[0120] S2.1.5. Each participating party trains and updates its underlying model to update the bias, obtaining the output α of the training assistant's underlying model. A The underlying model output α of the training initiator B ;

[0121] S2.2 Interaction Layer Model Training Method: Based on the output results of the bottom-level model in step S2.1, the training participants interact and pass the data to the interaction layer model using Paillier homomorphic encryption and differential noise to obtain the overall intermediate results of the interaction layer model as the input data of the activation function of the interaction layer model. The output results of the activation function of the interaction layer model are then used to proceed to the next step.

[0122] Furthermore, the specific implementation method of step S2.2 includes the following steps:

[0123] S2.2.1 The training assistant uses Paillier homomorphic encryption α A , thus obtaining α A Ciphertext data [α] A ], sent to the training initiator;

[0124] S2.2.2, The training initiator receives [α] A After that, perform the calculation. The calculation formula is:

[0125] [z A ]=[α A ]·W A

[0126] Among them, [z A W represents the intermediate result computed by the training assistant at the interaction layer in encrypted conditions. A The weights for the training assistant part of the interaction layer model;

[0127] S2.2.3, The training initiator simultaneously generates noise ε based on the dimension. B , will [z A ]+ε B The calculation formula sent to the training assistant for local computation is as follows:

[0128] z B =α B ·W B

[0129] Among them, z B W represents the intermediate result computed by the training initiator at the interaction layer in the unencrypted case. B These are the weights of the training initiator part of the interaction layer model;

[0130] S2.2.4 Training assistant calculates cumulative noise ε acc ·α A And decrypt the received ciphertext to obtain z A +ε B Then, the results are summed and sent to the training initiator. The calculation formula is:

[0131]

[0132] Where, ε acc The noise ε generated by the training assistant during each round of backpropagation. A The sum of This is an intermediate result of the training assistant interaction layer containing noise;

[0133] S2.2.5, Training Initiator Receives Then, calculate the overall intermediate result z of the interaction layer using the following formula:

[0134]

[0135] Using z as the input data to the activation function of the interaction layer, we obtain the output result g(z) of the activation function of the interaction layer model.

[0136] S2.3 Top-level model training method: The training initiator uses the output of the activation function of the interaction layer model as the input of the top-level model to train the final classification model and obtain the training result of the forward propagation method.

[0137] S3. Backpropagation method: The training results of the forward propagation method obtained in step S2 are used for backpropagation training. The training assistant and the training initiator update the model parameters respectively to complete one round of training.

[0138] Furthermore, the specific implementation method of step S3 includes the following steps:

[0139] S3.1 Top-level model training method: The training initiator trains the model based on the training results of the forward propagation method. Update the top-level model parameters for label y, calculate the loss δ based on the top-level model, and proceed to the next step;

[0140] S3.2 Interaction Layer Model Training Method: The training initiator calculates the loss of the interaction layer activation function based on the loss and the interaction layer model activation function. The training initiator calculates the loss to update its own underlying model parameters, and then updates the parameters of the interaction layer model corresponding to the training initiator. Then, the training initiator and the training assistant perform interactive calculations based on Paillier homomorphic encryption and differential noise to obtain the updated loss of the underlying model parameters of the training assistant. Then, the training initiator updates the parameters of the interaction layer model corresponding to the training assistant and sends it to the training assistant. The training assistant decrypts the received updated loss of the underlying model parameters of the training assistant and proceeds to the next step.

[0141] Furthermore, the specific implementation method of step S3.2 includes the following steps:

[0142] S3.2.1 The training initiator calculates the loss δ of the interaction layer activation function based on the loss δ and the interaction layer model activation function. act ;

[0143] S3.2.2, The training initiator first calculates the loss δ of some parameters of the underlying model. bottom_B This is used to update the parameters of the underlying model, and the calculation formula is:

[0144] δ bottom_B =δ act ·W B

[0145] S3.2.3, Based on δ act Update the parameters of the interaction layer model corresponding to the training initiator, calculated using the following formula:

[0146] W B -=η*δ act ·α B

[0147] Where η is the set model learning rate;

[0148] S3.2.4, Training initiator generates noise ε B The calculation is based on Paillier homomorphic encryption and differential noise, and the result δ is... act ·[α A ]+ε B Send to the training support provider;

[0149] S3.2.5, The training assistant will ε acc Encryption yields [ε] acc Then, noise ε is generated. A Update ε acc+=ε A After decrypting the calculation result of step S3.2.4, δ is obtained. act ·α A +ε B The training assistance party will [ε acc Transmitted to the training initiator;

[0150] S3.2.6 After receiving the result of step S3.2.5, the training initiator first sets δ... act ·(W A +[ε acc The loss of the underlying model of the training assistant is fed back to update the interaction layer model, which then updates the parameters W of the training assistant. A The calculation formula is:

[0151]

[0152] S3.2.7, Training Assistance Decryption δ act ·(W A +[ε acc The data is passed to the underlying model to update its parameters.

[0153] S3.3, Underlying Model Training Method: The training assistant decrypts the loss received from updating the underlying model parameters and transmits it to the underlying model. The training initiator and the training assistant update the underlying model parameters respectively.

[0154] S4. Repeat steps S2 and S3 until the training results meet the accuracy requirements or the stopping condition is met, thus completing the training of the longitudinal federated scene neural network based on multi-bias interaction.

[0155] The effectiveness of the vertical federated scene neural network training based on multi-bias interaction described in this embodiment is verified as follows:

[0156] To demonstrate the security of this scheme, we first define the data targets to be protected and the adversary's role. The non-leaking data includes the raw data used in training the model (i.e., the model's input dataset), denoted as X; and the model parameters of the participant's local underlying model, denoted as W. In addition, plaintext intermediate training results (gradients, updated parameters such as WX, etc.) also provide valuable information to the adversary and therefore must be protected; these intermediate results are denoted as α. The adversary is envisioned as a curious collaborator A, who always attempts to obtain data information from other collaborators through received data values.

[0157] Let the number of data points for both sides be n. For the averaging scheme, since the data has been aligned, each assisting party knows that the number of effective training data points is the same. This is equivalent to the adversary knowing that the normal bias transmission value is obtained by averaging n data points. Intuitively, it is difficult to decompose a value into multiple accurate values ​​without other information.

[0158] This process can be theoretically measured. The problem of guessing an arbitrary real number is I1, while the problem of calculating the average of the original n data points is I0. n So, question I... n This can be viewed as first guessing one piece of data, then guessing the remaining n-1 pieces of data based on that guess. In other words, we first complete problem I1, and then continue with problem I based on the result. n-1 The problem of solving it can be deduced based on the idea of ​​recursion. n The difficulty is equivalent to solving the I1 problem n-1 times simultaneously (the last one can be directly derived since the average value is known). In the non-discrete real number domain, the probability of guessing a real number correctly without any additional conditions is close to 0. Therefore, the behavior of inferring the original data based on the average value also approaches 0 as the number of data points increases. This ensures the security of the bias value interaction averaging scheme. That is, the probability of an adversary obtaining the intermediate result by relying on the bias value is equal to the probability of directly guessing the intermediate result. The intermediate result is secure, and the original data and model parameters are also protected from being reverse-trained or inferred.

[0159] An analysis of the increased time overhead of this scheme is conducted. Communication overhead mainly consists of pre-send processing and the transmission process itself. The communication overhead analysis should assume that data alignment, key and model parameter generation, and plaintext computation costs are ignored, as these time consumptions can be completed beforehand or are negligible compared to ciphertext computation. However, the newly added bias interaction communication process in the innovative underlying model uses summation and averaging data processing before data transmission, obfuscating a set of data entries into a single value. Furthermore, this value cannot be easily reversed, and its security issue is not addressed through encryption. In other words, the introduced communication is plaintext communication, which has an order-of-magnitude difference compared to the ciphertext communication overhead of the interaction layer. Therefore, the data transmission process within the newly introduced interaction cost can be ignored. Specific Implementation Method Two:

[0161] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the described longitudinal federated scene neural network training method based on multi-bias interaction.

[0162] The computer device of the present invention may include a processor and a memory, such as a microcontroller containing a central processing unit. Furthermore, when the processor executes the computer program stored in the memory, it implements the steps of the aforementioned recommendation method for modifyable relationship-driven recommendation data based on CREO software.

[0163] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0164] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices. Specific implementation method three:

[0166] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the aforementioned training method for a longitudinal federated scene neural network based on multi-bias interactions.

[0167] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by the processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. The computer-readable storage medium stores a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-described modeling method for modifyable relation-driven modeling data based on CREO software can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0168] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0169] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A training method for a vertical federated scene neural network based on multi-bias interaction, characterized in that: Includes the following steps: S1. Define the model structure of each training participant as a fully connected layer and a Dropout layer. At the same time, the underlying model of each training participant has the same number of layers. The training participants include the training initiator and the training assistant. Set the data characteristics of the training initiator as follows: , indicating that the training initiator has m-dimensional data, y is the label attribute, and the data has been aligned; The data characteristics of the training assistant are This indicates that the training assistant has n-dimensional data, and the data has been aligned. S2. Use the data from the training initiator and the training assistant in step S1 to train the forward propagation method and obtain the training results of the forward propagation method. The specific implementation method of step S2 includes the following steps: S2.1, Underlying Model Training Method: This includes training participants calculating the intermediate results of each layer of the underlying model, processing the resulting transmission bias, exchanging and transmitting the biases among the training participants, updating the biases of the underlying model based on the interaction data, and then training the underlying model with updated biases to obtain the output results of the underlying model and proceeding to the next step. S2.2, Interaction Layer Model Training Method: Based on the output results of the underlying model in step S2.1, all participating parties train the interaction layer model... Homomorphic encryption and differential noise are passed together to obtain the overall intermediate result of the interaction layer model, which is used as the input data of the activation function of the interaction layer model. The output result of the activation function of the interaction layer model is then used to proceed to the next step. S2.3 Top-level model training method: The training initiator uses the output of the activation function of the interaction layer model as the input of the top-level model to train the final classification model and obtain the training result of the forward propagation method. ; S3. Backpropagation method: The training results of the forward propagation method obtained in step S2 are used for backpropagation training. The training assistant and the training initiator update the model parameters respectively to complete one round of training. S4. Repeat steps S2 and S3 until the training results meet the accuracy requirements or the stopping condition is met, thus completing the training of the longitudinal federated scene neural network based on multi-bias interaction. To demonstrate the security of the scheme, we first present the data targets protected by the scheme and the adversary settings of the scheme; The bias interaction communication process in the underlying model of the scheme uses summation and averaging data processing before data transmission, which confuses a set of data entries into a set of single values.

2. The training method for a vertical federated scene neural network based on multi-bias interaction according to claim 1, characterized in that: The specific implementation method of step S2.1 includes the following steps: S2.1.1, Settings The parameters of the t-th layer of the underlying model used to train the initiator. For the parameters of the t-th layer of the underlying model used to train the assistive party: ； ； Among them, the training initiator's underlying model has i neurons in the t-th layer and j neurons in the (t+1)-th layer, while the training assistant's underlying model has l neurons in the t-th layer and k neurons in the (t+1)-th layer. S2.1.2 The training initiator and training assistant calculate the intermediate results of each layer of the underlying model, except for the last layer. The formula for calculating the intermediate results of the first layer is as follows: ； ； The formulas for calculating the intermediate results of the remaining layers are as follows: ； ； in and This represents the intermediate computation results at layer t for both the training initiator and the training assistant. This is a common example of an activation function for fully connected layers and Dropout layers in a neural network. S2.1.

3. Process the intermediate results obtained from the calculation. The processing formula is as follows: ； ； in, To train the transmission bias of the initiator, To train the transmission bias of the assistant, the sum_row function takes matrix data as input and sums each row of the input data; the trans function takes matrix data as input and averages each column of the input data. S2.1.4, The training initiator will Send to the training assistant, and simultaneously receive messages from them. The training organizer will Each term is used as a weighting coefficient, and the model bias is synchronously updated to the underlying model of the training initiator. ,Will Save as local result ; The training assistance party will Send to the training initiator, and simultaneously receive messages from them. The training assistance party will Each term is used as a weighting coefficient, and the model bias is synchronously updated to the underlying model of the training assistant. ,Will Save as local result ; The update formula for the first round of training is: ； ； in, for The i-th term, for The i-th term; The update formula for non-first round training is: ； ； in, for The i-th term, for The i-th term; S2.1.

5. Each participating party trains and updates its underlying model to update the bias, obtaining the output of the training assistant's underlying model. The output of the underlying model of the training initiator .

3. The method for training a vertical federated scene neural network based on multi-bias interaction according to claim 2, characterized in that: The specific implementation method of step S2.2 includes the following steps: S2.2.1 Training assistance party uses Homomorphic encryption ,get encrypted data Send it to the training initiator; S2.2.2, The training initiator receives... Then, calculations are performed using the following formula: ； in, This refers to the intermediate results calculated by the training assistant at the interaction layer under encrypted conditions. The weights for the training assistant part of the interaction layer model; S2.2.3, The training initiator simultaneously generates noise based on the dimension. ,Will The calculation formula sent to the training assistant for local computation is as follows: ； in, This represents the intermediate result computed by the training initiator at the interaction layer in the unencrypted case. These are the weights of the training initiator part of the interaction layer model; S2.2.4 Training Assistance Calculation of Cumulative Noise And decrypt the received ciphertext to obtain Then, the results are summed and sent to the training initiator. The calculation formula is: ； in, The noise generated by the training assistant during each round of backpropagation. The sum of This is an intermediate result of the training assistant interaction layer containing noise; S2.2.5, Training Initiator Receives Then, calculate the overall intermediate results of the interaction layer. The calculation formula is: ； Will The data is used as input to the activation function of the interaction layer to obtain the output of the activation function of the interaction layer model. .

4. The method for training a vertical federated scene neural network based on multi-bias interaction according to claim 3, characterized in that: The specific implementation method of step S3 includes the following steps: S3.1 Top-level model training method: The training initiator trains the model based on the training results of the forward propagation method. ,Label Update the top-level model parameters and calculate the loss based on the top-level model. Proceed to the next step; S3.2, Interaction Layer Model Training Method: The training initiator calculates the loss of the interaction layer activation function based on the loss and the interaction layer model activation function. The training initiator calculates the loss for updating its own underlying model parameters, then updates the parameters of the interaction layer model corresponding to the training initiator. Then, the training initiator and training assistant... Homomorphic encryption and differential noise are used to perform interactive calculations to obtain the loss of the updated training assistant's underlying model parameters. Then, the training initiator updates the parameters of the interactive layer model corresponding to the training assistant and sends them to the training assistant. The training assistant decrypts the received loss of the updated training assistant's underlying model parameters and proceeds to the next step. S3.3, Underlying Model Training Method: The training assistant decrypts the loss received from updating the underlying model parameters and transmits it to the underlying model. The training initiator and the training assistant then update the underlying model parameters respectively.

5. The training method for a vertical federated scene neural network based on multi-bias interaction according to claim 4, characterized in that: The specific implementation method of step S3.2 includes the following steps: S3.2.1, Training initiator based on loss 1. Activation function of the interaction layer model, and calculate the loss of the activation function of the interaction layer. ; S3.2.2 The training initiator first calculates the loss of some parameters of the underlying model. This is used to update the parameters of the underlying model, and the calculation formula is: ； S3.2.3, based on Update the parameters of the interaction layer model corresponding to the training initiator, calculated using the following formula: ； in, The set model learning rate; S3.2.4, Training initiator generates noise ,based on Homomorphic encryption and differential noise are calculated, and the results are... Send to the training support provider; S3.2.5, The training assistance party will Encryption obtained Then noise is generated. ,renew After decrypting the calculation result of step S3.2.4, we obtain... The training assistance party will , Transmitted to the training initiator; S3.2.6 After receiving the result of step S3.2.5, the training initiator first... The loss is fed back to the training assistant as the loss of the underlying model of the training assistant, and then the parameters of the interaction layer model corresponding to the training assistant are updated. The calculation formula is: ； S3.2.7, Training Assistance Decryption It is then passed to the underlying model to update the underlying model parameters.

6. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the longitudinal federated scene neural network training method based on multi-bias interaction as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the longitudinal federated scene neural network training method based on multi-bias interaction as described in any one of claims 1-5.

Images