MPC-based longitudinal federated neural network training method and apparatus, and reasoning method

Abstract

The invention discloses a training method and device of a longitudinal federated neural network based on MPC. In order to solve the problem that longitudinal federal splitting learning is easily affected by model completion attacks, the front n layers of the neural network adopt a model fragmentation technology, and a complete bottom model does not exist, so that the model completion attacks cannot be initiated. In order to solve the efficiency problem caused by model fragmentation, after n layers, arithmetic addition is carried out on the model fragmentation and the fragmentation of the calculation result by referring to the existing model sharing patent thought, namely, a single-server training and reasoning mode is adopted, so that the operation speed is increased. As n is usually 1 or 2, the training and reasoning efficiency of the whole system is close to the effect of a plaintext, and meanwhile, the security privacy characteristic is kept. For a scene of a multi-label provider, the model and the result are split again by utilizing the characteristic that the model and the model result can be split and combined, so as to protect the privacy of the label. And the whole structure is still suitable for an application scene of split learning.

Description

Technical Field

[0001] The present invention relates to the technical field of data security and privacy utilization, and is a privacy-preserving fast privacy calculation method for a vertical federated neural network, and in particular to a training method and device, and an inference method for a vertical federated neural network based on MPC. Background Art

[0002] Privacy computing provides a method to use data while protecting security and privacy. Currently, the commonly used privacy computing solutions include privacy computing based on secure multiparty computation (SMPC), privacy computing based on federated learning (FL), and privacy computing based on trusted execution environment (TEE).

[0003] Federated learning can be divided into horizontal federation, vertical federation and migration federation. For vertical federated learning, a large number of application scenarios are suitable for the split neural network method. This method splits the neural network model into two parts: bottom and top models. The bottom model is trained locally by the client that provides the data, and the intermediate results are sent to the server. The server aggregates the intermediate results of multiple parties and adds its own intermediate results to complete the remaining neural network operations. In the back-propagation stage, the server transmits the gradient to each client, and each client completes the subsequent back-propagation. Usually the server is operated by the active party with the label.

[0004] Many studies have shown that the client can launch a label inference attack on the label through the received gradient or the bottom model or the output of the model, and the server can obtain the client's data features through the intermediate results of the bottom output. Although there have been a lot of research to prevent the above attacks, according to the research "A Framework for Evaluating Privacy-Utility Trade-off in Vertical Federated Learning" updated in August 24, model inversion attacks and most label inference attacks can be protected according to existing methods, but model completion attacks are difficult to prevent. This is mainly because the trained bottom model is trained with labels and data, and already has the ability to divide data, so only a small amount of additional labeled data is needed to generate a complete neural network based on the bottom model for predicting labels.

[0005] It can be seen from this that the existing vertical federated neural network has the technical problem of model completion attack. Summary of the invention

[0006] In view of the deficiencies of the prior art, the present invention provides a training method and device, and an inference method for a longitudinal federated neural network based on MPC.

[0007] The technical solution adopted by the present invention is as follows:

[0008] In a first aspect, a training method for a vertical federated neural network based on MPC is provided, which is applied to a system including a data provider and two servers, wherein the first n layers of the neural network model are respectively deployed on a first server and a second server in an arithmetic slicing manner, and all layers above the n layers of the neural network model are deployed on the second server, wherein the training method comprises:

[0009] The data providers perform attribute expansion on their respective sample data, and divide the attribute-expanded sample data into slices, and then send the slices to the first server and the second server;

[0010] When the first server and the second server receive a certain number of sample fragments, if a sample has a label, the attributes of all fragments of the sample are arithmetic added and divided by the number of fragments to obtain a first vector as training data;

[0011] The first server and the second server run the first n layers of the neural network model for training based on the training data according to a known two-party security protocol. In the forward propagation stage, when the neural network model runs to the n+1 layer, the first server sends the calculation result of its own n-th layer output to the second server, and the second server performs arithmetically addition of the received calculation result and its own calculation result as the input of the n+1 layer, and runs the subsequent neural network; in the reverse propagation stage, when the neural network model runs to the nth layer, the second server performs arithmetically slicing on the calculated n-layer gradient, and sends one of the gradient slices to the first server, and then the first server and the second server each complete the reverse propagation process of the gradient, and n is a positive integer.

[0012] In one embodiment, when a sample has a label, the method further includes the data provider sharding the label.

[0013] In one embodiment, the method further includes: when a sample fragment has no label, the corresponding sample fragment is retained for next batch processing; if a sample fragment has no label within a preset time, the corresponding sample fragment is deleted.

[0014] In one embodiment, when only one of the data providers provides a tag, the method further includes: the second server is operated by the data provider that provides the tag.

[0015] In one embodiment, when there are multiple provided labels among the data providers, the method further includes: splitting the model weight parameters of the mk to m layers in the initialization phase, specifically:

[0016] In the forward propagation stage, when running to the mk layer, where k = 1 or 2, the second server splits the calculation results of the model in the last layer or two layers of the model, specifically: the input calculation results of the mk layer are sliced, and then the first server and the second server respectively use their own slices as input to run their own neural networks; in the backward propagation stage, the first server and the second server respectively calculate the difference between the label slice and the output of their own neural network, use the chain rule to calculate the gradient of the difference to each parameter of the model, and use the gradient to update the parameters of the model, where m represents the total number of layers of the neural network model;

[0017] In the back propagation stage, when the back propagation runs to the end of the mk layer, the first server sends the gradient slices propagated in this layer to the second server. The second server calculates the arithmetic sum of its own gradient slices and the received gradient slices, and then continues the back propagation using the chain rule. When the back propagation runs to the n layer, the gradient of the n layer is arithmetically sliced, and one of the gradient slices is sent to the first server; then the first server and the second server each calculate the gradient of the model parameters based on their own gradient slices using the chain rule, and use the gradient to update the model parameters.

[0018] Based on the same inventive concept, the second aspect of the present invention provides a vertical federated neural network training device based on MPC, which is applied to a system including a data provider and two servers, wherein the first n layers of the neural network model are respectively deployed on a first server and a second server in an arithmetic slicing manner, and all layers above the nth layer of the neural network model are deployed on the second server, and the training device comprises:

[0019] The sample attribute expansion module is used for the data provider to expand the attributes of their respective sample data, and to slice the attribute-expanded sample data and then send it to the first server and the second server;

[0020] A training data acquisition module, configured to, when the first server and the second server receive a certain number of sample fragments, if a sample has a label, perform arithmetic summation on the attributes of all fragments of the sample and divide the sum by the number of fragments to obtain a first vector as training data;

[0021] A training module is used for the first server and the second server to run the first n layers of the neural network model for training based on the training data according to a known two-party security protocol. In the forward propagation stage, when the neural network model runs to the n+1 layer, the first server sends the calculation result of its own n-th layer output to the second server, and the second server performs arithmetic addition of the received calculation result and its own calculation result as the input of the n+1 layer to run the subsequent neural network; in the reverse propagation stage, when the neural network model runs to the nth layer, the second server performs arithmetic slicing on the calculated n-layer gradient, and sends one of the gradient slices to the first server, and then the first server and the second server each complete the reverse propagation process of the gradient, and n is a positive integer.

[0022] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the training method of the MPC-based vertical federated neural network described in the first aspect is implemented.

[0023] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of the MPC-based vertical federated neural network described in the first aspect is implemented.

[0024] Based on the same inventive concept, the fourth aspect of the present invention provides an MPC-based longitudinal federated neural network reasoning method, which uses the neural network model obtained in the first aspect to perform reasoning calculations.

[0025] In one implementation, the neural network model obtained according to the first aspect is used to perform reasoning calculations, including:

[0026] Each data provider provides data, and the neural network model is run according to the forward propagation process. When only one data provider provides a label, the output of the neural network model is used as the inference result; when multiple data providers provide labels, the output results of the first server and the second server are arithmetic summed as the inference result.

[0027] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0028] Since the first n layers use model sharding technology, there is no complete bottom model, so model completion attacks cannot be launched. In order to solve the efficiency problem caused by model sharding, after the nth layer, the model shards and the shards of the calculation results are arithmetically added by drawing on the existing model sharing patent idea, that is, a single-server training and reasoning mode is adopted to speed up the operation. Since n is usually 1 or 2, the training and reasoning efficiency of the entire system is close to the plaintext effect, while maintaining security and privacy characteristics. For scenarios with multiple label providers, the model and model results can be split and merged to protect the privacy of the labels. And the overall structure is still suitable for the application scenario of split learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A flowchart of a training method for a longitudinal federated neural network based on MPC provided in an embodiment of the present invention;

[0031] Figure 2 A system overall architecture diagram of a training method for a longitudinal federated neural network based on MPC provided in an embodiment of the present invention;

[0032] Figure 3 Schematic diagram of data sample expansion in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The inventors of this application have found through extensive research and practice that although the existing machine learning algorithms based on secure multi-party computing are secure and private, they cannot be used in practice due to performance issues. Although vertical federation based on split learning has many protective measures for security and privacy protection, there is still the problem of model completion attack. Therefore, the present invention intends to use a method based on secure multi-party computing to solve the model completion attack problem faced by vertical federated deep neural networks, while improving the speed of the solution. In order to prevent the passive party from launching a model completion attack, this solution adopts an MPC-based machine learning method to perform arithmetic sharding on the neural network model parameters, so that the passive party does not have a complete sub-model, so it is impossible to launch a model completion attack. In order to reduce the efficiency reduction problem caused by model sharding, the neural network is layered, the bottom layer adopts a sharding method, and the high layer merges the model sharding and the result sharding, and adopts a single server method to improve the speed. The overall solution is safe and practical. When there are multiple label providers, in order to protect the privacy of the label, the model and the result are split again by using the splittable and mergeable characteristics of the model and the model results.

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Embodiment 1

[0036] The present invention discloses a training method of a vertical federated neural network based on MPC, which is applied to a system including a data provider and two servers. The first n layers of the neural network model are respectively deployed on the first server and the second server in an arithmetic sharding manner, and all layers above the nth layer of the neural network model are deployed on the second server. Figure 1 , the training method includes:

[0037] S1: The data providers extend the attributes of their respective sample data, split the attribute-extended sample data into pieces, and then send them to the first server and the second server;

[0038] S2: After the first server and the second server receive a certain number of sample shards, if a sample has a label, the attributes of all shards of the sample are arithmetic added and divided by the number of shards to obtain a first vector as training data;

[0039] S3: The first server and the second server run the first n layers of the neural network model for training based on the training data according to a known two-party security protocol. In the forward propagation stage, when the neural network model runs to the n+1 layer, the first server sends the calculation result of its own n-th layer output to the second server, and the second server performs arithmetically addition of the received calculation result and its own calculation result as the input of the n+1 layer, and runs the subsequent neural network; in the reverse propagation stage, when the neural network model runs to the nth layer, the second server performs arithmetically slicing on the calculated n-layer gradient, and sends one of the gradient slices to the first server, and then the first server and the second server each complete the reverse propagation process of the gradient, and n is a positive integer.

[0040] Specifically, see Figure 2 , which is a system overall architecture diagram of the MPC-based vertical federated neural network training method provided in an embodiment of the present invention.

[0041] The overall system architecture includes a data provider and a first server P1 and a second server P2. The value of n can be determined as needed, for example, n = 1 or 2. When n = 2, it means that the first two layers of the neural network model are deployed on P1 and P2 respectively in the form of arithmetic slicing, and all layers of the neural network model above layer 2 are deployed on P2. The parameters of the model are assigned random initial values, and the entire neural network has m layers.

[0042] See also Figure 3 , the data provider expands the attributes of its own sample data so that each sample data contains all the characteristic attributes of all data provider data, and the attributes of each data provider are relatively concentrated in the expanded sample attributes. If the sample of the data provider has a certain characteristic attribute, the actual value is directly assigned to the attribute, otherwise the attribute is assigned to 0. The data provider performs arithmetic slicing on the expanded sample data. In this way, the method of the present invention can be applied to more application scenarios.

[0043] In one embodiment, if a sample has a label, the data provider needs to segment the label and send it to the first server and the second server.

[0044] The training process includes the forward propagation stage and the back propagation stage. The back propagation stage mainly includes two servers calculating the difference (error) between the sample label and the output result of the neural network forward propagation, and calculating the derivative of the error to each parameter of the model according to the chain rule, that is, the gradient. According to the learning rate, the various parameters of the model are updated.

[0045] In one embodiment, the method further includes: when a sample fragment has no label, the corresponding sample fragment is retained for next batch processing; if a sample fragment has no label within a preset time, the corresponding sample fragment is deleted.

[0046] Specifically, after the first server and the second server receive a certain number of the above sample fragments, if a sample has a label, the attributes of all fragments of the sample are arithmetic summed and divided by the number of fragments (the result is a vector, each value of which represents a characteristic attribute, and the value is the average value of the fragments of the attribute); the sample fragments without labels are reserved for the next batch processing, and if the sample has no label within a preset time, the sample fragment is deleted. In addition, the sample also includes a sample ID, which is a number and needs to have the ability to identify the sample.

[0047] It should be noted that after arithmetic slicing of all attributes of a sample, two vectors are obtained, and the above result is one of the vectors. However, since the attributes of the sample are provided by multiple providers, each provider provides a slicing, it is necessary to perform arithmetic addition on these slicing and calculate the average.

[0048] Regarding the neural network after running, for P2, the neural network run by P2 can select a commonly used neural network model depending on the application scenario, and run it according to the rules of the normal neural network model. The result obtained is the output result of the neural network model.

[0049] Specifically, the model has weight parameters, and arithmetically sharing these weight parameters is model sharding. Model sharding is actually the random initialization of each independently. In order to speed up the convergence of the system, the initial weight parameters of the model can also use the weight parameters of similar existing models, and then perform arithmetic sharding. Gradient refers to the derivative of the error with respect to the model parameters, and arithmetically sharing this value is gradient sharding. Representation refers to the calculation results of each layer of the neural network, and arithmetically sharing these results is representation sharding.

[0050] For the case of only one label party, during forward propagation, P1 sends the output representation of layer n to P2 at the end of layer n. P2 adds the slice and its own slice as the input of the neural network of layer n+1. During backward propagation, the gradient from layer n+1 is sliced ​​and sent to layer n to calculate the gradient of layer n and update the model parameters.

[0051] For the case of multiple label parties, during forward propagation, P1 sends the output Representation slice of layer n to P2 at the end of layer n. P2 adds the slice and its own slice as the input of the n+1 layer neural network.

[0052] In one embodiment, when only one of the data providers provides a tag, the method further includes: the second server is operated by the data provider that provides the tag.

[0053] In one embodiment, when there are multiple provided labels among the data providers, the method further includes: splitting the model weight parameters of the mk to m layers in the initialization phase, specifically:

[0054] In the forward propagation stage, when running to the mk layer, where k = 1 or 2, the second server splits the model in the last layer or two layers of the model, specifically: the input calculation result of the mk layer is sliced, and then the first server and the second server respectively use their respective calculation results as input to run their respective neural networks; in the backward propagation stage, the first server and the second server respectively calculate the difference between the label slice and the output of their respective neural networks, use the chain rule to calculate the gradient of the difference to each parameter of the model, and use the gradient to update the parameters of the model, where m represents the total number of layers of the neural network model;

[0055] In the back propagation stage, when the back propagation runs to the end of the mk layer, the first server sends the gradient slices propagated in this layer to the second server. The second server calculates the arithmetic sum of its own gradient slices and the received gradient slices, and then continues the back propagation using the chain rule. When the back propagation runs to the n layer, the gradient of the n layer is arithmetically sliced, and one of the gradient slices is sent to the first server; then the first server and the second server each calculate the gradient of the model parameters based on their own gradient slices using the chain rule, and use the gradient to update the model parameters.

[0056] Specifically, when there are multiple data providers providing labels (multiple label providers), they all have trust issues with P2. In order to protect the privacy of labels of all parties, the labels need to be sharded. Figure 2As shown in the dotted box, in the forward propagation stage, when the system runs to the mk layer, P2 slices the input representation of the mk layer (the calculation results of each layer of the neural network) and passes one of the slices to P1. Then P1 and P2 use their respective representations as input and run their respective neural networks. In the backward propagation stage, P1 calculates the difference between the label slice and the output of the P1 neural network, uses the chain rule to calculate the derivative (i.e., gradient) of the difference to the P1 model parameters, and uses the gradient to update the model parameters. The processing of P2 is similar. When the back propagation runs to the end of the mk layer, P1 sends the gradient slice propagated by this layer to P2. P2 calculates the arithmetic sum of its own gradient slice and the received gradient slice, and then continues the back propagation using the chain rule. When the back propagation runs to the n layer, the gradient of the n layer needs to be arithmetically sliced, and one of the gradient slices is sent to P1. Then P1 and P2 each calculate the gradient of the model parameters based on their own gradient slices using the chain rule, and use them to update the model parameters. So far, a round of training has been completed.

[0057] It should be noted that splitting the model means doing arithmetic secret sharing for each parameter of the model, that is, mod_var = mod_var1 + mod_var2, where mod_var1 and mod_var2 are the parameters of the P1 and P2 models respectively. representation = representation1 + representation2, where representation1 and representation2 are the input vectors of the P1 and P2 models respectively. When there are multiple label providers, the high-level neural network is split into two models in the last 1 or 2 layers to protect the privacy of the labels.

[0058] Embodiment 2

[0059] Based on the same inventive concept, this embodiment discloses a vertical federated neural network training device based on MPC, which is applied to a system including a data provider and two servers. The first n layers of the neural network model are respectively deployed on the first server and the second server in an arithmetic slicing manner, and all layers above the n layers of the neural network model are deployed on the second server. The training device includes:

[0060] The sample attribute expansion module is used for the data provider to expand the attributes of their respective sample data, and to slice the attribute-expanded sample data and then send it to the first server and the second server;

[0061] A training data acquisition module, configured to, when the first server and the second server receive a certain number of sample fragments, if a sample has a label, perform arithmetic summation on the attributes of all fragments of the sample and divide the sum by the number of fragments to obtain a first vector as training data;

[0062] A training module is used for the first server and the second server to run the first n layers of the neural network model for training based on the training data according to a known two-party security protocol. In the forward propagation stage, when the neural network model runs to the n+1 layer, the first server sends the calculation result of its own n-th layer output to the second server, and the second server performs arithmetic addition of the received calculation result and its own calculation result as the input of the n+1 layer to run the subsequent neural network; in the reverse propagation stage, when the neural network model runs to the nth layer, the second server performs arithmetic slicing on the calculated n-layer gradient, and sends one of the gradient slices to the first server, and then the first server and the second server each complete the reverse propagation process of the gradient, and n is a positive integer.

[0063] Since the device introduced in the second embodiment of the present invention is a device used to implement the training method of the longitudinal federated neural network based on MPC in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the device, so it is not repeated here. All devices used in the method in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0064] Embodiment 3

[0065] Based on the same inventive concept, this embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first embodiment is implemented.

[0066] Since the computer-readable storage medium introduced in the third embodiment of the present invention is a computer-readable storage medium used to implement the training method of the vertical federated neural network based on MPC in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer-readable storage medium, so it is not repeated here. All computer-readable storage media used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0067] Embodiment 4

[0068] This embodiment further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method described in the first embodiment is implemented when the processor executes the program.

[0069] Since the computer device introduced in the fourth embodiment of the present invention is a computer device used to implement the training method of the longitudinal federated neural network based on MPC in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer device, so it is not repeated here. All computer devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0070] Embodiment 5

[0071] This embodiment also provides an MPC-based longitudinal federated neural network reasoning method, which uses the neural network model obtained by the training method described in Embodiment 1 to perform reasoning calculations.

[0072] The process of inference calculation includes:

[0073] Each data provider provides data, and the neural network model is run according to the forward propagation process. When only one data provider provides a label, the output of the neural network model is used as the inference result; when multiple data providers provide labels, the output results of the first server and the second server are arithmetic summed as the inference result.

[0074] In general, in terms of security and privacy, compared with the neural network based on two-party secure computation, P1 of the present invention does not obtain additional information, so as long as the neural network based on two-party secure computation is secure, P1 cannot launch any reasoning attack on the label, including MC attack. For P2, it obtains the representation of P1 = σ 1 (W 1 (σ 0 (W 0 (A||B), where A represents the data shard owned by P1, B represents the data shard sent by P2, and W 0 , W 1 is the weight or convolution kernel of the first two layers of the neural network owned by P1, and σ 0 , σ 1 is the corresponding activation function. Since P2 does not know A, W 0 , W 1 Therefore, it is impossible to obtain the exact information of input data A from representation, nor can it obtain the model parameters of P1. Moreover, neither P1 nor P2 can obtain complete model information independently, and they still need cooperation to perform model reasoning.

[0075] In terms of speed, compared with the centralized model training, the present invention only has 1 or 2 layers of additional computation. If the number of neural network layers is relatively large, this solution will achieve a significant speed improvement and achieve practical efficiency. For multi-label scenarios, only 2 to 4 layers of additional computation are required.

[0076] Due to the feature engineering of data, this solution can be applied to more application scenarios and make better use of data. For example, if multiple departments of multiple hospitals jointly model, multiple hospitals can provide labels. The current solution generally assumes that the labels are provided by one party.

[0077] Moreover, due to the centralized data processing of this scheme, P1 and P2 can shuffle the data, which naturally solves the non-independent and identically distributed problem caused by multiple data, making the model achieve better accuracy.

[0078] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic creative concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A training method for a longitudinal federated neural network based on MPC, characterized in that: Applied to a system including a data provider and two servers, the first n layers of the neural network model are respectively deployed on a first server and a second server in an arithmetic slicing manner, and all layers above the n layers of the neural network model are deployed on the second server, and the training method includes: The data providers perform attribute expansion on their respective sample data, and divide the attribute-expanded sample data into slices, and then send the slices to the first server and the second server; When the first server and the second server receive a certain number of sample fragments, if a sample has a label, the attributes of all fragments of the sample are arithmetic added and divided by the number of fragments to obtain a first vector as training data; The first server and the second server run the first n layers of the neural network model for training based on the training data according to a known two-party security protocol. In the forward propagation stage, when the neural network model runs to the n+1 layer, the first server sends the calculation result of its own n-th layer output to the second server, and the second server performs arithmetically addition of the received calculation result and its own calculation result as the input of the n+1 layer, and runs the subsequent neural network; in the reverse propagation stage, when the neural network model runs to the nth layer, the second server performs arithmetically slicing on the calculated n-layer gradient, and sends one of the gradient slices to the first server, and then the first server and the second server each complete the reverse propagation process of the gradient, and n is a positive integer.

2. The method for training a longitudinal federated neural network based on MPC as claimed in claim 1, characterized in that: When a sample has a label, the method further includes the data provider sharding the label.

3. The training method of the MPC-based longitudinal federated neural network according to claim 2, characterized in that: The method further includes: when a sample segment has no label, the corresponding sample segment is reserved for the next batch processing; if a sample segment has no label within a preset time, the corresponding sample segment is deleted.

4. The training method of the MPC-based longitudinal federated neural network according to claim 1, characterized in that: When only one of the data providers provides a tag, the method further includes: the second server is operated by the data provider that provides the tag.

5. The training method of the MPC-based longitudinal federated neural network according to claim 1, characterized in that: When there are multiple provided labels among the data providers, the method further includes: splitting the model weight parameters of the mk to m layers in the initialization stage, specifically: In the forward propagation stage, when running to the mk layer, where k = 1 or 2, the second server splits the calculation results of the model in the last layer or two layers of the model, specifically: the input result of the mk layer is sliced, and then the first server and the second server respectively use their own slices as input to run their own neural networks; in the backward propagation stage, the first server and the second server respectively calculate the difference between the label slice and the output of their own neural network, use the chain rule to calculate the gradient of the difference to each parameter of the model, and use the gradient to update the parameters of the model, where m represents the total number of layers of the neural network model; In the back propagation stage, when the back propagation runs to the end of the mk layer, the first server sends the gradient slices propagated in this layer to the second server. The second server calculates the arithmetic sum of its own gradient slices and the received gradient slices, and then continues the back propagation using the chain rule. When the back propagation runs to the n layer, the gradient of the n layer is arithmetically sliced, and one of the gradient slices is sent to the first server; then the first server and the second server each calculate the gradient of the model parameters based on their own gradient slices using the chain rule, and use the gradient to update the model parameters.

6. A longitudinal federated neural network training device based on MPC, characterized in that: Applied to a system including a data provider and two servers, the first n layers of the neural network model are respectively deployed on a first server and a second server in an arithmetic slicing manner, and all layers above the n layers of the neural network model are deployed on the second server, and the training device includes: The sample attribute expansion module is used for the data providers to expand the attributes of their respective sample data, and to slice the attribute-expanded sample data and then send them to the first server and the second server; A training data acquisition module, configured to, when the first server and the second server receive a certain number of sample fragments, if a sample has a label, perform arithmetic summation on the attributes of all fragments of the sample and divide the sum by the number of fragments to obtain a first vector as training data; A training module is used for the first server and the second server to run the first n layers of the neural network model for training based on the training data according to a known two-party security protocol. In the forward propagation stage, when the neural network model runs to the n+1 layer, the first server sends the calculation result of its own n-th layer output to the second server, and the second server performs arithmetic addition of the received calculation result and its own calculation result as the input of the n+1 layer to run the subsequent neural network; in the reverse propagation stage, when the neural network model runs to the nth layer, the second server performs arithmetic slicing on the calculated n-layer gradient, and sends one of the gradient slices to the first server, and then the first server and the second server each complete the reverse propagation process of the gradient, and n is a positive integer.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the training method of the MPC-based longitudinal federated neural network as claimed in any one of claims 1 to 5 is implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the training method of the MPC-based vertical federated neural network as described in any one of claims 1 to 5 is implemented.

9. A reasoning method for a longitudinal federated neural network based on MPC, characterized in that: The neural network model obtained according to any one of claims 1 to 5 is used to perform inference calculations.

10. The reasoning method of the MPC-based longitudinal federated neural network according to claim 9, characterized in that: Using the neural network model obtained according to any one of claims 1 to 5 to perform reasoning calculations, comprising: Each data provider provides data, and the neural network model is run according to the forward propagation process. When only one data provider provides a label, the output of the neural network model is used as the inference result; when multiple data providers provide labels, the output results of the first server and the second server are arithmetic summed as the inference result.

Images