Method and apparatus for training neural network model by multiple parties

Abstract

The embodiment of the specification provides a method and device for jointly training a neural network model by multiple parties, the method comprising: obtaining first weight fragments and first feature fragments of a model weight parameter and feature data of a training sample at a first precision in a first ring; inputting the first feature fragments and the first weight fragments into the neural network model, and obtaining each output fragment of each network layer by performing multi-party joint calculation corresponding to forward propagation with other parties; obtaining each output fragment, the weight parameter and a label of the training sample, and obtaining second output fragments, second weight fragments and second label fragments at a second precision in a second ring; the second ring is larger than the first ring, and the second precision is higher than the first precision; and performing multi-party joint calculation corresponding to back propagation with other parties by using the second output fragments, the second weight fragments and the second label fragments, and obtaining each gradient fragment of each network layer. The model training efficiency can be improved under the premise of ensuring the final precision of the model.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computers, and more particularly to methods and apparatus for multi-party joint training of neural network models. Background Technology

[0002] Currently, data held by different data holders may contain users' private information, and data sharing between data holders may infringe on users' privacy. In order to facilitate the flow of data among multiple parties, secure multi-party computation can be used to support joint computation among multiple parties, extract the value of data, and ensure that plaintext information of each party's private data is not leaked during multi-party interactions.

[0003] Secure multi-party computation enables multiple untrusted parties to securely compute a given function without revealing the inputs or intermediate results other than the final output. Secret sharing involves distributing a secret among different parties, each receiving a portion of the secret, called a slice. The secret can only be reconstructed when enough slices are held; a single slice is insufficient. Due to its high efficiency in arithmetic and linear algebra operations, secret sharing is widely used for secure computation in various scenarios, such as multi-party joint training of neural network models.

[0004] While existing secure computation based on secret sharing offers robust security, it incurs significant communication overhead, resulting in low efficiency for joint computation. This is particularly problematic in machine learning scenarios, where model training requires multiple iterations. As models become more complex, the communication and computational overhead further hinders its practical usability. Therefore, a method for joint training of neural network models is needed that can improve training efficiency while maintaining the model's final accuracy. Summary of the Invention

[0005] This specification describes one or more embodiments of a method and apparatus for multi-party joint training of a neural network model, which can improve model training efficiency while ensuring the final accuracy of the model.

[0006] Firstly, a method for jointly training a neural network model by multiple parties is provided, wherein the method is performed by any one of the parties, including:

[0007] The current weight parameters of the model and the feature data of the training samples are obtained, respectively, in the first ring, with the first weight slice and the first feature slice having the first precision.

[0008] The first feature slice and the first weight slice are input into the neural network model to be trained, and each output slice of each network layer is obtained by performing a first multi-party joint calculation with other parties corresponding to the forward propagation.

[0009] Each output slice is obtained, and the weight parameters and the labels of the training samples are respectively in the second ring with the second output slice, the second weight slice, and the second label slice having the second precision; wherein, the second ring is larger than the first ring, and the second precision is higher than the first precision;

[0010] Using the second output slice, the second weight slice, and the second label slice, a second multi-party joint computation corresponding to backpropagation is performed with other parties to obtain the gradient slices of each network layer with the second precision in the second ring.

[0011] The second weight slice is updated using each gradient slice to achieve the current round of training of the neural network model.

[0012] In one possible implementation, the current training round is the first training round; the acquisition of the first weight slice includes:

[0013] Obtain the initial weight slices of the second precision in the second ring, based on the pre-set weight parameters of the model;

[0014] By performing a right-shifting operation with other parties, subtracting the first precision bits from the second precision bit, the initial weight slice is converted into a transformed weight slice with the first precision in the second ring.

[0015] By performing a modular conversion operation in conjunction with other parties, the conversion weight slice is converted into the first weight slice.

[0016] In one possible implementation, the current training round is not the first training round; the acquisition of the first weight slice includes:

[0017] For the second weight slice obtained in the previous training, by performing a right shift operation of the second precision minus the first precision bit in conjunction with other parties, the second weight slice is converted into a transformed weight slice of the first precision in the second ring.

[0018] By performing a modular conversion operation in conjunction with other parties, the conversion weight slice is converted into the first weight slice.

[0019] In one possible implementation, each network layer includes a first intermediate layer, a second intermediate layer, and a third intermediate layer arranged in sequence, the first and third intermediate layers corresponding to a first precision calculation, the second intermediate layer corresponding to a second precision calculation, and the first multi-party joint calculation including:

[0020] The first output fragment of the first precision in the first ring of the first intermediate layer is converted into the second output fragment of the second precision in the second ring through a modulus conversion operation performed in conjunction with other parties;

[0021] The second output fragment corresponding to the first intermediate layer is input into the second intermediate layer, and the second output fragment of the second intermediate layer is obtained through joint calculation with other parties;

[0022] The second output fragment of the second intermediate layer is converted into the first output fragment of the first precision in the first ring through a modulus conversion operation performed jointly with other parties, and the first output fragment is used as the input of the third intermediate layer.

[0023] Furthermore, the first intermediate layer and the third intermediate layer each include linear operations and activation function operations, and the second intermediate layer is used to perform batch normalization operations.

[0024] In one possible implementation, each output slice includes a first output slice of first precision in a first ring, output by a certain network layer in each network layer.

[0025] The acquisition of the second output fragment includes:

[0026] Obtain the first output fragment;

[0027] For the first output slice, through a modulus conversion operation performed jointly with other parties, the first output slice is converted into a first-precision converted output slice in the second ring;

[0028] By performing a left-shift operation with other parties, subtracting the first precision bit from the second precision bit, the converted output fragment is converted into a second output fragment with the second precision in the second ring.

[0029] In one possible implementation, the current training round is the first training round; the acquisition of the second weight slice includes:

[0030] Obtain the initial weight slices of the second precision in the second ring, based on the pre-set weight parameters of the model;

[0031] For the initial weighted slice, the second precision bit is subtracted from the first precision bit and set to 0 to obtain the second weighted slice.

[0032] In one possible implementation, the current training round is not the first training round; the acquisition of the second weight slice includes:

[0033] For the second weight slice obtained in the previous training round, subtract the first precision bit from its lower second precision bit and set it to 0 to obtain the second weight slice.

[0034] In one possible implementation, obtaining the second tag fragment includes:

[0035] Obtain the second label slice with second precision in the second ring from the pre-set labels of the training samples.

[0036] In one possible implementation, the second multi-party joint computation includes:

[0037] Based on the loss function, through layer-by-layer calculations performed in conjunction with other parties, the error slices of each layer with the second precision in the second ring are obtained;

[0038] Based on the error slices and activation functions of each layer, and through joint calculations with other parties, the gradient slices of the weights of each layer in the second ring with second precision are obtained.

[0039] In one possible implementation, the multi-party system includes three parties, and the weight parameter is split into three shards, with each of the three parties holding two shards that are different from each other; the first weight shard is obtained based on the two shards held by any one party.

[0040] Secondly, an apparatus for jointly training a neural network model by multiple parties is provided, the apparatus being disposed in any one of the multiple parties, comprising:

[0041] The acquisition unit is used to acquire the current weight parameters of the model and the feature data of the training samples, which are respectively the first weight slice and the first feature slice with the first precision in the first ring.

[0042] The first joint computation unit is used to input the first feature slice and the first weight slice obtained by the acquisition unit into the neural network model to be trained, and to obtain each output slice of each network layer by performing a first multi-party joint computation with other parties corresponding to the forward propagation.

[0043] The acquisition unit is further configured to acquire each output slice obtained by the first joint calculation unit, wherein the weight parameters and the labels of the training samples are respectively a second output slice, a second weight slice, and a second label slice with second precision in the second ring; wherein the second ring is larger than the first ring, and the second precision is higher than the first precision;

[0044] The second joint calculation unit is used to use the second output slice, second weight slice and second label slice obtained by the acquisition unit to perform a second multi-party joint calculation with other parties corresponding to backpropagation, so as to obtain each gradient slice with second precision in the second ring of each network layer.

[0045] The update unit is used to update the second weight slice using the gradient slices obtained by the second joint calculation unit, so as to realize the current round of training of the neural network model.

[0046] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0047] Fourthly, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0048] The method and apparatus provided in the embodiments of this specification first obtain the current weight parameters of the model and the feature data of the training samples, respectively, a first weight slice and a first feature slice with a first precision in a first ring; then, the first feature slice and the first weight slice are input into the neural network model to be trained, and each output slice of each network layer is obtained by performing a first multi-party joint calculation corresponding to forward propagation with other parties; next, each output slice is obtained, and the weight parameters and the labels of the training samples are respectively obtained as a second output slice, a second weight slice, and a second label slice with a second precision in a second ring; wherein, the second ring is larger than the first ring, and the second precision is higher than the first precision; then, using the second output slice, the second weight slice, and the second label slice, a second multi-party joint calculation corresponding to backpropagation is performed with other parties to obtain each gradient slice of each network layer with a second precision in the second ring; finally, the second weight slice is updated using each gradient slice to realize the current round of training of the neural network model. As can be seen from the above, the embodiments in this specification reduce the communication overhead required for low-precision calculations by using mixed computations of different precisions during model training, thereby reducing the overall cost of model training. Simultaneously, sufficient precision is ensured for precision-sensitive calculations, thus improving model training efficiency while maintaining the final accuracy of the model. Compared to conventional solutions, this better achieves a balance between training efficiency and model usability. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0051] Figure 2 A flowchart illustrating a method for jointly training a neural network model according to one embodiment is shown.

[0052] Figure 3This diagram illustrates a method flow chart for jointly training a neural network model according to another embodiment.

[0053] Figure 4 A schematic block diagram of an apparatus for multi-party joint training of a neural network model according to one embodiment is shown. Detailed Implementation

[0054] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0055] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification. This implementation scenario involves multi-party joint training of a neural network model. It is understood that the parties can be two, three, or four, etc.; only a three-party example is shown in the diagram. The model training is privacy-preserving machine learning, supporting the protection of the model or input data required for model training and ensuring the correctness of the output. Figure 1 As shown, the scenario of multi-party joint training of a neural network model involves participants A, B, and C, also referred to as the first party, second party, and third party, or simply A, B, and C. Each participant can be any device, platform, server, or device cluster with computing and processing capabilities. All parties must jointly train the neural network model while protecting data privacy.

[0056] Machine learning is typically based on floating-point arithmetic. However, using secret sharing to implement secure floating-point arithmetic can lead to high overhead, making it difficult to meet practical computational needs. Therefore, the common practice is to use fixed-point numbers to approximate floating-point numbers, achieving significant efficiency improvements at the cost of some precision. Secret sharing protocols are usually defined on rings or domains, each with its own advantages and disadvantages. Computation on rings is more efficient because the modulo operation can be implicitly handled by hardware, compared to computation on domains which requires manual modulo operations.

[0057] In the embodiments of this specification, fixed-point numbers can be mapped to rings for computation. The mapping construction process is as follows: Assume To be a rational number, we need to... Mapping to the integer field, let Where f is the precision in decimal places, and Int(·) is for rounding to the nearest integer. Next, regarding... Perform modulo operation to make Where k represents the number of bits in x, mapping the input to... On the ring.

[0058] Different precisions can be used for different calculations during model training, and fixed-point arithmetic is shared on rings of different sizes, i.e. These correspond to different parameters k and f, respectively. (k) can be used...m ,f m ), (k n ,f n The numbers () represent the parameter settings for low-precision and high-precision calculations, respectively, where m < n. For example, m = 32 and n = 64. It's important to understand that the values ​​of m and n are not limited to these; values ​​that satisfy m < n can be selected based on actual needs.

[0059] It should be noted that there can be any number of computation parties. The data holder can act as a computation party. The data holder and the computation party may overlap or not overlap at all. The multiple parties have equal status in the model training process. Therefore, in the description of the subsequent processing flow of the embodiments in this specification, only the processing flow of any one of the multiple parties will be described.

[0060] Furthermore, the training data is distributed among multiple parties in a secret sharing manner. The model's weight parameters, the feature data of the training samples, and the labels of the training samples are all in encrypted form, that is, in fragmented form. (See reference...) Figure 1 Party B is any one of the multiple parties, and it has the initial weighted slices with the aforementioned weight parameters. Feature fragmentation of the above feature data and the tag segments of the above tags. After training the neural network model jointly by multiple parties, the updated weight slices are obtained. m and n are used to indicate precision, with m corresponding to low precision and n corresponding to high precision. It is understandable that during model training, some plaintext data may be known to all parties; for example, the model's hyperparameters, such as the learning rate, can be plaintext.

[0061] In the embodiments of this specification, the neural network model includes several network layers. In the following examples, W, x, and y represent the weight parameters, the input of each layer (the first layer is feature data), and the label corresponding to the sample, respectively. ( ) represents the low-precision (high-precision) slice of each variable.

[0062] The following is a brief description of the common computational primitives used in the embodiments of this specification, including:

[0063] Addition and constant multiplication can be performed directly using the original addition secret sharing protocol. Each participant only needs to perform the calculation locally.

[0064] The truncation operation performs a right shift on the input, removing the last 's' bits of data, which is equivalent to dividing by 2. s ;

[0065] Multiplication, a fixed-point multiplication operation involves two key calculations. First, the participants perform integer multiplication using a standard multiplication protocol. After the multiplication is complete, due to the limited precision of fixed-point data, the result needs to be truncated. The truncation operation is used to truncate the data after doubling the precision: the last f digits are removed, i.e., divided by 2. f ;

[0066] The comparison operation uses the MSB protocol to determine whether the highest bit of the input is 0 or 1, and thus determines the sign of the input.

[0067] The fragmentation transformation operation uses the Share_Reduction protocol to move fragments of the ciphertext from a large ring to a small ring, and the Share_Extension protocol to move fragments of the ciphertext from a small ring to a large ring.

[0068] This specification provides embodiments that propose corresponding solutions to reduce the overall overhead of model training and improve model training efficiency.

[0069] Figure 2 This diagram illustrates a method for jointly training a neural network model by multiple parties according to one embodiment. The method can be based on... Figure 1 In the implementation scenario shown, the method is performed by any one of the multiple parties. For example... Figure 2 As shown, the method for multi-party joint training of a neural network model in this embodiment includes the following steps: Step 21, obtaining the current weight parameters of the model and the feature data of the training samples, respectively, a first weight slice and a first feature slice with a first precision in the first ring; Step 22, inputting the first feature slice and the first weight slice into the neural network model to be trained, and obtaining each output slice of each network layer by performing a first multi-party joint calculation corresponding to forward propagation with other parties; Step 23, obtaining each output slice, wherein the weight parameters and the labels of the training samples are respectively a second output slice, a second weight slice, and a second label slice with a second precision in the second ring; wherein the second ring is larger than the first ring, and the second precision is higher than the first precision; Step 24, using the second output slice, the second weight slice, and the second label slice, performing a second multi-party joint calculation corresponding to backpropagation with other parties to obtain each gradient slice of each network layer with a second precision in the second ring; Step 25, updating the second weight slice using each gradient slice to achieve the current round of training of the neural network model. The specific execution method of each of the above steps is described below.

[0070] First, in step 21, the current weight parameters of the model and the feature data of the training samples are obtained, specifically the first weight slice and the first feature slice with the first precision in the first ring. It is understood that model training requires multiple iterations, i.e., multiple training rounds, and the weight parameters are typically updated in each iteration. The current weight parameters are the weight parameters at the beginning of this iteration. If this is the first training round, the current weight parameters are the initialized weight parameters; if this is not the first training round, the current weight parameters are the weight parameters obtained in the previous training round.

[0071] In one example, the current training round is the first training round; the acquisition of the first weight slice includes:

[0072] Obtain the initial weight slices of the second precision in the second ring, based on the pre-set weight parameters of the model;

[0073] By performing a right-shifting operation with other parties, subtracting the first precision bit from the second precision bit, the initial weight slice is converted into a transformed weight slice with the first precision in the second ring.

[0074] By performing a modular conversion operation in conjunction with other parties, the conversion weight slice is converted into the first weight slice.

[0075] For example, the initial weighted sharding is High-precision initial weights are partitioned. Convert to low-precision first-weighted sharding The calculation can be expressed as:

[0076]

[0077] First, the Truncation protocol is invoked to shift the input to the right by f. n -f m The bits are used to obtain a precision of f. m Output Because the input and output precisions are different, superscripts are used here to distinguish them. And W. Furthermore, the output at this time... Still in Sharing on the ring requires calling the Share_Reduction protocol to convert the truncated output to the smaller ring. The above two steps combined complete the low-precision conversion of weights.

[0078] In one example, the current training round is not the first training round; the acquisition of the first weight slice includes:

[0079] For the second weight slice obtained in the previous training, by performing a right shift operation of the second precision minus the first precision bit in conjunction with other parties, the second weight slice is converted into a transformed weight slice of the first precision in the second ring.

[0080] By performing a modular conversion operation in conjunction with other parties, the conversion weight slice is converted into the first weight slice.

[0081] In this example, a low-precision transformation of the weights is performed on the second weight slice obtained from the previous training round. The specific transformation process is similar to that of the low-precision transformation of the weights on the initial weight slice, and will not be described in detail here.

[0082] In one example, the multi-party system includes three parties, and the weight parameter is split into three shards, with each of the three parties holding two different shards from the other; the first weight shard is obtained based on the two shards held by any one party.

[0083] For example, if either party has two initial weight slices with second precision in the second ring, then the processing performed by that party for the low-precision transformation of the weights includes processing performed on the two initial weight slices separately.

[0084] Furthermore, in the embodiments of this specification, the first feature slice can be an initialized feature slice. In other words, the feature data is initialized with low precision.

[0085] Then, in step 22, the first feature slice and the first weight slice are input into the neural network model to be trained. Through a first multi-party joint computation corresponding to forward propagation with other parties, the output slices of each network layer are obtained. It can be understood that forward propagation is to calculate and store the output results of each network layer in sequence, using the output of the previous layer as the input of the next layer, and calculating the output of the next layer, until the output layer is reached.

[0086] For example, segment the first feature and first weighted shard As input, it is calculated layer by layer until the last layer of the model, i.e. Where func is the defined activation function. Corresponding to the weight parameters of the current layer, Corresponding to the input of the current layer, This corresponds to the output of the current layer.

[0087] In the embodiments described in this specification, the input for forward propagation is low-precision data, mainly involving low-precision calculations. Special attention is needed because some calculations in the forward propagation are precision-sensitive, such as batch normalization, which involves exponential functions, division, or square root inversion. To ensure the precision of the output results, for these precision-sensitive calculations, the input can be explicitly converted to high-precision values ​​briefly. After the calculation is completed, the output is then converted back to low-precision values ​​as input for subsequent forward propagation.

[0088] In one example, the network layers include a first intermediate layer, a second intermediate layer, and a third intermediate layer arranged in sequence, the first and third intermediate layers corresponding to a first precision calculation, the second intermediate layer corresponding to a second precision calculation, and the first multi-party joint calculation including:

[0089] The first output fragment of the first precision in the first ring of the first intermediate layer is converted into the second output fragment of the second precision in the second ring through a modulus conversion operation performed in conjunction with other parties;

[0090] The second output fragment corresponding to the first intermediate layer is input into the second intermediate layer, and the second output fragment of the second intermediate layer is obtained through joint calculation with other parties;

[0091] The second output fragment of the second intermediate layer is converted into the first output fragment of the first precision in the first ring through a modulus conversion operation performed jointly with other parties, and the first output fragment is used as the input of the third intermediate layer.

[0092] In this example, the first intermediate layer, the second intermediate layer, and the third intermediate layer are three network layers arranged in sequence. The first intermediate layer can also be an input layer, and the third intermediate layer can also be an output layer.

[0093] Furthermore, the first intermediate layer and the third intermediate layer each include linear operations and activation function operations, and the second intermediate layer is used to perform batch normalization operations.

[0094] Next, in step 23, each output slice is obtained. The weight parameters and the labels of the training samples are respectively a second output slice, a second weight slice, and a second label slice with second precision in the second ring; wherein, the second ring is larger than the first ring, and the second precision is higher than the first precision. It can be understood that the above-mentioned output slices are obtained through forward propagation, and may include a second output slice with second precision in the second ring, or a first output slice with first precision in the first ring. For the first output slice, its corresponding second output slice can be obtained through precision conversion.

[0095] In one example, each output slice includes a first output slice of first precision in a first ring, output by a certain network layer in each network layer;

[0096] The acquisition of the second output fragment includes:

[0097] Obtain the first output fragment;

[0098] For the first output slice, through a modulus conversion operation performed jointly with other parties, the first output slice is converted into a first-precision converted output slice in the second ring;

[0099] By performing a left-shift operation with other parties, subtracting the first precision bit from the second precision bit, the converted output fragment is converted into a second output fragment with the second precision in the second ring.

[0100] In this example, the output of each layer during forward propagation is converted from low precision to high precision. For example, the first output slice is... Split the first output Convert to second output fragment The calculation can be expressed as:

[0101]

[0102] First, the Share_Extension protocol is invoked to place x in the small ring. Sharing on the platform is converted into sharing on the big circle. Next, multiply the converted output by... Make the precision of the output value 1 It should be noted that although the precision is higher, unlike right shift which may involve truncation, this is a left shift operation, so the actual value will not change. Converting the input to a larger ring first avoids overflow during the left shift.

[0103] In one example, the current training round is the first training round; the acquisition of the second weight slice includes:

[0104] Obtain the initial weight slices of the second precision in the second ring, based on the pre-set weight parameters of the model;

[0105] For the initial weighted slice, the second precision bit is subtracted from the first precision bit and set to 0 to obtain the second weighted slice.

[0106] In this example, the weight parameters used in the forward propagation calculation are converted from low precision to high precision. For example, the desired second weight slice is... The Share_Extension protocol can be invoked to... Convert to The embodiments in this specification have been optimized so that the second weight slice can be obtained only through local computation. For the first round of training, a low-precision first weight slice was observed. It can be partitioned by high-precision initial weights. The calculation shows that sharding can be directly performed based on the initial weights. Obtain the second weighted partition Specifically, according to the formula:

[0107]

[0108] Each party will hold The low bit of the fragment (f) n -f m Set the (nf) bit to 0, that is, only retain the high-order bits (nf) n +f m (bits), the plaintext value restored from these fragments and the target value The possible maximum error is on the order of magnitude of The impact on actual model training is negligible.

[0109] In one example, the current training round is not the first training round; the acquisition of the second weight slice includes:

[0110] For the second weight slice obtained in the previous training round, subtract the first precision bit from its lower second precision bit and set it to 0 to obtain the second weight slice.

[0111] In this example, the weight parameters used in the forward propagation calculation are converted from low precision to high precision. For non-first rounds of training, the second weight slice obtained from the previous round of training is used. Obtain the second weighted partition The specific transformation process is similar to the processing performed on the initial weighted shards, and will not be elaborated here.

[0112] In one example, obtaining the second tag fragment includes:

[0113] Obtain the second label slice with second precision in the second ring from the pre-set labels of the training samples.

[0114] In this example, the second tag fragment can be the initialized tag fragment. In other words, the tag fragments are initialized to high precision.

[0115] In step 24, the second output slice, the second weight slice, and the second label slice are used to perform a second multi-party joint calculation corresponding to backpropagation, obtaining the gradient slices of each network layer with second precision in the second ring. Backpropagation is the method for calculating the gradients of neural network parameters. This method, based on the chain rule in calculus, traverses the network from the output layer to the input layer in the reverse order of forward propagation.

[0116] In one example, the second multi-party joint computation includes:

[0117] Based on the loss function, through layer-by-layer calculations performed in conjunction with other parties, the error slices of each layer with the second precision in the second ring are obtained;

[0118] Based on the error slices and activation functions of each layer, and through joint calculations with other parties, the gradient slices of the weights of each layer in the second ring with second precision are obtained.

[0119] For example, the second output slice is The second weighted sharding is The second tag segment is The overall calculation process is consistent with the backpropagation calculation of traditional deep learning models. First, the error of the last layer is obtained based on the loss function: Understandable, The second output fragment corresponding to the output of the last layer is then calculated; next, the error of the previous layer is calculated one by one: Calculate the gradient of the weight parameters for each layer based on the error and activation function output of each layer:

[0120] Finally, in step 25, the second weight slice is updated using each gradient slice to complete the current round of training of the neural network model. It is understood that stochastic gradient descent can be used to update the second weight slice.

[0121] Among them, the stochastic gradient descent method, also known as incremental gradient descent, is an iterative method for optimizing differentiable objective functions, and is a stochastic approximation of gradient descent optimization.

[0122] In the embodiments described in this specification, updating the gradients of each layer to the sovereign weight to complete one round of training iteration can be represented as follows:

[0123] The embodiments in this specification use fixed-point computation primitives based on secret sharing protocols to achieve secure mixed-precision training of the model, which is used to accelerate model training in deep learning. This is achieved by using floating-point numbers of different lengths for forward and backward computations in model training, thereby speeding up local computation, reducing communication overhead, and thus accelerating model training.

[0124] Figure 3 A schematic flowchart illustrating a method for jointly training a neural network model according to another embodiment is shown. (Refer to...) Figure 3 The method mainly includes four stages: reduction, forward propagation, extension, and backpropagation. Before training begins, the model's weights are initialized to high-precision values ​​to ensure model convergence. These high-precision values ​​are called sovereign weights. For example, Figure 3 In This refers to the partitioning of the weight parameters for initialization. The feature data of the training samples is initialized with low-precision values ​​to be used as input for forward propagation, for example... Figure 3 In This involves partitioning the feature data; initializing the labels of the training samples with high-precision values, which are then used as input for backpropagation. Figure 3 In This refers to the segmentation of labels. Neural network models typically consist of multiple network layers, such as... Figure 3 In this context, "Linear" represents a linear processing network layer, "Act" represents a network layer with an activation function, and "Batch Norm" represents a network layer corresponding to batch normalization. Since most of the computation in forward propagation is low-precision, it is necessary to perform low-precision weight transformation through piecewise reduction. That is, the high-precision model weights are converted into low-precision model weights as input for forward propagation. For example, in the piecewise reduction stage... Figure 3 In Converted to Converted to

[0125] During the forward propagation phase, low-precision feature data and weight parameters are used as input, and computation is performed layer by layer until the final output layer of the model. For certain computations requiring high precision, such as batch normalization, the input is converted to high precision, the computation is completed, and then the result is converted back to low-precision output as the input for subsequent layers. For example, Figure 3 In It was converted through fragmentation expansion. Using this as input to Batch Norm., the output of Batch Norm. is... The reduction was achieved through fragmentation. Will As input to subsequent network layers.

[0126] In the slice expansion stage, high-precision transformations are implemented for the output of the activation function and the weight parameters. After the forward computation is complete, the outputs of the intermediate activation function layers and the low-precision weights used in the forward computation are converted into high-precision values, which are then used as inputs for backpropagation. For example, Figure 3 In It was converted through fragmentation expansion. It was converted through fragmentation expansion.

[0127] During the backpropagation phase, the main responsibility is to calculate the error and gradient of each layer in the reverse order of forward propagation, and update the gradients to the high-precision principal weights, completing one round of model training iteration. For example, Figure 3 In Represents the error of each layer. This represents the gradient of each layer.

[0128] The method provided in the embodiments of this specification first obtains the current weight parameters of the model and the feature data of the training samples, respectively, a first weight slice and a first feature slice with a first precision in a first ring; then, the first feature slice and the first weight slice are input into the neural network model to be trained, and each output slice of each network layer is obtained by performing a first multi-party joint calculation corresponding to forward propagation with other parties; next, each output slice is obtained, and the weight parameters and the labels of the training samples are respectively obtained as a second output slice, a second weight slice, and a second label slice with a second precision in a second ring; wherein, the second ring is larger than the first ring, and the second precision is higher than the first precision; then, using the second output slice, the second weight slice, and the second label slice, a second multi-party joint calculation corresponding to backpropagation is performed with other parties to obtain each gradient slice of each network layer with a second precision in the second ring; finally, the second weight slice is updated using each gradient slice to realize the current round of training of the neural network model. As can be seen from the above, the embodiments in this specification reduce the communication overhead required for low-precision calculations by using mixed computations of different precisions during model training, thereby reducing the overall cost of model training. Simultaneously, sufficient precision is ensured for precision-sensitive calculations, thus improving model training efficiency while maintaining the final accuracy of the model. Compared to conventional solutions, this better achieves a balance between training efficiency and model usability.

[0129] According to another embodiment, an apparatus for jointly training a neural network model by multiple parties is also provided, wherein the apparatus is disposed at any of the multiple parties and is used to perform the present specification. Figure 2 The method provided in the illustrated embodiment. Figure 4 A schematic block diagram of an apparatus for jointly training a neural network model according to one embodiment is shown. Figure 4As shown, the device 400 includes:

[0130] The acquisition unit 41 is used to acquire the current weight parameters of the model and the feature data of the training samples, which respectively have a first weight slice and a first feature slice with a first precision in the first ring.

[0131] The first joint calculation unit 42 is used to input the first feature slice and the first weight slice obtained by the acquisition unit 41 into the neural network model to be trained, and obtain each output slice of each network layer by performing a first multi-party joint calculation with other parties corresponding to the forward propagation.

[0132] The acquisition unit 41 is further configured to acquire each output slice obtained by the first joint calculation unit 42, wherein the weight parameters and the labels of the training samples are respectively a second output slice, a second weight slice, and a second label slice with second precision in the second ring; wherein the second ring is larger than the first ring, and the second precision is higher than the first precision;

[0133] The second joint calculation unit 43 is used to use the second output slice, the second weight slice, and the second label slice obtained by the acquisition unit 41 to perform a second multi-party joint calculation with other parties corresponding to backpropagation, so as to obtain each gradient slice with second precision in the second ring of each network layer.

[0134] The update unit 44 is used to update the second weight slice using the gradient slices obtained by the second joint calculation unit 43, so as to realize the current round of training of the neural network model.

[0135] Optionally, as an embodiment, this round of training is the first round of training; the acquisition unit 41 includes:

[0136] Get sub-units to obtain the initial weight slices of the second precision in the second ring for the pre-set weight parameters of the model;

[0137] The right shift subunit is used to perform a right shift operation by subtracting the first precision bit from the second precision bit in conjunction with other parties, so as to convert the initial weight slice obtained by the acquisition subunit into the first precision transformed weight slice in the second ring.

[0138] The transformation subunit is used to convert the transformation weight slice obtained by the right shift subunit into the first weight slice by performing a modulo transformation operation in conjunction with other parties.

[0139] Optionally, as an embodiment, this round of training is not the first round of training; the acquisition unit 41 includes:

[0140] The right shift subunit is used to transform the second weight slice obtained in the previous training round into a transformed weight slice of the first precision in the second ring by performing a right shift operation with other parties, subtracting the first precision bit from the second precision.

[0141] The transformation subunit is used to convert the transformation weight slice obtained by the right shift subunit into the first weight slice by performing a modulo transformation operation in conjunction with other parties.

[0142] Optionally, as an embodiment, each network layer includes a first intermediate layer, a second intermediate layer, and a third intermediate layer arranged in sequence, the first and third intermediate layers corresponding to calculations of a first precision, the second intermediate layer corresponding to calculations of a second precision, and the first joint calculation unit 42 including:

[0143] The first conversion subunit is used to convert the first output segment of the first precision in the first ring of the first intermediate layer output into the second output segment of the second precision in the second ring through a modulus conversion operation performed in conjunction with other parties.

[0144] The processing subunit is used to input the second output slice corresponding to the first intermediate layer obtained by the first conversion subunit into the second intermediate layer, and obtain the second output slice of the second intermediate layer through joint operations with other parties;

[0145] The second conversion subunit is used to convert the second output fragment of the second intermediate layer output obtained by the processing subunit into the first output fragment of the first precision in the first ring through a modulus conversion operation performed jointly with other parties, and use the first output fragment as the input of the third intermediate layer.

[0146] Furthermore, the first intermediate layer and the third intermediate layer each include linear operations and activation function operations, and the second intermediate layer is used to perform batch normalization operations.

[0147] Optionally, as an embodiment, each output slice includes a first output slice of first precision in the first ring, output by a certain network layer in each network layer;

[0148] The acquisition unit 41 includes:

[0149] A subunit is used to acquire the first output slice;

[0150] A conversion subunit is used to convert the first output slice obtained by the acquisition subunit into a first-precision converted output slice in the second ring through a modulus conversion operation performed jointly with other parties.

[0151] The shift subunit is used to convert the conversion output fragment obtained by the conversion subunit into a second output fragment of the second precision in the second ring by performing a left shift operation of the second precision minus the first precision bits in conjunction with other parties.

[0152] Optionally, as an embodiment, this round of training is the first round of training; the acquisition unit 41 includes:

[0153] Get sub-units to obtain the initial weight slices of the second precision in the second ring for the pre-set weight parameters of the model;

[0154] The shift subunit is used to subtract the first precision bit from the lower-order second precision bit and set it to 0 for the initial weight slice obtained by the acquisition subunit, so as to obtain the second weight slice.

[0155] Optionally, as an embodiment, the current training round is not the first training round; the acquisition unit 41 is specifically used to subtract the first precision bit from the lower second precision bit of the second weight slice obtained in the previous training round and set it to 0 to obtain the second weight slice.

[0156] Optionally, as an embodiment, the acquisition unit 41 is specifically used to acquire a second label slice of a pre-set training sample with a second precision in the second ring.

[0157] Optionally, as an embodiment, the second joint computing unit 43 includes:

[0158] The error calculation subunit is used to obtain the second-precision error slice of each layer in the second ring by performing layer-by-layer calculations in conjunction with other parties based on the loss function.

[0159] The gradient calculation subunit is used to obtain the second-precision gradient slice of the weights of each layer in the second ring by calculation in conjunction with other parties, based on the error slice and activation function of each layer obtained by the error calculation subunit.

[0160] Optionally, as an embodiment, the multiple parties include three parties, and the weight parameter is split into three shards, with each of the three parties holding two shards that are different from each other; the first weight shard is obtained based on the two shards held by any one party.

[0161] Using the apparatus provided in the embodiments of this specification, firstly, the acquisition unit 41 acquires the current weight parameters of the model and the feature data of the training samples, respectively, a first weight slice and a first feature slice with a first precision in the first ring; then, the first joint calculation unit 42 inputs the first feature slice and the first weight slice into the neural network model to be trained, and obtains each output slice of each network layer by performing a first multi-party joint calculation corresponding to forward propagation with other parties; next, the acquisition unit 41 acquires each output slice, and the weight parameters and the labels of the training samples respectively, a second output slice, a second weight slice, and a second label slice with a second precision in the second ring; wherein, the second ring is larger than the first ring, and the second precision is higher than the first precision; the second joint calculation unit 43 then uses the second output slice, the second weight slice, and the second label slice to perform a second multi-party joint calculation corresponding to backpropagation with other parties to obtain each gradient slice of each network layer with a second precision in the second ring; finally, the update unit 44 uses each gradient slice to update the second weight slice to realize the current round of training of the neural network model. As can be seen from the above, the embodiments in this specification reduce the communication overhead required for low-precision calculations by using mixed computations of different precisions during model training, thereby reducing the overall cost of model training. Simultaneously, sufficient precision is ensured for precision-sensitive calculations, thus improving model training efficiency while maintaining the final accuracy of the model. Compared to conventional solutions, this better achieves a balance between training efficiency and model usability.

[0162] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.

[0163] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The method described.

[0164] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0165] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for jointly training a neural network model in a multi-party secure computation system, the method being performed by any one of the parties, comprising: Obtain the current weight parameters of the model and the feature data of the training samples, respectively, in the first ring, the first weight slice and the first feature slice with the first precision; The first feature slice and the first weight slice are input into the neural network model to be trained, and each output slice of each network layer is obtained by performing a first multi-party joint calculation with other parties corresponding to the forward propagation. Each output slice is obtained, and the weight parameters and the labels of the training samples are respectively in the second ring with the second output slice, the second weight slice, and the second label slice having the second precision; wherein, the second ring is larger than the first ring, and the second precision is higher than the first precision; Using the second output slice, the second weight slice, and the second label slice, a second multi-party joint computation corresponding to backpropagation is performed with other parties to obtain the gradient slices of each network layer with the second precision in the second ring. The second weight slice is updated using each gradient slice to achieve the current round of training of the neural network model.

2. The method as described in claim 1, wherein, This round of training is the first round of training; The acquisition of the first weighted slice includes: Obtain the initial weight slices of the second precision in the second ring, based on the pre-set weight parameters of the model; By performing a right-shift operation with other parties, the initial weight slice is transformed into a transformed weight slice of the first precision in the second ring; the first bit is the result of subtracting the first precision from the second precision. By performing a modular conversion operation in conjunction with other parties, the conversion weight slice is converted into the first weight slice.

3. The method as described in claim 1, wherein, This round of training is not the first round of training; the acquisition of the first weight slice includes: For the second weight slice obtained in the previous training round, by performing a right shift operation with other parties, the second weight slice is transformed into a transformed weight slice with the first precision in the second ring; the first bit is the result of the second precision minus the first precision; By performing a modular conversion operation in conjunction with other parties, the conversion weight slice is converted into the first weight slice.

4. The method of claim 1, wherein, Each network layer includes a first intermediate layer, a second intermediate layer, and a third intermediate layer arranged in sequence. The first and third intermediate layers correspond to calculations of first precision, and the second intermediate layer corresponds to calculations of second precision. The first multi-party joint calculation includes: The first output fragment of the first precision in the first ring of the first intermediate layer is converted into the second output fragment of the second precision in the second ring through a modulus conversion operation performed in conjunction with other parties; The second output fragment corresponding to the first intermediate layer is input into the second intermediate layer, and the second output fragment of the second intermediate layer is obtained through joint calculation with other parties; The second output fragment of the second intermediate layer is converted into the first output fragment of the first precision in the first ring through a modulus conversion operation performed jointly with other parties, and the first output fragment is used as the input of the third intermediate layer.

5. The method of claim 4, wherein, The first and third intermediate layers each contain linear operations and activation function operations, while the second intermediate layer is used for batch normalization operations.

6. The method of claim 1, wherein, Each output slice includes a first output slice of first precision in the first ring, output by a certain network layer in each network layer. The acquisition of the second output fragment includes: Obtain the first output fragment; For the first output slice, through a modulus conversion operation performed jointly with other parties, the first output slice is converted into a first-precision converted output slice in the second ring; By performing a left-shift operation with other parties, the transformed output fragment is converted into a second output fragment with a second precision in the second ring; the first bit is the result of subtracting the first precision from the second precision.

7. The method of claim 1, wherein, This round of training is the first round of training; The acquisition of the second weighted slice includes: Obtain the initial weight slices of the second precision in the second ring, based on the pre-set weight parameters of the model; For the initial weighted slice, set its first low-order bit to 0 to obtain the second weighted slice; the first bit is the result of the second precision minus the first precision.

8. The method of claim 1, wherein, This round of training is not the first round of training; the acquisition of the second weight slice includes: For the second weight slice obtained in the previous training round, set its first low-order bit to 0 to obtain the second weight slice; the first bit is the result of the second precision minus the first precision.

9. The method of claim 1, wherein, The acquisition of the second tag fragment includes: Obtain the second label slice with second precision in the second ring from the pre-set labels of the training samples.

10. The method of claim 1, wherein, The second multi-party joint calculation includes: Based on the loss function, through layer-by-layer calculations performed in conjunction with other parties, the error slices of each layer with the second precision in the second ring are obtained; Based on the error slices and activation functions of each layer, and through joint calculations with other parties, the gradient slices of the weights of each layer in the second ring with second precision are obtained.

11. The method of claim 1, wherein, The multi-party system includes three parties, and the weight parameter is split into three shards. Each of the three parties holds two shards that are different from each other in the three shards. The first weighted shard is obtained based on the two shards held by either party.

12. An apparatus for jointly training a neural network model in a multi-party secure computation system, the apparatus being disposed in any of the parties, comprising: The acquisition unit is used to acquire the current weight parameters of the model and the feature data of the training samples, which are respectively the first weight slice and the first feature slice with the first precision in the first ring. The first joint computation unit is used to input the first feature slice and the first weight slice obtained by the acquisition unit into the neural network model to be trained, and to obtain each output slice of each network layer by performing a first multi-party joint computation with other parties corresponding to the forward propagation. The acquisition unit is further configured to acquire each output slice obtained by the first joint calculation unit, wherein the weight parameters and the labels of the training samples are respectively a second output slice, a second weight slice, and a second label slice with second precision in the second ring; wherein the second ring is larger than the first ring, and the second precision is higher than the first precision; The second joint calculation unit is used to use the second output slice, second weight slice and second label slice obtained by the acquisition unit to perform a second multi-party joint calculation with other parties corresponding to backpropagation, so as to obtain each gradient slice with second precision in the second ring of each network layer. The update unit is used to update the second weight slice using the gradient slices obtained by the second joint calculation unit, so as to realize the current round of training of the neural network model.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-11.

14. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-11.

Images