A model training method, device, storage medium, and equipment

Abstract

The specification discloses a model training method, device, storage medium and equipment, which are used for training encrypted training samples, the encrypted training samples are assigned to several computing parties, and each computing node in each computing party is connected to form a ring communication structure. For any computing node in any computing party: the computing node obtains a node gradient according to a to-be-trained model deployed in advance and a data shard assigned to the computing node itself, synchronizes the node gradient to other computing nodes in the computing party through the ring communication structure, and after obtaining the node gradients of all other computing nodes in the computing party, the computing node determines a computing party gradient of the computing party. The computing party gradient is synchronized to other computing parties, so that each computing party updates a gradient after receiving the computing party gradient synchronized by other computing parties, and trains the to-be-trained model according to the updated gradient, which can increase the efficiency of communication in model training.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and device for model training. Background Technology

[0002] With the development of information technology, data collection and processing are no longer often handled by the same entity. To maximize the value of their data, data holders often choose to collaborate with other holders and entrust their data to professional processing organizations. In this process, any party with security risks could potentially cause data leakage, resulting in significant losses for the data holders. Therefore, the theory of secure multi-party computation has been proposed.

[0003] Secure multi-party computation employs various encryption methods to protect the privacy of each participant's data from being known by others while ensuring the smooth operation of collaborative computation. However, due to the use of encryption, the amount of data generated by each party during model training using a secure multi-party computation architecture is far greater than in conventional computation, placing significant pressure on communication between the computation parties and impacting the efficiency of model training.

[0004] To address the above problems, the present invention provides a method, apparatus, storage medium, and device for model training. Summary of the Invention

[0005] This specification provides a method and apparatus for model training to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This manual provides a method for model training, including:

[0008] For each training sample, multiple data slices from that training sample are assigned to several computation parties, and the multiple data slices are obtained by encrypting the training sample.

[0009] For each computing party, which includes several computing nodes, the data shards allocated to the computing party are distributed to each computing node in the computing party, and the data shards allocated to different computing nodes are not exactly the same;

[0010] The computing nodes in this computing unit are connected to form a ring communication structure. In this ring structure, for any computing node, the computing node has two adjacent computing nodes. The computing node receives information from one of the adjacent computing nodes and sends information to the other adjacent computing node.

[0011] For any computing node in any computing party, the method includes:

[0012] The computing node obtains its gradient based on the pre-deployed model to be trained and the data shards allocated to the computing node itself.

[0013] The node gradient is synchronized to other computing nodes in the computing process through the ring communication structure.

[0014] After obtaining the node gradients of all other computing nodes in the computing party, the computing node determines the computing party gradient based on its own node gradient and the node gradients of all other computing nodes in the computing party. The computing party gradient is used to synchronize with other computing parties, so that each computing party, after receiving the synchronized computing party gradients from other computing parties, merges the received computing party gradients with its own computing party gradients to obtain an updated gradient, and trains the model to be trained based on the updated gradient.

[0015] Optionally, the multiple data slices from the training sample specifically include: a number of data slices divided from the training sample, equal to the number of computational methods, wherein the sum of the values ​​of the divided data slices is equal to the value corresponding to the undivided training sample.

[0016] Optionally, the data shards allocated to the computing party specifically include: a data shard group consisting of at most n-1 of the n data shards derived from the training sample shards, and the data shard group of the computing party is not exactly the same as the data shard group of other computing parties.

[0017] Optionally, synchronizing the node gradient with other computing nodes in the computation party through the ring communication structure specifically includes:

[0018] In the ring communication structure, the computing node that receives the computing node information is designated as the first communication node, and the computing node that sends information to the computing node is designated as the second communication node.

[0019] The node gradient is synchronized to the first communication node through the ring communication structure;

[0020] The node gradient synchronized by the second communication node is fused with its own node gradient;

[0021] The merged node gradient is then synchronized to the first communication node.

[0022] Continue to update its own node gradient based on the node gradient synchronized by the second communication node, until the node gradients of all other computing nodes in the computing process are merged.

[0023] Optionally, the node gradient is a gradient matrix;

[0024] The process of synchronizing the node gradient to the first communication node through the ring communication structure specifically includes:

[0025] Based on the number of computing nodes within the computation square, the gradient matrix is ​​divided into several gradient sub-matrices, which are equal in number to the number of computing nodes within the computation square.

[0026] The ring communication structure is used to synchronize the gradient submatrix of the computing node to the first communication node.

[0027] Optionally, synchronizing the node gradient with other computing nodes in the computation party through the ring communication structure specifically includes:

[0028] Based on the pre-defined correspondence between each computation node and each position in the gradient matrix, the gradient submatrix corresponding to the position of the computation node is taken as the submatrix to be synchronized.

[0029] The sub-matrix to be synchronized is synchronized to the first communication node;

[0030] Based on the position corresponding to the received sub-matrix synchronized by the second communication node, the received sub-matrix synchronized by the second communication node is fused with the gradient sub-matrix at the same position of the computing node itself.

[0031] The newly fused node gradient submatrix is ​​resynchronized to the first communication node as the submatrix to be synchronized, and the received submatrix synchronized by the second communication node is fused with the gradient submatrix at the same position of the computing node until the computing gradient matrix of the computing square is obtained by fusing all the gradient submatrixes of all computing nodes in the computing square.

[0032] Optionally, the gradient of the computational party is specifically used to be synchronized by the computational party to a designated other computational party, wherein the designated other computational party is another computational party that has at least one of the same data shards as the computational party.

[0033] This specification provides a model training apparatus, comprising:

[0034] For each training sample, multiple data slices from that training sample are assigned to several computation parties, and the multiple data slices are obtained by encrypting the training sample.

[0035] For each computing party, which includes several computing nodes, the data shards allocated to the computing party are distributed to each computing node in the computing party, and the data shards allocated to different computing nodes are not exactly the same;

[0036] The computing nodes in this computing unit are connected to form a ring communication structure. In this ring structure, for any computing node, the computing node has two adjacent computing nodes. The computing node receives information from one of the adjacent computing nodes and sends information to the other adjacent computing node.

[0037] For any computing node in any computing party, the apparatus includes:

[0038] The computation module obtains node gradients based on the pre-deployed model to be trained and the data slices allocated to the device itself.

[0039] The synchronization module synchronizes the node gradient with other computing nodes in the computing party through the ring communication structure.

[0040] The fusion module, after obtaining the node gradients of all other computing nodes in the computing party, determines the computing party gradient based on the node gradient obtained by the device itself and the node gradients of all other computing nodes in the computing party. The computing party gradient is used to synchronize with other computing parties, so that each computing party, after receiving the synchronized computing party gradients from other computing parties, merges the received computing party gradients with its own computing party gradients to obtain an updated gradient, and trains the model to be trained based on the updated gradient.

[0041] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for training the model.

[0042] This specification provides an apparatus including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for training the model described above.

[0043] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0044] In the model training method provided in this specification, for each training sample, multiple data slices from that training sample are allocated to several computation parties. These multiple data slices are obtained by encrypting the training sample. For each computation party, the computation party includes several computation nodes. The data slices allocated to the computation party are distributed among the computation nodes within that computation party. Different computation nodes are not assigned identical data slices. The computation nodes within the computation party are connected to form a ring communication structure. In this ring structure, for any computation node, the computation node has two adjacent computation nodes. The computation node receives information from one of its adjacent computation nodes and sends information to the other adjacent computation node. For any computation node within any computation party, the method includes:

[0045] The computing node obtains its node gradient based on the pre-deployed model to be trained and the data shards allocated to it. This node gradient is then synchronized to other computing nodes within the computing party via the ring communication structure. After obtaining the node gradients of all other computing nodes in the computing party, the computing node determines its own computing party gradient based on its own gradient and the gradients of all other computing nodes. This computing party gradient is then synchronized to other computing parties. Upon receiving the synchronized gradients from other computing parties, each computing party merges the received gradients with its own to obtain an updated gradient. Based on this updated gradient, the model to be trained is then trained. This ring communication structure evenly distributes the communication load across each computing node in the computing party, fully utilizing the ingress and egress bandwidth of each computing node, thereby increasing overall communication efficiency and reducing communication costs in secure multi-party computation. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0047] Figure 1 This is a schematic diagram of the deployment structure of each computing node in any computing method described in this specification;

[0048] Figure 2 This is a flowchart illustrating a model training method provided in this specification.

[0049] Figure 3 A schematic diagram of a model training device provided in this specification;

[0050] Figure 4 The corresponding information provided in this specification Figure 2 A schematic diagram of an electronic device. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0052] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0053] In the course of business operations, enterprises can acquire a large amount of user data, including personal preferences, information, and behaviors, through interaction with users. Enterprises often hope to analyze this user data to support their decision-making and business adjustments, and machine learning is a common method for this data analysis. However, because the user data acquired by enterprises in their operations is typically large in volume and multi-dimensional, applying machine learning to analyze this data places extremely high demands on the computing power of computing devices. When the data holder lacks the ability to analyze their own user data, they must delegate the data analysis task to other computing providers with sufficient computing power. Furthermore, because this user data contains sensitive information such as personal information, personal preferences, and even financial data (user information collected by institutions such as banks) and medical data (user information collected by institutions such as hospitals), its leakage would pose significant risks and losses to both the individual user and the data holder. Therefore, these data holders desire to utilize this user data through secure multi-party computation (MPC) architecture for model training. In MPC, multiple computing parties jointly compute on a batch of data; even if a party leaks data, its number cannot threaten the overall user data if it does not reach the encryption threshold. However, when training models using MPC, to ensure the homomorphism of the encrypted training models across all computing parties, the training data and models need to be pre-configured. During training with the configured model and training data, the amount of data generated by each computing party is far greater than in conventional computation, placing immense pressure on communication between the computing parties and impacting model training efficiency. Therefore, this specification provides a model training method.

[0054] The method provided in this specification is applied to training on the user data containing sensitive information mentioned above (i.e., the training samples in the model training process are user data containing sensitive information). Before the training process begins, the holder of the user data pre-encrypts the training sample into multiple data fragments. In one or more specific embodiments of this specification, the specific method for encrypting the training sample is to divide the training sample X into n data fragments X1, X2, ..., Xn with equal computational power. n This ensures that the sum of the values ​​of each data slice is equal to the value of the corresponding unsliced ​​training sample, meaning that the data dimension and data volume of each data slice are the same as the training sample before the slice, and the values ​​of the same variable for the same sample in different data slices satisfy X1 + X2 + ... + X n =X.

[0055] Subsequently, the segmented data fragments are allocated to each computational party, ensuring that each of the n data fragments is held by at least one computational party. Therefore, each computational party does not hold complete sample data, and none of them can reconstruct the sample data from their allocated data fragments. Furthermore, if the total number of different data fragments held by multiple computational parties attempting to reconstruct the sample data is less than n, they also cannot reconstruct the sample data. In one or more embodiments of this specification, these n data fragments can be pre-grouped into n data fragment groups, such that each data fragment group contains at most n-1 data fragments, and the data fragments contained in each data fragment group are not completely identical. These data fragment groups are then allocated to each computational party. Specifically, when there are three computational parties participating in model training, the three segmented data fragments can be divided into three data fragment groups, which are then allocated to the three computational parties. Thus, each computational party holds two data fragments, and none of them can reconstruct the sample data independently.

[0056] After determining the data shards to be allocated to each computing party, the parameters of the training model deployed by that computing party can be adjusted based on the allocated data shards. This allows the training models deployed by each computing party to complete computations based on the encrypted data shards and to use the computation results from each computing party for gradient synchronization. Specifically, a Trusted-Environment-based Cryptographic Computing (TECC) engine based on the Aby3 protocol can be used to enable the training models deployed by each computing party to perform common operator operations (addition, multiplication, etc.), and to update the training models deployed by each computing party through the synchronization of computation results. The training models to be adjusted can be adapted to the data analysis needs (prediction, classification, etc.) of the data holder and can select from various common machine learning models.

[0057] After the data holder allocates data shards to each computing party and deploys the corresponding training model for each data shard, for any given computing party, to improve model training efficiency, multiple computing nodes can be deployed to jointly complete the training task. Specifically, the training party can divide its allocated data shards into several segments equal to the number of computing nodes (the combination of these data shard segments constitutes the computing party's complete data shard) and distribute them to each computing node. Simultaneously, the adjusted training model deployed by the computing party is fully deployed on each training node, enabling data parallelism among the computing nodes during subsequent model computation.

[0058] Figure 1This diagram illustrates the deployment structure of each computing node in any computing unit described in this specification. The computing unit deploys m computing nodes in a data-parallel manner, and these nodes are interconnected to form a structure as shown below. Figure 1 The ring communication structure shown, taking computing node k1 as an example, in this ring communication structure, computing node k1 only receives data from computing node k m Receives information but only sends information to computing node k2. That is, computing node k1 communicates only in a counter-clockwise direction within this computing group. Consequently, other computing nodes in this group also communicate only in the same counter-clockwise direction as k1, thus forming a ring-shaped communication structure with interconnected nodes. It should be noted that, as... Figure 1 The deployment structure of the computing nodes shown is a topology that only identifies the connection relationships, rather than a constraint on the actual location of each computing node. Figure 1 The diagram shows the ring communication structure of each computing node in the computing system in a counterclockwise direction. In actual implementation, this specification does not restrict the communication direction of each computing node in the computing system.

[0059] Figure 2 This is a flowchart illustrating a model training method described in this specification. For any computation node in any computational process, the model training method specifically includes the following steps:

[0060] S200: The computing node obtains the node gradient based on the pre-deployed model to be trained and the data shards allocated to the computing node itself.

[0061] After the data holder distributes the data shards and the model to be trained to each computing party, and each computing party deploys its own computing nodes, the model training process can begin. During model training, for any given computing party, each computing node within that party deploys the same model to be trained and holds a portion of the complete data shard allocated to that computing party. Therefore, any computing node within that party can perform computation based on the model to be trained deployed there and the allocated data shard fragment. The loss value is obtained by applying the calculated result and the loss function of the model to be trained. The node gradient of the computing node is then obtained by taking the partial derivative of this loss value with respect to the parameters of the model to be trained. Since the data shard fragments allocated to each computing node of that computing party are part of the complete data shard of that computing party, it is necessary to synchronize the node gradients calculated by each computing node in subsequent steps to obtain the computing party's own gradient.

[0062] S202: The node gradient is synchronized to other computing nodes in the computing party through the ring communication structure.

[0063] During node gradient synchronization, each computing node in the computational party receives information from only one of its adjacent computing nodes and sends information to the other adjacent computing node. Thus, the node gradient calculated by each computing node can be synchronized one by one to all other computing nodes in the computational party via this ring communication structure, until any computing node merges the node gradients of all other computing nodes in the computational party. This distributes the communication load evenly across each computing node in the computational party, fully utilizing the ingress and egress bandwidth of each computing node, increasing the overall communication speed. In particular, under the secure multi-party computation architecture, because the training models deployed by each computing node are based on encrypted data fragments to ensure homomorphism of the computation results, the amount of data generated by each computing node's computation results is much larger than that generated in conventional machine learning. Distributing the communication load evenly across each computing node effectively avoids communication bottlenecks (where the time required for communication in a training round exceeds the time required for computation, preventing data parallelism from achieving the desired accelerated training effect).

[0064] In one or more embodiments of this specification,

[0065] In the ring communication structure, the computing node that receives the computing node information is designated as the first communication node, and the computing node that sends information to the computing node is designated as the second communication node. The node gradient is synchronized to the first communication node through the ring communication structure. The node gradient synchronized by the second communication node is fused with its own node gradient. The fused node gradient is then synchronized to the first communication node again. The node gradient is then updated according to the node gradient synchronized by the second communication node, until the node gradients of all other computing nodes in the computing node are fused.

[0066] When the computing nodes of the computing party synchronize through the ring communication structure, for any computing node, when the computing node receives the node gradient synchronized by the second communication node of the computing node, it can also merge the node gradient synchronized by the second communication node with its own node gradient, and continue to synchronize the merged node gradient to the first communication node of the computing node. Thus, when the computing party has m computing nodes, the computing node can merge the node gradients of all other computing nodes in the computing party to obtain the computing party gradient through only m-1 rounds of communication, which can further accelerate the speed of node gradient synchronization.

[0067] S204: After obtaining the node gradients of all other computing nodes in the computing party, the computing node determines the computing party gradient based on the node gradient obtained by the computing node itself and the node gradients of all other computing nodes in the computing party.

[0068] Since the data shards allocated to each computing node of this computing party are all part of the data shards allocated to this computing party, and each data shard can be combined into a complete data shard, and each computing node deploys the same adjusted model (which can guarantee the homomorphism of the computation), the gradient of the node gradients calculated by each computing node after weighted average is equivalent to the gradient obtained by directly calculating the data shard. Therefore, the gradient of the node gradients calculated by each computing node in this computing party after weighted average is used as the computing party gradient of this computing party.

[0069] For any computational agent, once it obtains its gradient through the model training process described above, it synchronizes this gradient with other computational agents. These other computational agents, receiving the gradient, can then fuse it with their own gradients to obtain an updated gradient, and use this updated gradient to train the model to be trained.

[0070] Since the models to be trained deployed by each computing party are all adjusted models based on the data slices held by that computing party, the trainer for the training model also needs to determine the gradient of the computing party containing that data slice based on the data slices at the time of adjustment, and update the training model with the corresponding computing party gradient.

[0071] In one or more embodiments of this specification, the gradient of the computational party is specifically used to be synchronized by the computational party to a designated other computational party, wherein the designated other computational party is another computational party that has at least one of the same data shards as the computational party.

[0072] like Figure 2 The model training method described above, through the aforementioned ring communication structure, distributes the communication load during the communication process of the computing nodes evenly to each computing node in the computing party, and makes full use of the ingress and egress bandwidth of each computing node, which can increase the overall communication efficiency and reduce the communication cost in secure multi-party computation.

[0073] In addition, such as Figure 2 In step S102, the node gradient is a gradient matrix. Step S102 specifically includes: dividing the gradient matrix into several gradient sub-matrices equal to the number of computing nodes in the computing square, and synchronizing the gradient sub-matrices of each computing node to the first communication node through the ring communication structure. For example...

[0074] In machine learning training, communication and computation are two independent and parallel processes. By partitioning the gradient matrix, the single communication process of synchronizing the complete gradient matrix is ​​broken down into multiple communications synchronizing several gradient submatrices. This further reduces the bandwidth pressure of a single communication without affecting the model computation speed.

[0075] Furthermore, in such Figure 2 In step S102, according to the preset correspondence between each computing node and each position in the gradient matrix, the gradient submatrix corresponding to the position of the computing node is taken as the submatrix to be synchronized. The submatrix to be synchronized is synchronized to the first communication node. According to the position corresponding to the received submatrix synchronized by the second communication node, the received submatrix synchronized by the second communication node is merged with the gradient submatrix at the same position of the computing node itself. The latest merged node gradient submatrix is ​​then synchronized to the first communication node again as the submatrix to be synchronized. The merged received submatrix synchronized by the second communication node is then continued to be merged with the gradient submatrix at the same position of the computing node itself until the computing gradient matrix of all computing nodes in the computing square is obtained.

[0076] For example, in such Figure 1 In the communication system formed by the computing nodes under any of the computing methods shown, when each computing node calculates the node gradient used to synchronize with other computing nodes, computing node k1 divides its own calculated gradient matrix into m gradient sub-matrices, and numbers the gradient sub-matrices as k according to their positions at the time of division. 1,1 k 1,2 ... k 1,m Similarly, compute node k m Number its own gradient submatrix as k. m,1 k m,2 ... k m,m (The gradient submatrices segmented from each computation node of the same computational method follow the same numbering rule, i.e., k) 1,1 With k m,1 The gradient submatrices represented by these submatrices occupy the same position in their respective gradient matrices before partitioning. Based on this, each computation node completes the partitioning and numbering of its own gradient matrix. Subsequently, the first round of communication for node gradient synchronization is performed, with computation node k1 sending k... 1,1 The gradient submatrix is ​​synchronized to computation node k2, and computation node k2 will then... 2,2 The gradient submatrix is ​​synchronized to computation nodes k3, ..., k. m k m,mThe gradient submatrix is ​​synchronized to computation node k1. After receiving the gradient submatrix synchronized from its second communication node, each computation node updates its own gradient submatrix at the corresponding position using the received gradient submatrix. Specifically, taking computation node k1 as an example, computation node k1 will receive the gradient submatrix from k1... m,m Gradient submatrix and its own k 1,m The updated k is obtained by weighted averaging of gradient submatrices. 1,m Gradient submatrix. In the second round and subsequent communication of this node gradient synchronization, each computing node synchronizes its latest updated gradient submatrix to the first communication node, and updates its own gradient submatrix at the corresponding position based on the gradient submatrix synchronized by the second communication node. Thus, after m-1 rounds of node gradient synchronization, each computing node can have a complete gradient submatrix that merges the gradient submatrixes of all computing nodes at the same position, and the position of the complete gradient submatrix of each computing node in the original gradient matrix is ​​different. After another m-1 rounds of node gradient synchronization, each computing node can have the complete gradient of the computing party. By performing node gradient synchronization through this process, the computational load, computation time, communication load, and communication time of a single round of gradient fusion are reduced, and the efficiency of computing node communication is accelerated.

[0077] In one or more embodiments of this specification, the training sample can be risk control data, which may include user authentication data, behavioral characteristic data, and personal information. Before using the risk control data, it can be labeled according to the security risk events corresponding to each piece of data (the labeling information may be risky behavior occurring / no risky behavior occurring). Thus, in the method provided in this specification, for each piece of risk control data, multiple data fragments from the risk control data are allocated to several computing parties. These multiple data fragments are obtained by encrypting the risk control data. For each computing party, the computing party includes several computing nodes. The data fragments allocated to the computing party are distributed to each computing node within the computing party. The data fragments allocated to different computing nodes are not completely identical. The computing nodes within the computing party are connected to form a ring communication structure. In this ring structure, for any given computing node, the computing node has two adjacent computing nodes. The computing node receives information from one of its adjacent computing nodes and sends information to the other adjacent computing node. The method for any computing node in the computing party includes: the computing node obtains a node gradient based on a pre-deployed risk control model to be trained and the data shards allocated to the computing node itself; the node gradient is synchronized to other computing nodes in the computing party through the ring communication structure; after obtaining the node gradients of all other computing nodes in the computing party, the computing node determines the computing party gradient based on the node gradient obtained by the computing node itself and the node gradients of all other computing nodes in the computing party; the computing party gradient is used to synchronize to other computing parties, so that after each computing party receives the computing party gradient synchronized by other computing parties, it merges the received computing party gradient with its own computing party gradient to obtain an updated gradient, and trains the risk control model to be trained based on the updated gradient, wherein the node gradient of the computing node can be obtained through the following steps: obtaining a loss value based on the computing node's calculation result, the label corresponding to the calculation result, and the loss function of the model to be trained; and obtaining the node gradient of the computing node by taking the partial derivative of the loss value with respect to the parameters in the model to be trained. For the risk control model obtained after training, the trained risk control model can respond to user requests, determine the risk control data to be analyzed according to the user requests, input the risk control data to be analyzed into the trained risk control model, obtain the risk control prediction result output by the model, determine the probability that the risk control data to be analyzed has risk based on the risk control prediction result, and perform risk control on the risk event corresponding to the risk control data based on the probability.

[0078] The above describes a model training method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding model training apparatus, such as... Figure 3 As shown.

[0079] Figure 3 This specification provides a schematic diagram of a model training apparatus, specifically including:

[0080] The computation module 300 obtains the node gradients based on the pre-deployed model to be trained and the data slices allocated to the device itself.

[0081] Synchronization module 302 synchronizes the node gradient to other computing nodes in the computing party through the ring communication structure;

[0082] The fusion module 304, after obtaining the node gradients of all other computing nodes in the computing party, determines the computing party gradient based on the node gradient obtained by the device itself and the node gradients of all other computing nodes in the computing party. The computing party gradient is used to synchronize with other computing parties, so that after each computing party receives the synchronized computing party gradient from other computing parties, it merges the received computing party gradient with its own computing party gradient to obtain an updated gradient, and trains the model to be trained based on the updated gradient.

[0083] Optionally, the multiple data slices from the training sample specifically include: a number of data slices divided from the training sample, equal to the number of computational methods, wherein the sum of the values ​​of the divided data slices is equal to the value corresponding to the undivided training sample.

[0084] Optionally, the data shards allocated to the computing party specifically include: a data shard group consisting of at most n-1 of the n data shards derived from the training sample shards, and the data shard group of the computing party is not exactly the same as the data shard group of other computing parties.

[0085] Optionally, the synchronization module 302 is specifically used to: designate the computing node receiving the computing node information in the ring communication structure as the first communication node, designate the computing node sending information to the computing node as the second communication node, synchronize the node gradient to the first communication node through the ring communication structure, merge the node gradient synchronized by the second communication node with its own node gradient, continue to synchronize the merged node gradient to the first communication node, and continue to update its own node gradient according to the node gradient synchronized by the second communication node, until the node gradients of all other computing nodes in the computing node are merged.

[0086] Optionally, the node gradient is a gradient matrix, and the synchronization module 302 is specifically used to: divide the gradient matrix into several gradient sub-matrices equal to the number of computing nodes in the computing square according to the number of computing nodes in the computing square, and synchronize the gradient sub-matrices of the computing node itself to the first communication node through the ring communication structure.

[0087] Optionally, the synchronization module 302 is specifically used to: based on the preset correspondence between each computing node and each position in the gradient matrix, take the gradient submatrix corresponding to the position of the computing node as the submatrix to be synchronized, synchronize the submatrix to be synchronized to the first communication node, merge the received submatrix synchronized by the second communication node with the gradient submatrix at the same position of the computing node itself based on the position corresponding to the received submatrix synchronized by the second communication node, resynchronize the newly merged node gradient submatrix as the submatrix to be synchronized to the first communication node, and continue to merge the received submatrix synchronized by the second communication node with the gradient submatrix at the same position of the computing node itself, until the computing gradient matrix formed by merging all gradient submatrixes of all computing nodes in the computing square is obtained.

[0088] Optionally, the gradient of the computational party is specifically used to be synchronized by the computational party to a designated other computational party, wherein the designated other computational party is another computational party that has at least one of the same data shards as the computational party.

[0089] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 2 The provided model training method.

[0090] This instruction manual also provides Figure 4 The diagram shows a schematic structural representation of the electronic device. Figure 4 At the hardware level, the equipment used for training the model includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business logic. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functionality. Figure 2 The method for training the model described herein. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0091] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0092] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0093] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0094] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0095] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0100] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0101] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0102] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0105] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0106] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A method for training a model, wherein for each training sample, multiple data slices from the training sample are allocated to a plurality of computation parties, the multiple data slices being obtained by encrypting the training sample; For each computing party, which includes several computing nodes, the data shards allocated to the computing party are distributed to each computing node in the computing party, and the data shards allocated to different computing nodes are not exactly the same; The computing nodes in this computing unit are connected to form a ring communication structure. In the ring communication structure, for any computing node, the computing node has two adjacent computing nodes. The computing node receives information from one of the adjacent computing nodes and sends information to the other adjacent computing node. For any computing node in any computing party, the method includes: The computing node obtains its gradient based on the pre-deployed model to be trained and the data shards allocated to the computing node itself. The node gradient is synchronized to other computing nodes in the computing process through the ring communication structure. After obtaining the node gradients of all other computing nodes in the computing party, the computing node determines the computing party gradient based on its own node gradient and the node gradients of all other computing nodes in the computing party. The computing party gradient is used to synchronize with other computing parties, so that each computing party, after receiving the synchronized computing party gradients from other computing parties, merges the received computing party gradients with its own computing party gradients to obtain an updated gradient, and trains the model to be trained based on the updated gradient.

2. The method as described in claim 1, wherein the multiple data slices from the training sample specifically include: The training samples are divided into several data segments, the number of which is equal to the number of computational data segments. The sum of the values ​​of these data segments is equal to the value of the undivided training samples.

3. The method as described in claim 2, wherein the data shards allocated to the computing party specifically include: The training sample is divided into n data segments, which are composed of at most n-1 data segments. The data segment group of this computational party is not exactly the same as the data segment group of other computational parties.

4. The method as described in claim 1, wherein synchronizing the node gradient with other computing nodes in the computing party through the ring communication structure specifically includes: In the ring communication structure, the computing node that receives the computing node information is designated as the first communication node, and the computing node that sends information to the computing node is designated as the second communication node. The node gradient is synchronized to the first communication node through the ring communication structure; The node gradient synchronized by the second communication node is fused with its own node gradient; The merged node gradient is then synchronized to the first communication node. Continue to update its own node gradient based on the node gradient synchronized by the second communication node, until the node gradients of all other computing nodes in the computing process are merged.

5. The method as described in claim 4, wherein the node gradient is a gradient matrix; The process of synchronizing the node gradient to the first communication node through the ring communication structure specifically includes: Based on the number of computing nodes within the computation square, the gradient matrix is ​​divided into several gradient sub-matrices, which are equal in number to the number of computing nodes within the computation square. The ring communication structure is used to synchronize the gradient submatrix of the computing node to the first communication node.

6. The method as described in claim 5, wherein synchronizing the node gradient with other computing nodes in the computing party through the ring communication structure specifically includes: Based on the pre-defined correspondence between each computation node and each position in the gradient matrix, the gradient submatrix corresponding to the position of the computation node is taken as the submatrix to be synchronized. The sub-matrix to be synchronized is synchronized to the first communication node; Based on the position corresponding to the received sub-matrix synchronized by the second communication node, the received sub-matrix synchronized by the second communication node is fused with the gradient sub-matrix at the same position of the computing node itself. The newly fused node gradient submatrix is ​​resynchronized to the first communication node as the submatrix to be synchronized, and the received submatrix synchronized by the second communication node is fused with the gradient submatrix at the same position of the computing node until the computing gradient matrix of the computing square is obtained by fusing all the gradient submatrixes of all computing nodes in the computing square.

7. The method of claim 3, wherein the gradient of the computational party is specifically used to be synchronized by the computational party to other designated computational parties, wherein, The designated other computing party is another computing party that has at least one of the same data shards as the computing party.

8. An apparatus for model training, wherein for each training sample, multiple data slices from the training sample are allocated to a plurality of computation parties, the multiple data slices being obtained by encrypting the training sample; For each computing party, which includes several computing nodes, the data shards allocated to the computing party are distributed to each computing node in the computing party, and the data shards allocated to different computing nodes are not exactly the same; The computing nodes in this computing unit are connected to form a ring communication structure. In the ring communication structure, for any computing node, the computing node has two adjacent computing nodes. The computing node receives information from one of the adjacent computing nodes and sends information to the other adjacent computing node. The device is applied to any computing node in any computing party, and the device includes: The computation module obtains node gradients based on the pre-deployed model to be trained and the data slices allocated to the device itself. The synchronization module synchronizes the node gradient with other computing nodes in the computing party through the ring communication structure. The fusion module, after obtaining the node gradients of all other computing nodes in the computing party, determines the computing party gradient based on the node gradient obtained by the device itself and the node gradients of all other computing nodes in the computing party. The computing party gradient is used to synchronize with other computing parties, so that each computing party, after receiving the synchronized computing party gradients from other computing parties, merges the received computing party gradients with its own computing party gradients to obtain an updated gradient, and trains the model to be trained based on the updated gradient.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 7.

Images