Distributed-based model training method and apparatus

Abstract

Embodiments of the present specification provide a distributed-based model training method and device, wherein the distributed-based model training method comprises: each first participant distributes sample data blocks to different training processes, performs forward propagation based on a client model, sends forward intermediate results obtained by the different training processes to a second participant, the second participant distributes the forward intermediate results to corresponding training processes, performs forward propagation and backward propagation based on a server model, sends backward intermediate results obtained in the backward propagation to each first participant, synchronizes gradient data obtained in the backward propagation with other training processes and updates server model parameters, each first participant distributes the backward intermediate results to corresponding training processes, performs backward propagation based on the client model, synchronizes the gradient data obtained in the backward propagation with other training processes and updates client model parameters.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a distributed model training method and apparatus. Background Technology

[0002] Split learning is a distributed machine learning paradigm that allows participants to train machine learning models without sharing any raw data. By splitting the model structure among participants, split learning enables them to collaborate on solving machine learning problems, training models more efficiently while ensuring data security. With the continuous development of Internet technology, the application areas of split learning are gradually increasing, such as in the medical field. In this process, using split learning techniques for machine learning modeling has gradually become a key focus for modelers. Summary of the Invention

[0003] This specification provides one or more embodiments of a first distributed model training method, applied to multiple participants running multiple training processes, where the training processes of different participants correspond to each other. A first participant among the multiple participants has a client model, and a second participant among the multiple participants has a server model. The method includes: each first participant assigns sample data blocks to different training processes and performs forward propagation based on the client model, and sends the forward intermediate results obtained from the forward propagation of different training processes to the second participant. Entity alignment is performed between the sample data blocks assigned to the corresponding training processes by different first participants. The second participant receives the forward intermediate results sent by different first participants, assigns them to the corresponding training processes, and performs forward and backward propagation sequentially based on the server model, sending the backward intermediate results obtained in the backward propagation to each first participant. Furthermore, after calculating gradient data during the backward propagation process, gradient data synchronization is performed with other training processes, and the server model parameters are updated. Each of the first participants receives the reverse intermediate results from the second participant and assigns them to the corresponding training process to continue backpropagation based on the client model. During the backpropagation process, after calculating the gradient data, the gradient data is synchronized with other training processes and the client model parameters are updated.

[0004] This specification provides one or more embodiments of a second distributed model training method, applied to a first participant running multiple training processes. The training processes of each first participant correspond to each other and also correspond to the training processes of a second participant. Each first participant has a client model, and the second participant has a server model. The method includes: allocating sample data blocks to different training processes for forward propagation based on the client model, and sending the forward intermediate results obtained from the forward propagation of different training processes to the second participant. Entity alignment is performed between sample data blocks allocated to corresponding training processes by different first participants. The method also includes receiving the reverse intermediate results sent by the second participant, allocating them to the corresponding training processes for backpropagation based on the client model, calculating gradient data during backpropagation, synchronizing gradient data with other training processes, and updating client model parameters. The reverse intermediate results are obtained by the second participant after allocating the forward intermediate results sent by different first participants to the corresponding training processes and sequentially performing forward and reverse propagation based on the server model.

[0005] This specification provides one or more embodiments of a third distributed model training method, applied to a second participant running multiple training processes. The multiple training processes of the second participant correspond to the multiple training processes of each first participant. Each first participant has a client model, and the second participant has a server model. The method includes: receiving forward intermediate results sent by each first participant. The forward intermediate results are obtained by each first participant allocating sample data blocks to different training processes and performing forward propagation based on the client model, with entity alignment between the sample data blocks allocated by different first participants to their corresponding training processes. The forward intermediate results are then allocated to the corresponding training processes and forward and backward propagation is performed sequentially based on the server model. The backward intermediate results obtained during backward propagation are sent to each first participant, and gradient data is calculated during backward propagation. Gradient data is then synchronized with other training processes, and server model parameters are updated.

[0006] This specification provides one or more embodiments of a distributed model training system, including multiple participants running multiple training processes, with the training processes of different participants corresponding to each other. A first participant among the multiple participants has a client model, and a second participant among the multiple participants has a server model. Each first participant is configured to allocate sample data blocks to different training processes for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of different training processes to the second participant. It also receives reverse intermediate results from the second participant, allocates them to the corresponding training process, and continues reverse propagation based on the client model. During reverse propagation, gradient data is calculated, and gradient data is synchronized with other training processes, and client model parameters are updated. Entity alignment is achieved between sample data blocks allocated to corresponding training processes by different first participants. The second participant is configured to receive forward intermediate results sent by different first participants, allocate them to the corresponding training process, and sequentially perform forward and reverse propagation based on the server model, sending the reverse intermediate results obtained during reverse propagation to each first participant. Furthermore, after calculating the gradient data during backpropagation, the gradient data is synchronized with other training processes and the server-side model parameters are updated.

[0007] This specification provides one or more embodiments of a first distributed model training device, running on multiple participants with multiple training processes, wherein the training processes of different participants correspond to each other. A first participant among the multiple participants has a client model, and a second participant among the multiple participants has a server model. The device includes: a data block allocation module, configured to have each first participant allocate sample data blocks to different training processes for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of different training processes to the second participant. Entity alignment is achieved between the sample data blocks allocated by different first participants to their corresponding training processes. An intermediate result receiving module is configured to have the second participant receive the forward intermediate results sent by different first participants, allocate them to their corresponding training processes for forward and backward propagation based on the server model, and send the backward intermediate results obtained during backward propagation to each first participant. Furthermore, after calculating gradient data during backward propagation, the gradient data is synchronized with other training processes, and the server model parameters are updated. The intermediate result allocation module is configured such that each of the first participants receives the reverse intermediate results from the second participant and allocates them to the corresponding training process to continue backpropagation based on the client model. During the backpropagation process, gradient data is calculated and then synchronized with other training processes to update the client model parameters.

[0008] This specification provides one or more embodiments of a second distributed model training device, running on a first participant with multiple training processes. The training processes of each first participant correspond to each other and also correspond to the training processes of a second participant. Each first participant has a client model, and the second participant has a server model. The device includes: a forward intermediate result sending module, configured to distribute sample data blocks to different training processes for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of different training processes to the second participant. Entity alignment is achieved between sample data blocks distributed by different first participants to their corresponding training processes. A reverse intermediate result receiving module is configured to receive the reverse intermediate results sent by the second participant, distribute them to the corresponding training processes for backpropagation based on the client model, calculate gradient data during backpropagation, synchronize gradient data with other training processes, and update client model parameters. The reverse intermediate results are obtained by the second participant distributing the forward intermediate results sent by different first participants to their corresponding training processes and sequentially performing forward and reverse propagation based on the server model.

[0009] This specification provides one or more embodiments of a third distributed model training device, running on a second participant with multiple training processes. These training processes correspond to the training processes of each first participant. Each first participant has a client model, and each second participant has a server model. The device includes: a forward intermediate result receiving module configured to receive forward intermediate results sent by each first participant. The forward intermediate results are obtained by each first participant allocating sample data blocks to different training processes and performing forward propagation based on the client model, with entity alignment between sample data blocks allocated by different first participants to their corresponding training processes; a forward intermediate result allocation module configured to allocate the forward intermediate results to corresponding training processes and perform forward and backward propagation sequentially based on the server model; and a backward intermediate result sending module configured to send the backward intermediate results obtained during backward propagation to each first participant, and, after calculating gradient data during backward propagation, synchronize gradient data with other training processes and update server model parameters.

[0010] This specification provides one or more embodiments of a first distributed model training device, comprising: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: allocate sample data blocks to different training processes for forward propagation based on a client model, and send forward intermediate results obtained from forward propagation in different training processes to a second participant. Entity alignment is achieved between sample data blocks allocated to corresponding training processes by different first participants. The second participant receives the forward intermediate results sent by different first participants and allocates them to corresponding training processes for forward and backward propagation based on a server model, sending the backward intermediate results obtained during backward propagation to each of the first participants. Furthermore, after calculating gradient data during backward propagation, the participant synchronizes gradient data with other training processes and updates server model parameters. Each first participant receives the backward intermediate results from the second participant and allocates them to corresponding training processes for continued backward propagation based on a client model. After calculating gradient data during backward propagation, the participant synchronizes gradient data with other training processes and updates client model parameters.

[0011] This specification provides one or more embodiments of a second distributed model training device, comprising: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: allocate sample data blocks to different training processes for forward propagation based on a client model, and send forward intermediate results obtained from forward propagation in different training processes to a second participant. Entity alignment is performed between sample data blocks allocated to corresponding training processes by different first participants. The processor receives reverse intermediate results sent by the second participant, allocates them to corresponding training processes for backpropagation based on a client model, calculates gradient data during backpropagation, synchronizes gradient data with other training processes, and updates client model parameters. The reverse intermediate results are obtained by the second participant after allocating forward intermediate results sent by different first participants to corresponding training processes and sequentially performing forward and reverse propagation based on a server model.

[0012] This specification provides one or more embodiments of a third distributed model training device, comprising: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: receive forward intermediate results sent by each first participant. The forward intermediate results are obtained by each first participant allocating sample data blocks to different training processes and performing forward propagation based on a client model, with entity alignment between sample data blocks allocated by different first participants to their respective training processes. The forward intermediate results are then allocated to corresponding training processes and subjected to forward and backward propagation sequentially based on a server model. The backward intermediate results obtained during backward propagation are sent to each first participant, and gradient data is calculated during backward propagation, synchronized with other training processes, and the server model parameters are updated.

[0013] This specification provides one or more embodiments of a first computer-readable storage medium for storing computer-executable instructions, which, when executed, perform the following steps: Each first participant allocates sample data blocks to different training processes for forward propagation based on a client model, and sends the forward intermediate results obtained from the forward propagation of different training processes to a second participant. Entity alignment is achieved between sample data blocks allocated to corresponding training processes by different first participants. The second participant receives the forward intermediate results sent by different first participants, allocates them to corresponding training processes, and performs forward and backward propagation sequentially based on a server model. The second participant sends the backward intermediate results obtained during backward propagation to each first participant. Furthermore, after calculating gradient data during backward propagation, the second participant synchronizes gradient data with other training processes and updates the server model parameters. Each first participant receives the backward intermediate results from the second participant, allocates them to corresponding training processes, and continues backward propagation based on a client model. After calculating gradient data during backward propagation, the second participant synchronizes gradient data with other training processes and updates the client model parameters.

[0014] This specification provides one or more embodiments of a second computer-readable storage medium for storing computer-executable instructions that, when executed, perform the following steps: allocating sample data blocks to different training processes for forward propagation based on a client model, and sending the forward intermediate results obtained from the forward propagation of different training processes to a second participant. Entity alignment is achieved between sample data blocks allocated to corresponding training processes by different first participants. The reverse intermediate results sent by the second participant are received and allocated to the corresponding training processes for backpropagation based on the client model. During backpropagation, gradient data is calculated and synchronized with gradient data from other training processes, and client model parameters are updated. The reverse intermediate results are obtained by the second participant after allocating the forward intermediate results sent by different first participants to corresponding training processes and performing forward and reverse propagation sequentially based on a server model.

[0015] This specification provides one or more embodiments of a third computer-readable storage medium for storing computer-executable instructions that, when executed, perform the following steps: receiving forward intermediate results sent by each first participant. The forward intermediate results are obtained by each first participant allocating sample data blocks to different training processes and performing forward propagation based on the client model, with entity alignment between sample data blocks allocated to corresponding training processes by different first participants. The forward intermediate results are then allocated to corresponding training processes and forward and backward propagation is performed sequentially based on the server model. The backward intermediate results obtained during backward propagation are sent to each first participant, and gradient data is calculated during backward propagation, synchronized with other training processes, and the server model parameters are updated. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram illustrating an implementation environment for a distributed model training method provided in one or more embodiments of this specification;

[0018] Figure 2 A flowchart illustrating the first distributed model training method provided for one or more embodiments of this specification;

[0019] Figure 3A schematic diagram illustrating a distributed model training architecture provided for one or more embodiments of this specification;

[0020] Figure 4 A schematic diagram illustrating a forward propagation process and a backward propagation process provided for one or more embodiments of this specification;

[0021] Figure 5 A flowchart illustrating a distributed model training method for distributed training scenarios, provided for one or more embodiments of this specification.

[0022] Figure 6 A flowchart illustrating a second distributed model training method provided for one or more embodiments of this specification;

[0023] Figure 7 This document provides a timing diagram for a distributed model training method applied to a distributed training scenario, as illustrated in one or more embodiments of this specification.

[0024] Figure 8 A flowchart illustrating a third distributed model training method provided in one or more embodiments of this specification;

[0025] Figure 9 A schematic diagram illustrating an embodiment of a distributed model training system provided in one or more embodiments of this specification;

[0026] Figure 10 A schematic diagram of a first embodiment of a distributed model training device provided for one or more embodiments of this specification;

[0027] Figure 11 A schematic diagram of a second embodiment of a distributed model training device provided for one or more embodiments of this specification;

[0028] Figure 12 A schematic diagram of a third embodiment of a distributed model training device provided for one or more embodiments of this specification;

[0029] Figure 13 A schematic diagram of the structure of a first distributed model training device provided for one or more embodiments of this specification;

[0030] Figure 14 A schematic diagram of the structure of a second distributed model training device provided for one or more embodiments of this specification;

[0031] Figure 15 This is a schematic diagram of the structure of a third distributed model training device provided for one or more embodiments of this specification. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0033] The distributed model training method provided in one or more embodiments of this specification is applicable to the implementation environment of a training system. (Refer to...) Figure 1 The implementation environment includes at least:

[0034] Multiple participants, including each first participant 101 and second participant 102; the total number of first participants is not specifically limited, such as including first participant A, first participant B, etc.

[0035] Each of the first participants 101 can be deployed on different devices, such as mobile phones, personal computers, tablets, e-book readers, wearable devices, devices that interact based on AR (Augmented Reality) / VR (Virtual Reality), and laptop computers, etc. The second participant 102 can be deployed on a server, which can be a single server, a server cluster consisting of several servers, or one or more cloud servers in a cloud computing platform.

[0036] In this implementation environment, multiple participants run multiple training processes, which correspond to each other. The first participant (101) has a client-side model, and the second participant has a server-side model. During model training, each first participant (101) allocates sample data blocks to different training processes and performs forward propagation based on the client-side model. The forward intermediate results obtained from the forward propagation of different training processes are sent to the second participant (102). Entity alignment is achieved between the sample data blocks allocated to the corresponding training processes by different first participants. This entity alignment synchronizes the training progress between the corresponding training processes of different first participants, avoiding training errors caused by entity misalignment and improving the effectiveness of model training. Furthermore, by introducing multiple training processes in each first participant, these processes can process corresponding sample data blocks in a distributed manner, increasing the processing capacity of sample data blocks and improving model training efficiency. Based on this, after receiving the forward intermediate results sent by different first participants, the second participant (102) distributes the forward intermediate results... In the corresponding training process, forward and backward propagation are performed based on the server-side model. During backward propagation, intermediate results and gradient data are obtained. The intermediate results are returned to each first participant 101, and the gradient data is synchronized with other training processes to update the server-side model parameters. By introducing multiple training processes in the second participant, multiple training processes can process the corresponding intermediate results in a distributed manner, improving model training efficiency. Each first participant 101 receives the intermediate results from the second participant 102, distributes the intermediate results to the corresponding training process, performs backward propagation based on the client-side model, and synchronizes the gradient data obtained during backward propagation with other training processes to update the client-side model parameters. By synchronizing gradient data and updating model parameters, the model parameters of multiple training processes in each participant are kept synchronized, which facilitates timely acquisition of the trained client-side and server-side models after training. Furthermore, multiple training processes enable distributed training on multiple GPUs (Graphics Processing Units) on multiple machines or a single machine, making full use of cluster resources and improving the processing capacity of sample data and model training efficiency.

[0037] One or more embodiments of the first distributed model training method provided in this specification are as follows:

[0038] Reference Figure 2The distributed model training method provided in this embodiment can be applied to multiple participants running multiple training processes, and the multiple training processes of different participants correspond to each other. The first participant among the multiple participants has a client model, and the second participant among the multiple participants has a server model. Specifically, it includes steps S202 to S206.

[0039] In step S202, each first participant allocates sample data blocks to different training processes for forward propagation based on the client model, and sends the forward intermediate results obtained from the forward propagation of different training processes to the second participant.

[0040] In practical applications, when training a model through split learning, the amount of sample data may be large. If multiple participants use a single process for training, the single process may be limited by resources, resulting in slow computation speed and thus slow training speed. This embodiment introduces multiple training processes based on split learning, enabling each participant to perform distributed model training through multiple training processes, thereby improving model training efficiency.

[0041] In this embodiment, the multiple participants refer to multiple participants who jointly train the model through split learning. The multiple participants include each first participant and each second participant. Each first participant includes each client. Each first participant is independent, and the number of first participants can be arbitrary and is not specifically limited here. The second participant includes the server.

[0042] Each of the multiple participants can run multiple training processes, and there are corresponding training processes between different participants. In this embodiment, the training process refers to the training process run by each participant to train the model. These training processes are carried out in parallel in a distributed manner. For example, the first participant A runs training process 0 and training process 1, the first participant B runs training process 0 and training process 1, and the second participant runs training process 0 and training process 1.

[0043] Optionally, the number of training processes run by each first participant is equal, and the number of training processes run by each first participant is equal to the number of training processes run by the second participant; the training processes of each first participant correspond to each other, and also correspond to the training processes of the second participant; each training process in the training processes of each first participant has its own client model, and each training process in the training processes of the second participant has its own server model.

[0044] like Figure 3As shown, each first participant includes first participant A and first participant B. The training process of first participant A includes training process 0 and training process 1, the training process of first participant B includes training process 0 and training process 1, and the training process of the second participant also includes training process 0 and training process 1. The training process 0 of first participant A, the training process 0 of first participant B, and the training process 0 of second participant A correspond to each other, as do the training process 1 of first participant A, the training process 1 of first participant B, and the training process 1 of second participant B. The training process 0 of first participant A may have a client model A, the training process 1 of first participant A may have a client model A, and the client model A of the training process 0 of first participant A is related to the training process. The client model A of the first participant can have the same model structure, and the model parameters during the model training process can be the same or different. The first participant B is similar. The training process 0 and training process 1 of the first participant B each have their own client model B. The model structure of the client model B of the training process 0 and training process 1 of the first participant B can be the same, and the model parameters during the model training process can be the same or different. The model structure of client model A and client model B can be the same or different. The training process 0 and training process 1 of the second participant can each have a server model. The model structure of the server model of the training process 0 and the server model of the training process 1 of the second participant can be the same, but the model parameters during the model training process can be the same or different.

[0045] The sample data block mentioned in this embodiment refers to the data block read from the data shards of each first participant. The data shards of each first participant refer to the data units obtained after performing relevant data processing on the training data of each first participant. The number of data shards of each first participant can be equal to the number of training processes. Multiple training processes of each first participant can correspond to each other with data shards. For example, data shards can be represented by shards. If the training processes of first participant A are training process 0 and training process 1, then first participant A has shard0 and shard1. Optionally, entity alignment can be performed between sample data blocks allocated to the corresponding training processes by different first participants; for example, entity alignment can be performed between sample data blocks allocated to training process 0 by first participant A and sample data blocks allocated to training process 0 by first participant B, and entity alignment can be performed between sample data blocks allocated to training process 1 by first participant A and sample data blocks allocated to training process 1 by first participant B; entity alignment includes the same entity identifier in the sample data blocks, and the entity identifier can be a user identifier. For example, if the entity identifier in the sample data block allocated to training process 0 by first participant A is r1, then the entity identifier in the sample data block allocated to training process 0 by first participant B is also r1.

[0046] In practical applications, the model training process usually involves many rounds, and the amount of data in the data slices may also be large. Therefore, in order to improve the data processing speed of the model in each round and avoid the large amount of data processing causing the model to be too slow and affecting the training efficiency, in an optional implementation of this embodiment, the sample data blocks of each first participant are read from the data slices of each first participant.

[0047] The data shards of each first participant are obtained in the following manner:

[0048] The training data of each first participant is subjected to entity alignment processing, and the obtained aligned data is split into multiple data fragments of each first participant; optionally, the number of data fragments obtained by data splitting is equal to the number of training processes.

[0049] The sample data blocks of each first participant can be read from the data shards of each first participant according to a preset number. The preset number can be a pre-set number, such as m. Then, sample data blocks can be read from the data shards of each first participant according to the preset number m. For example, first participant A reads sample data block 0 of the current training round from data shard 0 according to the preset number m, and reads sample data block 1 of the current training round from data shard 1 according to the preset number m. The above data segmentation can be performed by the data segmentation module of each first participant. After the data segmentation module of each first participant performs data segmentation to obtain multiple data shards, the multiple data shards can be stored. The data segmentation module here can be a DataSplitCoordinator.

[0050] Based on the above-mentioned data segmentation of the obtained aligned data to obtain multiple data fragments for each first participant, optionally, the multiple data fragments of each first participant are allocated to multiple training processes of each first participant, and there is a one-to-one correspondence between the allocated data fragments and training processes; specifically, each first participant can read multiple data fragments from its own data segmentation module and allocate the multiple data fragments to multiple training processes of each first participant; for example, first participant A reads data fragment 0 and data fragment 1 from its data segmentation module, allocates data fragment 0 to training process 0 of first participant A, and allocates data fragment 1 to training process 1 of first participant A, that is, data fragment 0 of first participant A corresponds to training process 0 and data fragment 1 corresponds to training process 1, and the same applies to first participant B.

[0051] In specific implementation, during model training, there are various first participants. To enable synchronous collaboration among these first participants in model training, entity alignment processing can be performed on the training data of each first participant. This helps each first participant read the aligned training data for model training, improving the accuracy and convenience of model training. Based on the above-mentioned entity alignment processing of the training data of each first participant and the data segmentation of the obtained aligned data to obtain multiple data fragments of each first participant, in an optional implementation of this embodiment, the following operation is performed during the entity alignment processing of the training data of each first participant:

[0052] The intersection of the training data of each first participant is calculated based on the entity identifiers in the training data to obtain the intersection data of each first participant.

[0053] The intersection data of each of the first participants is aligned according to the entity identifiers in the intersection data.

[0054] The training data of each first participant refers to the initial training data of each first participant. The entity identifier in the training data refers to the identifier of the entity object in the training data. Since there may be multiple training data, the entity identifier in the training data can be used to represent which training data in the training data. The entity identifier can be a user identifier. For example, the training data of first participant A includes user resource behavior data stored by institution a. The user identifiers in each user resource behavior data are also entity identifiers, namely r1, r2, r3, r4, etc. That is, the user resource behavior data includes user resource behavior data of user r1, user resource behavior data of user r2, and user resource behavior data of user r3. The training data of first participant B includes user occupation data stored by institution b, which may include user occupation data of user r2, user occupation data of user r1, and user occupation data of user r3. The entity identifiers in the training data of first participant B are r2, r1, and r3.

[0055] The intersection calculation includes selecting intersection data with the same entity identifiers from the training data of each first participant based on the entity identifiers in the training data; that is, the entity identifiers in the intersection data of each first participant are the same, but the data content of the intersection data may differ; for example, based on the entity identifiers in the training data of first participant A and the entity identifiers in the training data of first participant B, the intersection data of first participant A is calculated as user resource behavior data corresponding to r1, r2, and r3, and the intersection data of first participant B is user occupation data corresponding to r2, r1, and r3. That is, the entity identifiers in the intersection data of first participant A and first participant B are the same, but the data content of the intersection data is different; in this way, the data of each first participant is aligned through intersection calculation, avoiding invalid model training caused by misaligned data.

[0056] Specifically, based on the entity identifiers in the training data of each first participant, training data with the same entity identifiers can be selected as the intersection data of each first participant. To improve the convenience of data reading in each round during model training, the intersection data of each first participant can be further sorted according to the entity identifiers in the intersection data to obtain the aligned data of each first participant. Further data alignment through sorting helps to improve the synchronization and frequency of subsequent model training.

[0057] Continuing with the previous example, the intersection data of the first participant A consists of user resource behavior data corresponding to r1, r2, and r3. The intersection data of the first participant B consists of user occupation data corresponding to r2, r1, and r3. The intersection data of the first participant A and the first participant B are sorted according to the entity identifiers in the intersection data. The aligned data of the first participant A is then obtained as user resource behavior data corresponding to r1, r2, and r3. The data, the alignment data of the first participant B, is the user occupation data corresponding to r1, r2, and r3, realizing the alignment of the first participant A and the first participant B in the entity dimension and the order dimension. Here, there is no specific limitation on the sorting method. It can be from large to small, from small to large, or the intersection data of one first participant can be used as a benchmark to sort the intersection data of another first participant, so that the entity identifiers in the intersection data of the other first participant are aligned with the entity identifiers in the intersection data of one first participant.

[0058] After performing entity alignment processing on the training data of each first participant, the obtained aligned data is split into multiple data slices for each first participant. This allows multiple training processes of each first participant to have their own data slices and perform model training synchronously. In the process of splitting the obtained aligned data into multiple data slices for each first participant, the data can be split into multiple data slices for each first participant based on the number of data in the aligned data of each first participant and the number of training processes running by each first participant. The number of data in the aligned data includes the number of entity identifiers in the aligned data of each first participant. For example, if the entity identifiers include r1, r2, and r3, then the number of data in the aligned data is 3.

[0059] Specifically, the ratio of the number of data points in the alignment data of each first participant to the number of processes in the training process can be calculated as the data interval identifier for each first participant to perform data segmentation. The alignment data of each first participant is then segmented according to the data interval identifier to obtain multiple data fragments for each first participant. Here, the data interval identifier can represent the segmentation range for data segmentation.

[0060] For example, if the number of training processes for first participant A and first participant B is 2, and the number of data points in first participant A's alignment data is 1000, then the data interval identifiers for first participant A's data segmentation are 0 and 500, and 500 and 1000. Data segmentation is performed on first participant A's alignment data according to these interval identifiers, resulting in data segment 0 and data segment 1. Data segment 0 corresponds to the alignment data within the interval [0, 500) in first participant A's alignment data, and data segment 1 corresponds to the alignment data within the interval [500, 1000) in first participant A's alignment data. If the number of data points in first participant B's alignment data is 1000, then first participant B performs data segmentation... The data intervals are labeled as 0 and 500, and 500 and 1000. The aligned data of the first participant B is segmented according to the data interval labels to obtain data segment 0 and data segment 1 of the first participant B. Data segment 0 of the first participant B corresponds to the aligned data in the interval [0, 500) of the aligned data of the first participant B, and data segment 1 of the first participant B corresponds to the aligned data in the interval [500, 1000) of the aligned data of the first participant B. In this way, the data segmentation enables multiple training processes of each first participant to have their own data segments, and each data segment is aligned, so that the model training is performed synchronously among multiple training processes, reducing the probability of model training errors.

[0061] In practical applications, entity alignment can be achieved among the various first participants. However, when the sample labels are not carried by the first participants and are carried by the second participants, the second participants also have a need to align the sample labels with the entity labels of the first participants. That is, the data of the first and second participants all have a need for entity alignment. Therefore, the training data of the first participants can be aligned with the initial sample labels of the second participants. This improves the accuracy and effectiveness of backpropagation in the subsequent backpropagation process, ensuring that the results of the forward propagation of the server model of the second participant match the sample labels, thus improving the accuracy of backpropagation. In an optional implementation of this embodiment, before the first participants allocate sample data blocks to different training processes for forward propagation based on the client model and send the forward intermediate results obtained from the forward propagation of different training processes to the second participant for execution, the following operation is also performed:

[0062] Entity alignment is performed on the training data of each first participant and the initial sample labels of the second participant.

[0063] The obtained aligned data of each first participant and the aligned sample labels of the second participant are split into multiple data fragments of each first participant and sample labels of the second participant.

[0064] It should be noted that the initial sample labels here also include entity identifiers; the process of entity alignment and data segmentation here is similar to the specific implementation process of entity alignment and data segmentation of the training data of each first participant described above. The difference is that the above refers to each first participant, while here there are second participants in addition to each first participant, but the execution process is similar. You can refer to the reading, and this embodiment will not be described again here.

[0065] like Figure 3 As shown, the first participant A has data shard 0 and data shard 1, the first participant B has data shard 0 and data shard 1, and the second participant has sample label 0 and sample label 1. Data shard 0 of the first participant A and data shard 0 of the first participant B correspond to sample label 0 of the second participant, and data shard 1 of the first participant A and data shard 1 of the first participant B correspond to sample label 1 of the second participant.

[0066] It should be added that, in this embodiment, the client model set by the first participant and the server model set by the second participant can be obtained by splitting the model structure of the model to be trained; in this way, training of the model to be trained with a more complex model structure can be achieved by split learning, that is, by splitting the model structure, thereby improving the flexibility and efficiency of model training.

[0067] It should also be noted that the client model in this embodiment can be any application domain, any type, and any model structure, and the server model can also be any application domain, any type, and any model structure. This embodiment does not impose specific limitations on the model structure or application scenario of the client model and the server model.

[0068] In the specific execution process, each first participant can allocate sample data blocks to the corresponding training process for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of each training process to the second participant; here, forward propagation can be data processing of the input sample data blocks; for example, if the input is image data, forward propagation can be image feature extraction; such as... Figure 3 As shown, the first participant A reads sample data block 0 from data shard 0 of the first participant A, allocates the read sample data block 0 to the training process 0 of the first participant A for forward propagation based on client model A, reads sample data block 1 from data shard 1 of the first participant A, allocates the read sample data block 1 to the training process 1 of the first participant A for forward propagation based on client model A, and sends the forward intermediate results of training process 0 and training process 1 to the second participant; the processing of training process 0 and training process 1 of the first participant B is similar to that of the first participant A, and will not be described again.

[0069] It should be noted that in this embodiment, the last layer of each client model can be a segmentation layer, that is, the boundary layer between the client model and the server model; for example... Figure 3 As shown, the last layer of each client model is a segmentation layer. Each first participant allocates sample data blocks to different training processes and performs forward propagation based on the client model. The forward intermediate results of the segmentation layer output obtained from the forward propagation of different training processes are sent to the second participant; for example... Figure 4 The forward propagation process shown is as follows: In the client model A of training process 0 of the first participant A, layer 1 of client model A processes sample data block 0 and outputs y1; layer 2 of client model A processes y1 and outputs y2; layer 3 (segmentation layer) of client model A processes y2 to obtain the segmentation layer output, which is the forward intermediate result. The processing process of client model A in training process 1 of the first participant A is similar to that in training process 0, and will not be repeated here. The segmentation layer output of client model A is sent to the second participant as the forward intermediate result. In this way, the first participant and the second participant are decoupled through the segmentation layer. By sending only the segmentation layer output to the second participant, data isolation is achieved between the first participant and the second participant, ensuring data security between the participants.

[0070] In step S204, the second participant receives the forward intermediate results sent by different first participants and allocates them to the corresponding training processes. Based on the server-side model, forward propagation and backward propagation are performed sequentially. The reverse intermediate results obtained in the backward propagation are sent to each of the first participants. After calculating the gradient data during the backward propagation process, the gradient data is synchronized with other training processes and the server-side model parameters are updated.

[0071] Each of the aforementioned first participants allocates sample data blocks to different training processes and performs forward propagation based on the client model. The forward intermediate results obtained from the forward propagation of different training processes are sent to the second participants. In this step, after receiving the forward intermediate results sent by different first participants, the second participants allocate the forward intermediate results to the corresponding training processes and perform forward and backward propagation based on the server model in sequence. The backward intermediate results obtained in the backward propagation are returned to each of the first participants. Furthermore, the gradient data obtained in the backward propagation is synchronized with the gradient data of other training processes and the server model parameters are updated.

[0072] In practice, after receiving the forward intermediate results sent by different first participants, the second participant distributes the forward intermediate results to the corresponding training process and performs forward propagation based on the server-side model. Based on the forward propagation results, it performs back propagation in the server-side model and sends the reverse intermediate results obtained in the back propagation to each first participant. In addition, after calculating the gradient data during the back propagation process, it synchronizes the gradient data with other training processes and updates the parameters of its respective server-side model based on the synchronization results.

[0073] The reverse intermediate result can be the gradient data corresponding to the segmentation layer output (i.e., the forward intermediate result). The gradient data calculated during the backpropagation process can be the gradient data of the server-side model parameters, including the gradient data corresponding to the weights, biases, and other parameters of the server-side model.

[0074] like Figure 3 As shown, the second participant receives the forward intermediate results sent by each of the first participants. The second participant distributes the forward intermediate results of training process 0 of the first participant A and the forward intermediate results of training process 0 of the first participant B to the training process 0 of the second participant, and performs forward propagation and backward propagation in sequence based on the server-side model. The second participant distributes the forward intermediate results of training process 1 of the first participant A and the forward intermediate results of training process 1 of the first participant B to the training process 1 of the second participant, and performs forward propagation and backward propagation in sequence based on the server-side model. The second participant sends the backward intermediate results of training process 0 to the first participant A and the first participant B respectively, and also sends the backward intermediate results of training process 1 to the first participant A and the first participant B respectively.

[0075] In the aforementioned backpropagation process on the server-side model based on the forward propagation results, loss can be calculated based on the forward propagation results and corresponding sample labels, and then backpropagation can be performed on the server-side model based on the calculated loss; for example... Figure 3 As shown, the positive intermediate results of training process 0 of the first participant A and the positive intermediate results of training process 0 of the first participant B are allocated to the training process 0 of the second participant. Forward propagation is performed based on the server-side model. Loss is calculated based on the forward propagation results and sample labels 0, and backpropagation is performed on the server-side model based on the calculated loss. Figure 4 The forward and backward propagation processes are illustrated. The segmentation layer output of client model A (training process 0 of the first participant A) is input to the server model (training process 0 of the second participant). Layer 4 of the server model processes the segmentation layer output, outputting y4. Layer 5 of the server model processes y4, outputting y5. Loss is calculated based on y5 and its corresponding sample labels. Backpropagation is then performed on the server model based on this calculated loss. Specifically, during backpropagation, the gradient data corresponding to y5 is first calculated, and then the gradient data of the model parameters of layer 5 of the server model is calculated based on the gradient data corresponding to y5. This gradient data may include the weights, biases, and other parameters of layer 5. The gradient data corresponding to y5 is used to calculate the gradient data corresponding to y4. The gradient data corresponding to y4 is then propagated backward to layer 4. The layer 4 of the server model calculates the gradient data corresponding to the weights, biases, and other parameters of layer 4 based on the gradient data corresponding to y4. Then, the gradient data corresponding to the output of the segmentation layer is calculated based on the gradient data corresponding to y4 as the reverse intermediate result. The reverse intermediate result is sent to the first participant A. The forward and backward propagation processes of training process 1 are similar to those of training process 0, and will not be described in detail here. In this way, by returning the gradient data corresponding to the output of the segmentation layer to the first participant, the data security between the first participant and the second participant is ensured.

[0076] During backpropagation, after calculating the gradient data, the process of synchronizing gradient data with other training processes and updating the server-side model parameters can be performed. This process involves calculating the gradient data of the server-side model parameters during backpropagation and then synchronizing the gradient data with other training processes to update the server-side model parameters. The gradient data of the server-side model parameters includes the gradient data of each layer's parameters. Figure 4 As shown, the gradient data of the server-side model parameters includes the gradient data corresponding to the weights, biases, and other parameters of the 5th layer and the gradient data corresponding to the weights, biases, and other parameters of the 4th layer.

[0077] In practical applications, after backpropagation is performed on each training process of the second participant to obtain gradient data, since the model training of each training process of the second participant is distributed, in order to facilitate continuing model training for at least one more round or to obtain the trained model after terminating model training, gradient data can be synchronized with other training processes, and the server-side model parameters can be updated based on the synchronization results, so that the server-side model achieves parameter synchronization in each round. In an optional implementation provided in this embodiment, the following operations are performed during the process of synchronizing gradient data with other training processes and updating server-side model parameters:

[0078] The gradient data of the server-side model parameters is synchronized with other training processes through inter-process communication algorithms, and the gradient data of each process is aggregated with the gradient data synchronized with other training processes.

[0079] The server-side model parameters are updated based on the aggregated gradient data obtained from the aggregation.

[0080] The inter-process communication algorithm can be the all-reduce algorithm (a communication and computation algorithm used in distributed deep learning).

[0081] Specifically, gradient data of the server-side model parameters can be synchronized (i.e. sent) to other training processes through inter-process communication algorithms, so that each training process has the gradient data of the server-side model parameters of each training process. The gradient data of the server-side model parameters of each training process are aggregated, and the server-side model parameters are updated based on the aggregated gradient data. Specifically, aggregation can be achieved by calculating the average value of the gradient data of the server-side model parameters of each training process, so the aggregated gradient data can be the average value of the gradient data of the server-side model parameters of each training process.

[0082] like Figure 4 As shown, the gradient data corresponding to the weights, biases, and other parameters of layers 4 and 5 are synchronized with other training processes through an inter-process communication algorithm. The gradient data corresponding to the weights, biases, and other parameters of layers 4 of each process are then aggregated with the gradient data corresponding to the weights, biases, and other parameters of layers 4 synchronized with other training processes. The parameters of layer 4 of each process's server model are updated based on the aggregated gradient data of layer 4 of layer 0. That is, the parameters of layer 4 of the server model of training process 0 are updated based on the aggregated gradient data of layer 4 of layer 0. Similarly, for layer 5, the gradient data corresponding to the weights, biases, and other parameters of layers 5 of each process are aggregated with the gradient data corresponding to the weights, biases, and other parameters of layers 5 synchronized with other training processes. The parameters of layer 5 of each process's server model are updated based on the aggregated gradient data of layer 5 of layer 5.

[0083] In step S206, each of the first participants receives the reverse intermediate results from the second participant and assigns them to the corresponding training process to continue backpropagation based on the client model. After calculating the gradient data during the backpropagation process, the gradient data is synchronized with other training processes and the client model parameters are updated.

[0084] The aforementioned second participant receives the forward intermediate results sent by different first participants and distributes them to the corresponding training processes. Based on the server-side model, forward propagation and backward propagation are performed sequentially. The backward intermediate results obtained in the backward propagation are sent to each first participant. Based on the gradient data calculated during the backward propagation, gradient data synchronization is performed with other training processes, and the server-side model parameters are updated. In this step, each first participant receives the backward intermediate results from the second participant, distributes the backward intermediate results to the corresponding training processes, performs backward propagation based on the client-side model, and synchronizes gradient data with other training processes based on the gradient data calculated during the backward propagation, and updates the client-side model parameters.

[0085] In this embodiment, the reverse intermediate result of the second participant can be the gradient data of the segmentation layer, that is, the gradient data corresponding to the output of the segmentation layer; such as Figure 4 As shown, the gradient data corresponding to the segmentation layer output of the 4th layer of the server-side model from the training process 0 of the second participant is the inverse intermediate result; the gradient data obtained by backpropagation based on the client-side model can be the gradient data of the client-side model parameters, such as... Figure 4 As shown, the gradient data obtained by backpropagation of client model A based on training process 0 of the first participant A are the gradient data corresponding to the weights, biases and other parameters of layers 1, 2 and 3. The gradient data obtained by backpropagation of client model A based on training process 1 of the first participant A are the gradient data corresponding to the weights, biases and other parameters of layers 1, 2 and 3.

[0086] In practice, after each first participant receives the reverse intermediate results sent by the second participant, they allocate the reverse intermediate results to the corresponding training process and continue backpropagation based on the client model to obtain gradient data; for example... Figure 4As shown, the first participant A distributes the inverse intermediate results of training process 0 sent by the second participant to training process 0 of the first participant A. Training process 0 of the first participant continues to perform backpropagation based on client model A. Specifically, in the third layer of client model A, the gradient data corresponding to the weights, biases and other parameters of the third layer are calculated based on the inverse intermediate results, and the gradient data corresponding to the output y2 of the second layer is calculated. The gradient data corresponding to y2 is propagated to the second layer to calculate the gradient data corresponding to the weights, biases and other parameters of the second layer, and the gradient data corresponding to y1 is calculated. The gradient data corresponding to y1 is propagated to the first layer to calculate the gradient data corresponding to the weights, biases and other parameters of the first layer. The inverse intermediate results of training process 1 sent by the second participant are distributed to training process 1 of the first participant A. The backpropagation process of training process 1 is similar to that of training process 0, and will not be described in detail here.

[0087] In specific execution, after the gradient data is calculated during the backpropagation process of each first participant's training process, gradient data synchronization can be performed between the training processes to advance the model training progress. This facilitates subsequent model parameter updates in each training process, unifying the client models across multiple training processes into a single set of model parameters, and ensuring timely access to the trained model after training termination. In one optional implementation of this embodiment, the following operations are performed during the process of synchronizing gradient data with other training processes and updating client model parameters:

[0088] The gradient data of the client model parameters is synchronized with the gradient data of other training processes through inter-process communication algorithms, and the gradient data of each process is aggregated with the gradient data synchronized with the gradient data of other training processes.

[0089] The client model parameters are updated based on the aggregated gradient data obtained from the aggregation.

[0090] The inter-process communication algorithm can also be the all-reduce algorithm.

[0091] Specifically, gradient data of client model parameters can be sent to other training processes through inter-process communication algorithms, so that each training process has gradient data of its own client model parameters. The gradient data of client model parameters of each training process are aggregated, and the client model parameters are updated based on the aggregated gradient data. Specifically, aggregation can be achieved by calculating the average value of the gradient data of client model parameters of each training process, so the aggregated gradient data can be the average value of the gradient data of client model parameters of each training process.

[0092] like Figure 4As shown, the gradient data corresponding to the weights, biases, and other parameters of the client model in layers 1, 2, and 3 are synchronized with other training processes through an inter-process communication algorithm. The gradient data corresponding to the weights, biases, and other parameters of the client model in each process are aggregated with the gradient data corresponding to the weights, biases, and other parameters of the client model in the same process as the synchronized gradient data of the client model in the same process. The parameters of the client model in layer 1 are updated based on the aggregated gradient data of layer 1 in training process 0. The same applies to other layers. The gradient data corresponding to the weights, biases, and other parameters of the client model in each process are aggregated with the gradient data corresponding to the weights, biases, and other parameters of the client model in the same process as the synchronized gradient data ....

[0093] In practical applications, to improve the flexibility of model training and allow for control over the training duration at any time, training termination conditions can be set during model training. In one optional implementation of this embodiment, the following operations can also be performed:

[0094] Check whether the updated server-side model triggers the training termination condition;

[0095] If triggered, the updated client model and the updated server model will be used as the model training results;

[0096] If not triggered, return to step S202.

[0097] Optionally, the training termination condition includes convergence of the updated server-side model.

[0098] It should be added that, if the updated server-side model triggers the training termination condition, either the updated client-side model or the updated server-side model can be used as the model training result, or the updated client-side model and the updated server-side model can be combined and the combined model can be used as the model training result.

[0099] It should also be noted that if the updated server-side model does not trigger the training termination condition, iterative training can continue according to the model training process provided above. Subsequent iterative training can be performed once or multiple times until the updated server-side model triggers the training termination condition.

[0100] It should be noted that each optional implementation method and each execution step in each optional implementation method provided in this embodiment can be arbitrarily referenced, combined, or separated and executed independently as needed. This embodiment does not specifically limit the execution order.

[0101] In this embodiment, the client-side model can be a machine learning model, and the server-side model can be a machine learning model. The client-side models and server-side models of each first participant can form a model to be trained. The model structure of the client-side models of each first participant can be the same; the model structure of the client-side models of each first participant can also be different. For example, client-side model A of first participant A is an image color processing model, client-side model B of first participant B is an image text processing model, and server-side model of second participant is an image quality detection model. Another example is that client-side model A of first participant A is a career recognition model that identifies users' careers through user career images, client-side model B of first participant B is an extraction model that extracts key behavioral data from user resource behavior data, and server-side model of second participant is a risk detection model that detects whether a user is a risky user.

[0102] It should also be noted that the examples of each first participant and second participant in this embodiment are merely illustrative, and the specific number of each first participant can be determined according to the actual application scenario.

[0103] In summary, this embodiment provides one or more distributed model training methods applicable to multiple participants running multiple training processes, where the training processes of different participants correspond to each other. The first participant among the participants has a client model, and the second participant among the participants has a server model. First, each first participant allocates sample data blocks to different training processes and performs forward propagation based on the client model, sending the forward intermediate results obtained from the forward propagation of different training processes to the second participant. Entity alignment is achieved between the sample data blocks allocated to the corresponding training processes by different first participants. This entity alignment synchronizes the training progress between the corresponding training processes of different first participants, avoiding training errors caused by entity misalignment and improving the effectiveness of model training. Simultaneously, by introducing multiple training processes in each first participant, the multiple training processes can process the corresponding sample data blocks in a distributed manner, increasing the processing capacity of sample data blocks and improving model training efficiency.

[0104] Secondly, the second participant receives the forward intermediate results sent by different first participants and allocates them to the corresponding training processes. Based on the server-side model, forward and backward propagation are performed sequentially. The backward intermediate results obtained in the backward propagation are sent to each first participant. After calculating the gradient data during the backward propagation process, the gradient data of the server-side model parameters is synchronized with other training processes through inter-process communication algorithms. The gradient data of each participant is aggregated with the gradient data synchronized with other training processes. The server-side model parameters are updated based on the aggregated gradient data. This facilitates the subsequent model training for at least one more round or the termination of model training to obtain the trained model. The aggregation efficiency of gradient data is improved through inter-process communication algorithms.

[0105] Finally, each first participant receives the reverse intermediate results from the second participant and is assigned to the corresponding training process to continue backpropagation based on the client model. After calculating the gradient data during backpropagation, the gradient data of the client model parameters is synchronized with other training processes through inter-process communication algorithms. The gradient data of each participant is aggregated with the gradient data synchronized with other training processes, and the client model parameters are updated based on the aggregated gradient data. This process synchronizes gradient data among the training processes, which facilitates subsequent updates of model parameters in each training process. It unifies the client models of multiple training processes into a single model parameter, and obtains the trained model in a timely manner after training is terminated. Furthermore, the multiple training processes enable distributed training on multiple GPUs (Graphics Processing Units) on multiple machines or a single machine, making full use of cluster resources and improving the processing capacity of sample data and model training efficiency.

[0106] The following description uses the application of a distributed model training method provided in this embodiment in a distributed training scenario as an example to further illustrate the distributed model training method provided in this embodiment. (See also...) Figure 5 This is a distributed model training method applied to distributed training scenarios. It can be applied to various clients and servers running multiple training processes, and specifically includes the following steps.

[0107] In step S502, each client allocates sample data blocks to different training processes, performs forward propagation based on the client model, and sends the output results of the segmentation layer obtained from the forward propagation of different training processes to the server.

[0108] Among them, the multiple training processes of each client correspond to each other and also correspond to the multiple training processes of the server. Each client has a client model and the server has a server model. Optionally, entity alignment is performed between the sample data blocks allocated to the corresponding training processes of different clients.

[0109] In step S504, the server receives the output results of the segmentation layer sent by different clients and allocates them to the corresponding training process. The server performs forward propagation based on the model, calculates the loss based on the forward propagation results and the corresponding sample labels, performs backpropagation based on the calculated loss in the model, and sends the segmentation layer gradient data obtained in the backpropagation to each client.

[0110] In step S506, after the corresponding training process on the server calculates the gradient data of the server model parameters during the backpropagation process based on the server model, it synchronizes the gradient data of the server model parameters with other training processes through an inter-process communication algorithm, and aggregates its own gradient data with the gradient data synchronized with the other training processes.

[0111] In step S508, the corresponding training process on the server updates the server-side model parameters based on the aggregated gradient data obtained from the aggregation.

[0112] In step S510, each client receives the segmentation layer gradient data from the server and assigns it to the corresponding training process to continue backpropagation based on the client model. After calculating the gradient data of the client model parameters during the backpropagation process, the client model parameter gradient data is synchronized with other training processes through an inter-process communication algorithm, and the gradient data of each client is aggregated with the gradient data synchronized with other training processes.

[0113] In step S512, the corresponding training process of each client updates the client model parameters based on the aggregated gradient data obtained from the aggregation.

[0114] After this, it can be checked whether the updated server model triggers the training termination condition. If it does, the updated client model and the updated server model are used as the model training results. If it does not trigger, return to step S502 to continue iterative training until the updated server model triggers the training termination condition, and the updated client model and the updated server model are used as the model training results.

[0115] It should be noted that any one or any combination of steps S502 to S512 can be replaced by the corresponding technical means provided in steps S202 to S206 above, depending on the needs of implementation and deployment. These will not be elaborated here.

[0116] One or more embodiments of the second distributed model training method provided in this specification are as follows:

[0117] Reference Figure 6The distributed model training method provided in this embodiment can be applied to a first participant running multiple training processes. The multiple training processes of each first participant correspond to each other and also correspond to the multiple training processes of a second participant. Each first participant has a client model and the second participant has a server model. Specifically, it includes steps S602 to S604.

[0118] Step S602: Distribute the sample data blocks to different training processes for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of different training processes to the second participant.

[0119] In practical applications, when training a model through split learning, the amount of sample data may be large. If multiple participants use a single process for training, the single process may be limited by resources, resulting in slow computation speed and thus slow training speed. This embodiment introduces multiple training processes based on split learning, enabling each participant to perform distributed model training through multiple training processes, thereby improving model training efficiency.

[0120] In this embodiment, each first participant includes a client, and each first participant is independent. The number of first participants can be arbitrary and is not specifically limited here. The second participant includes a server. In this embodiment, each first participant and second participant refers to multiple participants that jointly train the model through split learning.

[0121] Each participant can run multiple training processes, and there are corresponding training processes between different participants. In this embodiment, the training process refers to the training process run by each participant to train the model. These training processes are carried out in parallel in a distributed manner. For example, the first participant A runs training process 0 and training process 1, the first participant B runs training process 0 and training process 1, and the second participant runs training process 0 and training process 1.

[0122] Optionally, the number of training processes run by each first participant is equal, and the number of training processes run by each first participant is equal to the number of training processes run by the second participant; the training processes of each first participant correspond to each other, and also correspond to the training processes of the second participant; each training process in the training processes of each first participant has its own client model, and each training process in the training processes of the second participant has its own server model.

[0123] like Figure 3As shown, each first participant includes first participant A and first participant B. The training process of first participant A includes training process 0 and training process 1, the training process of first participant B includes training process 0 and training process 1, and the training process of the second participant also includes training process 0 and training process 1. The training process 0 of first participant A, the training process 0 of first participant B, and the training process 0 of second participant A correspond to each other, as do the training process 1 of first participant A, the training process 1 of first participant B, and the training process 1 of second participant B. The training process 0 of first participant A may have a client model A, the training process 1 of first participant A may have a client model A, and the client model A of the training process 0 of first participant A is related to the training process. The client model A of the first participant can have the same model structure, and the model parameters during the model training process can be the same or different. The first participant B is similar. The training process 0 and training process 1 of the first participant B each have their own client model B. The model structure of the client model B of the training process 0 and training process 1 of the first participant B can be the same, and the model parameters during the model training process can be the same or different. The model structure of client model A and client model B can be the same or different. The training process 0 and training process 1 of the second participant can each have a server model. The model structure of the server model of the training process 0 and the server model of the training process 1 of the second participant can be the same, but the model parameters during the model training process can be the same or different.

[0124] The sample data block mentioned in this embodiment refers to the data block read from the data shards of each first participant. The data shards of each first participant refer to the data units obtained after performing relevant data processing on the training data of each first participant. The number of data shards of each first participant can be equal to the number of training processes. Multiple training processes of each first participant can correspond to each other with data shards. For example, data shards can be represented by shards. If the training processes of first participant A are training process 0 and training process 1, then first participant A has shard0 and shard1. Optionally, entity alignment can be performed between sample data blocks allocated to the corresponding training processes by different first participants; for example, entity alignment can be performed between sample data blocks allocated to training process 0 by first participant A and sample data blocks allocated to training process 0 by first participant B, and entity alignment can be performed between sample data blocks allocated to training process 1 by first participant A and sample data blocks allocated to training process 1 by first participant B; entity alignment includes the same entity identifier in the sample data blocks, and the entity identifier can be a user identifier. For example, if the entity identifier in the sample data block allocated to training process 0 by first participant A is r1, then the entity identifier in the sample data block allocated to training process 0 by first participant B is also r1.

[0125] In practical applications, the model training process usually involves many rounds, and the amount of data in the data slices may also be large. Therefore, in order to improve the data processing speed of the model in each round and avoid the large amount of data processing causing the model to be too slow and affecting the training efficiency, the sample data block can be read from the data slices of each of the first participants.

[0126] The data shards of each of the first participating parties can be obtained in the following way:

[0127] The training data of each first participant is subjected to entity alignment processing, and the obtained aligned data is split into multiple data fragments of each first participant; optionally, the number of data fragments obtained by data splitting is equal to the number of training processes.

[0128] The sample data blocks can be read from the data slices of each of the first participants according to a preset number. The preset number can be a pre-set number, such as m. Then, sample data blocks can be read from the data slices of each of the first participants according to the preset number m. For example, first participant A reads sample data block 0 of the current training round from data slice 0 according to the preset number m, and reads sample data block 1 of the current training round from data slice 1 according to the preset number m. The above data segmentation can be performed by the data segmentation module of each first participant. After the data segmentation module of each first participant performs data segmentation to obtain multiple data slices, the multiple data slices can be stored. The data segmentation module here can be a DataSplitCoordinator.

[0129] Based on the above-mentioned data segmentation of the obtained aligned data to obtain multiple data fragments for each first participant, optionally, the multiple data fragments of each first participant are allocated to multiple training processes of each first participant, and there is a one-to-one correspondence between the allocated data fragments and training processes; specifically, each first participant can read multiple data fragments from its own data segmentation module and allocate the multiple data fragments to multiple training processes of each first participant; for example, first participant A reads data fragment 0 and data fragment 1 from its data segmentation module, allocates data fragment 0 to training process 0 of first participant A, and allocates data fragment 1 to training process 1 of first participant A, that is, data fragment 0 of first participant A corresponds to training process 0 and data fragment 1 corresponds to training process 1, and the same applies to first participant B.

[0130] In practical implementation, various first participants exist during model training. To enable synchronous collaboration among these first participants during model training, entity alignment processing can be performed on the training data of each first participant. This helps each first participant read the aligned training data for model training, improving the accuracy and efficiency of model training. Based on the aforementioned entity alignment processing of the training data of each first participant, and the subsequent data segmentation of the aligned data to obtain multiple data fragments for each first participant, the above-mentioned entity alignment processing of the training data of each first participant can be implemented in the following way:

[0131] The intersection of the training data of each first participant is calculated based on the entity identifiers in the training data to obtain the intersection data of each first participant.

[0132] The intersection data of each of the first participants is aligned according to the entity identifiers in the intersection data.

[0133] Wherein, the training data of each first participant refers to the initial training data of each first participant; the entity identifier in the training data refers to the identifier of the entity object in the training data. Since there may be multiple training data, the entity identifier in the training data can be used to represent which training data in the training data. The entity identifier can be a user identifier. For example, the training data of first participant A includes user resource behavior data stored by institution a, and the user identifiers, i.e., entity identifiers, in each user resource behavior data are r1, r2, r3, r4, ... That is, the user resource behavior data includes user resource behavior data of user r1, user resource behavior data of user r2, and user resource behavior data of user r3. The training data of first participant B includes user occupation data stored by institution b, which may specifically include user occupation data of user r2, user occupation data of user r1, and user occupation data of user r3. The entity identifiers in the training data of first participant B are r2, r1, and r3.

[0134] The intersection calculation includes selecting intersection data with the same entity identifiers from the training data of each first participant based on the entity identifiers in the training data; that is, the entity identifiers in the intersection data of each first participant are the same, but the data content of the intersection data may differ; for example, based on the entity identifiers in the training data of first participant A and the entity identifiers in the training data of first participant B, the intersection data of first participant A is calculated as user resource behavior data corresponding to r1, r2, and r3, and the intersection data of first participant B is user occupation data corresponding to r2, r1, and r3. That is, the entity identifiers in the intersection data of first participant A and first participant B are the same, but the data content of the intersection data is different; in this way, the data of each first participant is aligned through intersection calculation, avoiding invalid model training caused by misaligned data.

[0135] Specifically, based on the entity identifiers in the training data of each first participant, training data with the same entity identifiers can be selected as the intersection data of each first participant. To improve the convenience of data reading in each round during model training, the intersection data of each first participant can be further sorted according to the entity identifiers in the intersection data to obtain the aligned data of each first participant. Further data alignment through sorting helps to improve the synchronization and frequency of subsequent model training.

[0136] Continuing with the previous example, the intersection data of the first participant A consists of user resource behavior data corresponding to r1, r2, and r3. The intersection data of the first participant B consists of user occupation data corresponding to r2, r1, and r3. The intersection data of the first participant A and the first participant B are sorted according to the entity identifiers in the intersection data. The aligned data of the first participant A is then obtained as user resource behavior data corresponding to r1, r2, and r3. The data, the alignment data of the first participant B, is the user occupation data corresponding to r1, r2, and r3, realizing the alignment of the first participant A and the first participant B in the entity dimension and the order dimension. Here, there is no specific limitation on the sorting method. It can be from large to small, from small to large, or the intersection data of one first participant can be used as a benchmark to sort the intersection data of another first participant, so that the entity identifiers in the intersection data of the other first participant are aligned with the entity identifiers in the intersection data of one first participant.

[0137] After performing entity alignment processing on the training data of each first participant, the obtained aligned data is split into multiple data slices for each first participant. This allows multiple training processes of each first participant to have their own data slices and perform model training synchronously. In the process of splitting the obtained aligned data into multiple data slices for each first participant, the data can be split into multiple data slices for each first participant based on the number of data in the aligned data of each first participant and the number of training processes running by each first participant. The number of data in the aligned data includes the number of entity identifiers in the aligned data of each first participant. For example, if the entity identifiers include r1, r2, and r3, then the number of data in the aligned data is 3.

[0138] Specifically, the ratio of the number of data points in the alignment data of each first participant to the number of processes in the training process can be calculated as the data interval identifier for each first participant to perform data segmentation. The alignment data of each first participant is then segmented according to the data interval identifier to obtain multiple data fragments for each first participant. Here, the data interval identifier can represent the segmentation range for data segmentation.

[0139] For example, if the number of training processes for first participant A and first participant B is 2, and the number of data points in first participant A's alignment data is 1000, then the data interval identifiers for first participant A's data segmentation are 0 and 500, and 500 and 1000. Data segmentation is performed on first participant A's alignment data according to these interval identifiers, resulting in data segment 0 and data segment 1. Data segment 0 corresponds to the alignment data within the interval [0, 500) in first participant A's alignment data, and data segment 1 corresponds to the alignment data within the interval [500, 1000) in first participant A's alignment data. If the number of data points in first participant B's alignment data is 1000, then first participant B performs data segmentation... The data intervals are labeled as 0 and 500, and 500 and 1000. The aligned data of the first participant B is segmented according to the data interval labels to obtain data segment 0 and data segment 1 of the first participant B. Data segment 0 of the first participant B corresponds to the aligned data in the interval [0, 500) of the aligned data of the first participant B, and data segment 1 of the first participant B corresponds to the aligned data in the interval [500, 1000) of the aligned data of the first participant B. In this way, the data segmentation enables multiple training processes of each first participant to have their own data segments, and each data segment is aligned, so that the model training is performed synchronously among multiple training processes, reducing the probability of model training errors.

[0140] In practical applications, entity alignment can be achieved among the various first-party participants. However, when the first-party participants do not carry sample labels, and the sample labels are carried by the second-party participants, the second-party participants also have a need to align the sample labels with the entity labels of the first-party participants. That is, entity alignment is required for the data from all parties, including the first and second-party participants. Therefore, entity alignment can be performed between the training data of the first-party participants and the initial sample labels of the second-party participants. This improves the accuracy and effectiveness of backpropagation, ensuring that the forward propagation results of the second-party participant's server-side model match the sample labels, thus improving the accuracy of backpropagation. The alignment process can be performed as follows:

[0141] Entity alignment is performed on the training data of each first participant and the initial sample labels of the second participant.

[0142] The obtained aligned data of each first participant and the aligned sample labels of the second participant are split into multiple data fragments of each first participant and sample labels of the second participant.

[0143] It should be noted that the entity alignment and data segmentation process here is similar to the specific implementation process of entity alignment and data segmentation of the training data of each first participant described above. The difference is that the above refers to each first participant, while here there are second participants in addition to each first participant. However, the execution process is similar and can be referred to. This embodiment will not be described in detail here.

[0144] like Figure 3 As shown, the first participant A has data shard 0 and data shard 1, the first participant B has data shard 0 and data shard 1, and the second participant has sample label 0 and sample label 1. Data shard 0 of the first participant A and data shard 0 of the first participant B correspond to sample label 0 of the second participant, and data shard 1 of the first participant A and data shard 1 of the first participant B correspond to sample label 1 of the second participant.

[0145] It should be added that, in this embodiment, the client model set by the first participant and the server model set by the second participant can be obtained by decomposing the model to be trained. In this way, training of the model to be trained with a more complex model structure can be achieved by learning by decomposition, i.e., decomposing the model structure, thereby improving the convenience and efficiency of model training.

[0146] It should also be noted that the client model in this embodiment can be any application domain, any type, and any model structure, and the server model can also be any application domain, any type, and any model structure. This embodiment does not impose specific limitations on the model structure or application scenario of the client model and the server model.

[0147] In the specific execution process, each first participant can allocate sample data blocks to the corresponding training process for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of each training process to the second participant; here, forward propagation can be data processing of the input sample data blocks; for example, if the input is image data, forward propagation can be image feature extraction; such as... Figure 3 As shown, the first participant A reads sample data block 0 from data shard 0 of the first participant A, allocates the read sample data block 0 to the training process 0 of the first participant A for forward propagation based on client model A, reads sample data block 1 from data shard 1 of the first participant A, allocates the read sample data block 1 to the training process 1 of the first participant A for forward propagation based on client model A, and sends the forward intermediate results of training process 0 and training process 1 to the second participant; the processing of training process 0 and training process 1 of the first participant B is similar to that of the first participant A, and will not be described again.

[0148] It should be noted that in this embodiment, the last layer of each client model can be a segmentation layer, that is, the boundary layer between the client model and the server model; for example... Figure 3 As shown, the last layer of each client model is a segmentation layer. Each first participant allocates sample data blocks to different training processes and performs forward propagation based on the client model. The forward intermediate results of the segmentation layer output obtained from the forward propagation of different training processes are sent to the second participant; for example... Figure 4 The forward propagation process shown is as follows: In the client model A of training process 0 of the first participant A, layer 1 of client model A processes sample data block 0 and outputs y1; layer 2 of client model A processes y1 and outputs y2; layer 3 (segmentation layer) of client model A processes y2 to obtain the segmentation layer output. The processing process of client model A in training process 1 of the first participant A is similar to that in training process 0, and will not be repeated here. The segmentation layer output of client model A is sent to the second participant as a forward intermediate result. In this way, the first participant and the second participant are decoupled through the segmentation layer. By sending only the segmentation layer output to the second participant, data isolation is achieved between the first participant and the second participant, ensuring data security between the participants.

[0149] Step S604: Receive the reverse intermediate result sent by the second participant and assign it to the corresponding training process for backpropagation based on the client model. After calculating the gradient data during the backpropagation process, synchronize the gradient data with other training processes and update the client model parameters.

[0150] Each of the aforementioned first participants allocates sample data blocks to different training processes for forward propagation based on the client model, and sends the forward intermediate results obtained from the forward propagation of different training processes to the second participant. Upon receiving the forward intermediate results from the different first participants, the second participant can allocate the forward intermediate results to the corresponding training processes and perform forward and backward propagation sequentially based on the server model. The backward intermediate results obtained during backward propagation are returned to each of the first participants. Furthermore, the gradient data obtained during backward propagation is synchronized with the gradient data of other training processes, and the server model parameters are updated. In this step, each first participant receives the backward intermediate results sent by the second participant, allocates them to the corresponding training processes, and performs backward propagation based on the client model. During the backward propagation process, the gradient data is calculated and then synchronized with the gradient data of other training processes, and the client model parameters are updated. Optionally, the backward intermediate results are obtained by the second participant after allocating the forward intermediate results sent by the different first participants to the corresponding training processes and performing forward and backward propagation sequentially based on the server model.

[0151] In this embodiment, the reverse intermediate result of the second participant can be the gradient data of the segmentation layer, that is, the gradient data corresponding to the output of the segmentation layer (the forward intermediate result); for example Figure 4 As shown, the gradient data corresponding to the segmentation layer output of the 4th layer of the server-side model from the training process 0 of the second participant is the inverse intermediate result; the gradient data obtained by backpropagation based on the client-side model can be the gradient data of the client-side model parameters, such as... Figure 4 As shown, the gradient data obtained by backpropagation of client model A based on training process 0 of the first participant A are the gradient data corresponding to the weights, biases and other parameters of layers 1, 2 and 3. The gradient data obtained by backpropagation of client model A based on training process 1 of the first participant A are the gradient data corresponding to the weights, biases and other parameters of layers 1, 2 and 3.

[0152] In practice, after receiving the forward intermediate results sent by different first participants, the second participant can allocate the forward intermediate results to the corresponding training process and perform forward propagation based on the server-side model. Based on the forward propagation results, it can perform back propagation on the server-side model and send the reverse intermediate results obtained in the back propagation to each first participant. Furthermore, after calculating the gradient data during the back propagation process, it can synchronize the gradient data with other training processes and update the parameters of its respective server-side model based on the synchronization results.

[0153] The reverse intermediate result can be the gradient data corresponding to the segmentation layer output (i.e., the forward intermediate result). The gradient data calculated during the backpropagation process can be the gradient data of the server-side model parameters, including the gradient data corresponding to the weights, biases, and other parameters of the server-side model.

[0154] like Figure 3 As shown, the second participant receives the forward intermediate results sent by each of the first participants. The second participant distributes the forward intermediate results of training process 0 of the first participant A and the forward intermediate results of training process 0 of the first participant B to the training process 0 of the second participant, and performs forward propagation and backward propagation in sequence based on the server-side model. The second participant distributes the forward intermediate results of training process 1 of the first participant A and the forward intermediate results of training process 1 of the first participant B to the training process 1 of the second participant, and performs forward propagation and backward propagation in sequence based on the server-side model. The second participant sends the backward intermediate results of training process 0 to the first participant A and the first participant B respectively, and also sends the backward intermediate results of training process 1 to the first participant A and the first participant B respectively.

[0155] During the backpropagation process of the second participant in the server-side model based on the forward propagation results, loss can be calculated based on the forward propagation results and the corresponding sample labels, and backpropagation can be performed on the server-side model based on the calculated loss; for example... Figure 3 As shown, the positive intermediate results of training process 0 of the first participant A and the positive intermediate results of training process 0 of the first participant B are allocated to the training process 0 of the second participant. Forward propagation is performed based on the server-side model. Loss is calculated based on the forward propagation results and sample labels 0, and backpropagation is performed on the server-side model based on the calculated loss. Figure 4The forward and backward propagation processes are illustrated. The segmentation layer output of client model A (training process 0 of the first participant A) is input to the server model (training process 0 of the second participant). Layer 4 of the server model processes the segmentation layer output, outputting y4. Layer 5 of the server model processes y4, outputting y5. Loss is calculated based on y5 and its corresponding sample labels. Backpropagation is then performed on the server model based on this calculated loss. Specifically, during backpropagation, the gradient data corresponding to y5 is first calculated, and then the gradient data of the model parameters of layer 5 of the server model is calculated based on the gradient corresponding to y5. This gradient may include the weights, biases, and other parameters of layer 5. The gradient data of y5 is used to calculate the gradient data of y4. The gradient data of y4 is then propagated backward to layer 4. The layer 4 of the server model calculates the gradient data of the weights, biases and other parameters of layer 4 based on the gradient data of y4. Then, the gradient data of the segmentation layer is calculated based on the gradient data of y4 as the reverse intermediate result. The reverse intermediate result is sent to the first participant A. The forward and backward propagation processes of training process 1 are similar to those of training process 0, and will not be described in detail here. In this way, by returning the gradient data of the segmentation layer to the first participant, the data security between the first participant and the second participant is ensured.

[0156] During backpropagation, after calculating the gradient data, the process of synchronizing gradient data with other training processes and updating the server-side model parameters can be performed. This process involves calculating the gradient data of the server-side model parameters during backpropagation and then synchronizing the gradient data with other training processes to update the server-side model parameters. The gradient data of the server-side model parameters includes the gradient data of each layer's parameters. Figure 4 As shown, the gradient data of the server-side model parameters includes the gradient data corresponding to the weights, biases, and other parameters of the 5th layer and the gradient data corresponding to the weights, biases, and other parameters of the 4th layer.

[0157] In practical applications, after backpropagation is performed on each training process of the second participant to obtain gradient data, since the model training of the various training processes of the second participant is distributed, in order to facilitate continuing model training for at least one more round or to obtain the trained model after terminating model training, gradient data can be synchronized with other training processes, and the server-side model parameters can be updated based on the synchronization results, so that the server-side model achieves parameter synchronization in each round. During the process of synchronizing gradient data with other training processes and updating server-side model parameters, the corresponding training process of the second participant can perform the following operations:

[0158] The gradient data of the server-side model parameters is synchronized with other training processes through inter-process communication algorithms, and the gradient data of each process is aggregated with the gradient data synchronized with other training processes.

[0159] The server-side model parameters are updated based on the aggregated gradient data obtained from the aggregation.

[0160] The inter-process communication algorithm can be the all-reduce algorithm (a communication and computation algorithm used in distributed deep learning).

[0161] Specifically, gradient data of server-side model parameters can be sent to other training processes through inter-process communication algorithms, so that each training process has gradient data of its own server-side model parameters. The gradient data of server-side model parameters of each training process are aggregated, and the server-side model parameters are updated based on the aggregated gradient data. Specifically, aggregation can be achieved by calculating the average value of the gradient data of server-side model parameters of each training process, so the aggregated gradient data can be the average value of the gradient data of server-side model parameters of each training process.

[0162] like Figure 4 As shown, the gradient data corresponding to the weights, biases, and other parameters of layers 4 and 5 are synchronized with other training processes through an inter-process communication algorithm. The gradient data corresponding to the weights, biases, and other parameters of layers 4 of each process are then aggregated with the gradient data corresponding to the weights, biases, and other parameters of layers 4 synchronized with other training processes. The parameters of layer 4 of each process's server model are updated based on the aggregated gradient data of layer 4. That is, the parameters of layer 4 of the server model of training process 0 are updated based on the aggregated gradient data of layer 4 of layer 0. The same applies to layer 5. The gradient data corresponding to the weights, biases, and other parameters of layers 5 of each process are aggregated with the gradient data corresponding to the weights, biases, and other parameters of layers 5 synchronized with other training processes. The parameters of layer 5 of each process's server model are updated based on the aggregated gradient data of layer 5.

[0163] In practice, after each first participant receives the reverse intermediate result sent by the second participant, it can allocate the reverse intermediate result to the corresponding training process to continue backpropagation based on the client model, and obtain gradient data based on backpropagation; for example... Figure 4As shown, the first participant A distributes the inverse intermediate results of training process 0 sent by the second participant to training process 0 of the first participant A. Training process 0 of the first participant continues to perform backpropagation based on client model A. Specifically, in the third layer of client model A, the gradient data corresponding to the weights, biases and other parameters of the third layer are calculated based on the inverse intermediate results, and the gradient data corresponding to the output y2 of the second layer is calculated. The gradient data corresponding to y2 is propagated to the second layer to calculate the gradient data corresponding to the weights, biases and other parameters of the second layer, and the gradient data corresponding to y1 is calculated. The gradient data corresponding to y1 is propagated to the first layer to calculate the gradient data corresponding to the weights, biases and other parameters of the first layer. The inverse intermediate results of training process 1 sent by the second participant are distributed to training process 1 of the first participant A. The backpropagation process of training process 1 is similar to that of training process 0, and will not be described in detail here.

[0164] In the specific execution process, after the training processes of each first participant have calculated the gradient data during backpropagation, in order to advance the model training progress, gradient data synchronization can be performed between the training processes. This facilitates subsequent model parameter updates in each training process, unifying the client models across multiple training processes into a single set of model parameters, and ensuring timely access to the trained model after training terminates. During the process of synchronizing gradient data with other training processes and updating client model parameters, each first participant's training process can perform the following operations:

[0165] The gradient data of the client model parameters is synchronized with the gradient data of other training processes through inter-process communication algorithms, and the gradient data of each process is aggregated with the gradient data synchronized with the gradient data of other training processes.

[0166] The client model parameters are updated based on the aggregated gradient data obtained from the aggregation.

[0167] The inter-process communication algorithm can also be the all-reduce algorithm.

[0168] Specifically, gradient data of client model parameters can be sent to other training processes through inter-process communication algorithms, so that each training process has gradient data of its own client model parameters. The gradient data of client model parameters of each training process are aggregated, and the client model parameters are updated based on the aggregated gradient data. Specifically, aggregation can be achieved by calculating the average value of the gradient data of client model parameters of each training process, so the aggregated gradient data can be the average value of the gradient data of client model parameters of each training process.

[0169] like Figure 4As shown, the gradient data corresponding to the weights, biases, and other parameters of the client model in layers 1, 2, and 3 are synchronized with other training processes through an inter-process communication algorithm. The gradient data corresponding to the weights, biases, and other parameters of the client model in each process are aggregated with the gradient data corresponding to the weights, biases, and other parameters of the client model in the same process as the synchronized gradient data of the client model in the same process. The parameters of the client model in layer 1 are updated based on the aggregated gradient data of layer 1 in training process 0. The same applies to other layers. The gradient data corresponding to the weights, biases, and other parameters of the client model in each process are aggregated with the gradient data corresponding to the weights, biases, and other parameters of the client model in the same process as the synchronized gradient data ...

[0170] It should be noted that each execution operation process and each execution step provided in this embodiment can be arbitrarily referenced, combined, or separated for independent execution as needed. This embodiment does not impose specific limitations on the execution order.

[0171] In this embodiment, the client-side model can be a machine learning model, and the server-side model can be a machine learning model. The client-side models and server-side models of each first participant can form a model to be trained. The model structure of the client-side models of each first participant can be the same; the model structure of the client-side models of each first participant can also be different. For example, client-side model A of first participant A is an image color processing model, client-side model B of first participant B is an image text processing model, and server-side model of second participant is an image quality detection model. Another example is that client-side model A of first participant A is a career recognition model that identifies users' careers through user career images, client-side model B of first participant B is an extraction model that extracts key behavioral data from user resource behavior data, and server-side model of second participant is a risk detection model that detects whether a user is a risky user.

[0172] It should also be noted that the examples of each first participant and second participant in this embodiment are merely illustrative, and the specific number of each first participant can be determined according to the actual application scenario.

[0173] The implementation process of the second distributed model training method provided above can be executed by the first participant. The implementation process of the third distributed model training method provided in the following method embodiments can be executed by the second participant. The two can cooperate with each other during the execution process. Therefore, when reading the above implementation process, you can refer to the corresponding content of the following third distributed model training method embodiment. Similarly, when reading the following third distributed model training method embodiment, you can also refer to the corresponding content of the above method embodiments.

[0174] The following description uses the application of a distributed model training method provided in this embodiment in a distributed training scenario as an example to further illustrate the distributed model training method provided in this embodiment. (See also...) Figure 7 This is a distributed model training method applied to distributed training scenarios. It can be applied to various clients running multiple training processes and includes the following steps.

[0175] Step S702: Distribute the sample data blocks to different training processes for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of different training processes to the server.

[0176] Among them, the multiple training processes of each client correspond to each other and also correspond to the multiple training processes of the server. Each client has a client model and the server has a server model. Optionally, entity alignment is performed between the sample data blocks allocated to the corresponding training processes of different clients.

[0177] Step S710: Receive the reverse intermediate results sent by the server and assign them to the corresponding training process for backpropagation based on the client model. After calculating the gradient data during the backpropagation process, synchronize the gradient data with other training processes and update the client model parameters.

[0178] It should be noted that any one or any combination of steps S702 and S710 can be replaced by the corresponding technical means provided in steps S602 to S604 above, depending on the needs of implementation and deployment. These will not be elaborated here.

[0179] In this embodiment, steps S702 and S710 are executed by each client. It should be noted that the execution of steps S702 and S710 by each client can cooperate with steps S704 to S708 executed by the server in the following embodiment. Therefore, when reading this embodiment, please refer to the corresponding content of steps S704 to S708 provided in the following method embodiment, and when reading the following method embodiment, please refer to the corresponding content of steps S702 and S710 provided in this embodiment.

[0180] One or more embodiments of the third distributed model training method provided in this specification are as follows:

[0181] Reference Figure 8 The distributed model training method provided in this embodiment can be applied to a second participant running multiple training processes. The multiple training processes of the second participant correspond to the multiple training processes of each first participant. Each first participant has a client model, and the second participant has a server model. Specifically, it includes steps S802 to S806.

[0182] Step S802: Receive the positive intermediate results sent by each of the first participating parties.

[0183] In practical applications, when training a model through split learning, the amount of sample data may be large. If multiple participants use a single process for training, the single process may be limited by resources, resulting in slow computation speed and thus slow training speed. This embodiment introduces multiple training processes based on split learning, enabling each participant to perform distributed model training through multiple training processes, thereby improving model training efficiency.

[0184] In this embodiment, each first participant includes a client, and each first participant is independent. The number of first participants can be arbitrary and is not specifically limited here. The second participant includes a server. Each first participant and each second participant refers to multiple participants that jointly train the model through split learning.

[0185] Each participant can run multiple training processes, and there are corresponding training processes between different participants. In this embodiment, the training process refers to the training process run by each participant to train the model. These training processes are carried out in parallel in a distributed manner. For example, the first participant A runs training process 0 and training process 1, the first participant B runs training process 0 and training process 1, and the second participant runs training process 0 and training process 1.

[0186] Optionally, the positive intermediate result is obtained by each first participant allocating sample data blocks to different training processes and performing forward propagation based on the client model, with entity alignment between sample data blocks allocated to the corresponding training processes by different first participants.

[0187] In practice, each first participant can allocate sample data blocks to different training processes for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of different training processes to the second participant; the second participant receives the forward intermediate results sent by each first participant.

[0188] Step S804: The positive intermediate results are assigned to the corresponding training processes and forward and backward propagation is performed sequentially based on the server-side model.

[0189] The second participant receives the forward intermediate results sent by each first participant. In this step, the second participant distributes the forward intermediate results to the corresponding training process and performs forward and backward propagation based on the server-side model.

[0190] In practice, after receiving the forward intermediate results sent by each of the first participants, the second participant can allocate the forward intermediate results to the corresponding training process for forward propagation based on the server-side model, and then perform backward propagation based on the forward propagation results to obtain the backward intermediate results and gradient data.

[0191] Step S806: Send the reverse intermediate results obtained in the backpropagation to each of the first participants, and after calculating the gradient data in the backpropagation process, synchronize the gradient data with other training processes and update the server-side model parameters.

[0192] The reverse intermediate result described in this embodiment can be the gradient data of the segmentation layer, that is, the gradient data corresponding to the output of the segmentation layer (forward intermediate result); such as Figure 4 As shown, the gradient data corresponding to the segmentation layer output of the 4th layer of the server-side model from the training process 0 of the second participant is the inverse intermediate result; the gradient data obtained by backpropagation based on the client-side model can be the gradient data of the client-side model parameters, such as... Figure 4 As shown, the gradient data obtained by backpropagation of client model A based on training process 0 of the first participant A are the gradient data corresponding to the weights, biases and other parameters of layers 1, 2 and 3. The gradient data obtained by backpropagation of client model A based on training process 1 of the first participant A are the gradient data corresponding to the weights, biases and other parameters of layers 1, 2 and 3.

[0193] In practice, the second participant sends the intermediate backpropagation results obtained during backpropagation to each of the first participants. After calculating the gradient data during backpropagation, it synchronizes the gradient data with other training processes and updates its respective server-side model parameters based on the synchronization results. The intermediate backpropagation results can be the gradient data corresponding to the segmentation layer output. The gradient data calculated during backpropagation can be the gradient data of the server-side model parameters, including the gradient data corresponding to the server-side model's weights, biases, and other parameters.

[0194] like Figure 3As shown, the second participant receives the forward intermediate results sent by each of the first participants. The second participant distributes the forward intermediate results of training process 0 of the first participant A and the forward intermediate results of training process 0 of the first participant B to the training process 0 of the second participant, and performs forward propagation and backward propagation in sequence based on the server-side model. The second participant distributes the forward intermediate results of training process 1 of the first participant A and the forward intermediate results of training process 1 of the first participant B to the training process 1 of the second participant, and performs forward propagation and backward propagation in sequence based on the server-side model. The second participant sends the backward intermediate results of training process 0 to the first participant A and the first participant B respectively, and also sends the backward intermediate results of training process 1 to the first participant A and the first participant B respectively.

[0195] In the aforementioned backpropagation process on the server-side model based on the forward propagation results, loss can be calculated based on the forward propagation results and corresponding sample labels, and then backpropagation can be performed on the server-side model based on the calculated loss; for example... Figure 3 As shown, the positive intermediate results of training process 0 of the first participant A and the positive intermediate results of training process 0 of the first participant B are allocated to the training process 0 of the second participant. Forward propagation is performed based on the server-side model. Loss is calculated based on the forward propagation results and sample labels 0, and backpropagation is performed on the server-side model based on the calculated loss. Figure 4 The forward and backward propagation processes are illustrated. The segmentation layer output of client model A (training process 0 of the first participant A) is input to the server model (training process 0 of the second participant). Layer 4 of the server model processes the segmentation layer output, outputting y4. Layer 5 of the server model processes y4, outputting y5. Loss is calculated based on y5 and its corresponding sample labels. Backpropagation is then performed on the server model based on this calculated loss. Specifically, during backpropagation, the gradient data corresponding to y5 is first calculated, and then the gradient data of the model parameters of layer 5 of the server model is calculated based on the gradient corresponding to y5. This gradient may include the weights, biases, and other parameters of layer 5. The gradient data of y5 is used to calculate the gradient data of y4. The gradient data of y4 is then propagated backward to layer 4. The layer 4 of the server model calculates the gradient data of the weights, biases and other parameters of layer 4 based on the gradient data of y4. Then, the gradient data of the segmentation layer is calculated based on the gradient data of y4 as the reverse intermediate result. The reverse intermediate result is sent to the first participant A. The forward and backward propagation processes of training process 1 are similar to those of training process 0, and will not be described in detail here. In this way, by returning the gradient data of the segmentation layer to the first participant, the data security between the first participant and the second participant is ensured.

[0196] During backpropagation, after calculating the gradient data, the process of synchronizing gradient data with other training processes and updating the server-side model parameters can be performed. This process involves calculating the gradient data of the server-side model parameters during backpropagation and then synchronizing the gradient data with other training processes to update the server-side model parameters. The gradient data of the server-side model parameters includes the gradient data of each layer's parameters. Figure 4 As shown, the gradient data of the server-side model parameters includes the gradient data corresponding to the weights, biases, and other parameters of the 5th layer and the gradient data corresponding to the weights, biases, and other parameters of the 4th layer.

[0197] In practical applications, after backpropagation is performed on each training process of the second participant to obtain gradient data, since the model training of the various training processes of the second participant is distributed, in order to facilitate continuing model training for at least one more round or to obtain the trained model after terminating model training, gradient data can be synchronized with other training processes, and the server-side model parameters can be updated based on the synchronization results, so that the server-side model achieves parameter synchronization in each round. During the process of synchronizing gradient data with other training processes and updating server-side model parameters, the corresponding training process of the second participant can perform the following operations:

[0198] The gradient data of the server-side model parameters is synchronized with other training processes through inter-process communication algorithms, and the gradient data of each process is aggregated with the gradient data synchronized with other training processes.

[0199] The server-side model parameters are updated based on the aggregated gradient data obtained from the aggregation.

[0200] The inter-process communication algorithm can be the all-reduce algorithm (a communication and computation algorithm used in distributed deep learning).

[0201] Specifically, gradient data of server-side model parameters can be sent to other training processes through inter-process communication algorithms, so that each training process has gradient data of its own server-side model parameters. The gradient data of server-side model parameters of each training process are aggregated, and the server-side model parameters are updated based on the aggregated gradient data. Specifically, aggregation can be achieved by calculating the average value of the gradient data of server-side model parameters of each training process, so the aggregated gradient data can be the average value of the gradient data of server-side model parameters of each training process.

[0202] like Figure 4As shown, the gradient data corresponding to the weights, biases, and other parameters of layers 4 and 5 are synchronized with other training processes through an inter-process communication algorithm. The gradient data corresponding to the weights, biases, and other parameters of layers 4 of each process are then aggregated with the gradient data corresponding to the weights, biases, and other parameters of layers 4 synchronized with other training processes. The parameters of layer 4 of each process's server model are updated based on the aggregated gradient data of layer 4. That is, the parameters of layer 4 of the server model of training process 0 are updated based on the aggregated gradient data of layer 4 of layer 0. The same applies to layer 5. The gradient data corresponding to the weights, biases, and other parameters of layers 5 of each process are aggregated with the gradient data corresponding to the weights, biases, and other parameters of layers 5 synchronized with other training processes. The parameters of layer 5 of each process's server model are updated based on the aggregated gradient data of layer 5.

[0203] In practical applications, to improve the flexibility of model training and allow for control over the training duration at any time, training termination conditions can be set; the following operations can also be performed:

[0204] Check whether the updated server-side model triggers the training termination condition;

[0205] If triggered, the updated server-side model will be sent to the training statistics module, and a training termination instruction will be sent to each first participant, so that the updated client-side model and the updated server-side model can be used as the model training results in the training statistics module.

[0206] If not triggered, return to step S802.

[0207] It should be noted that each execution operation process and each execution step provided in this embodiment can be arbitrarily referenced, combined, or separated for independent execution as needed. This embodiment does not impose specific limitations on the execution order.

[0208] The following description uses the application of a distributed model training method provided in this embodiment in a distributed training scenario as an example to further illustrate the distributed model training method provided in this embodiment. (See also...) Figure 7 The distributed model training method applied to distributed training scenarios includes the following steps.

[0209] Step S704: Receive the positive intermediate results sent by each client.

[0210] Step S706: Distribute the forward intermediate results to the corresponding training processes and perform forward and backward propagation sequentially based on the server-side model.

[0211] Step S708: Send the intermediate backpropagation results obtained during backpropagation to each client, and after calculating the gradient data during backpropagation, synchronize the gradient data with other training processes and update the server-side model parameters.

[0212] It should be noted that any one or any combination of steps S704 to S708 can be replaced with the corresponding technical means provided in steps S802 to S806 as needed for implementation and deployment, and will not be elaborated here.

[0213] This specification provides an example of a distributed model training system as follows:

[0214] Reference Figure 9 This illustration shows a schematic diagram of an embodiment of a distributed model training system provided in this embodiment.

[0215] This embodiment provides a distributed model training system, including multiple participants running multiple training processes, and the multiple training processes of different participants correspond to each other. The first participant among the multiple participants has a client model, and the second participant among the multiple participants has a server model.

[0216] Each first participant 910 is configured to allocate sample data blocks to different training processes for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of different training processes to the second participant; and receive the reverse intermediate results from the second participant, allocate them to the corresponding training process for continued reverse propagation based on the client model, calculate gradient data during the reverse propagation process, synchronize gradient data with other training processes, and update the client model parameters; and align entities between sample data blocks allocated to the corresponding training processes by different first participants.

[0217] The second participant 920 is configured to receive forward intermediate results sent by different first participants and allocate them to the corresponding training process for forward and backward propagation based on the server model in sequence, and send the reverse intermediate results obtained in the backward propagation to each of the first participants; and after calculating the gradient data during the backward propagation process, synchronize the gradient data with other training processes and update the server model parameters.

[0218] The first embodiment of a distributed model training device provided in this specification is as follows:

[0219] In the above embodiments, a first distributed model training method is provided, and correspondingly, a first distributed model training device is also provided, which will be described below with reference to the accompanying drawings.

[0220] Reference Figure 10 This illustration shows a schematic diagram of an embodiment of a distributed model training device provided in this embodiment.

[0221] Since the apparatus embodiments correspond to the method embodiments, the descriptions are relatively simple. For relevant parts, please refer to the corresponding descriptions of the method embodiments provided above. The apparatus embodiments described below are merely illustrative.

[0222] This embodiment provides a distributed model training device that runs on multiple participants with multiple training processes, and the training processes of different participants correspond to each other. The first participant among the multiple participants has a client model, and the second participant among the multiple participants has a server model. The device includes:

[0223] The data block allocation module 1002 is configured such that each first participant allocates sample data blocks to different training processes for forward propagation based on the client model, and sends the forward intermediate results obtained from the forward propagation of different training processes to the second participant; entity alignment is performed between sample data blocks allocated to the corresponding training processes by different first participants.

[0224] The intermediate result receiving module 1004 is configured to receive the forward intermediate results sent by different first participants, and allocate them to the corresponding training process for forward and backward propagation based on the server model in sequence, and send the reverse intermediate results obtained in the backpropagation to each of the first participants; and after calculating the gradient data during the backpropagation process, synchronize the gradient data with other training processes and update the server model parameters.

[0225] The intermediate result allocation module 1006 is configured such that each of the first participants receives the reverse intermediate results from the second participant and allocates them to the corresponding training process to continue backpropagation based on the client model. During the backpropagation process, gradient data is calculated and then synchronized with other training processes to update the client model parameters.

[0226] The second embodiment of the distributed model training device provided in this specification is as follows:

[0227] In the above embodiments, a second distributed model training method is provided, and correspondingly, a second distributed model training device is also provided, which will be described below with reference to the accompanying drawings.

[0228] Reference Figure 11 This illustration shows a schematic diagram of an embodiment of a distributed model training device provided in this embodiment.

[0229] Since the apparatus embodiments correspond to the method embodiments, the descriptions are relatively simple. For relevant parts, please refer to the corresponding descriptions of the method embodiments provided above. The apparatus embodiments described below are merely illustrative.

[0230] This embodiment provides a distributed model training device that runs on a first participant having multiple training processes. The training processes of each first participant correspond to each other and also correspond to the training processes of a second participant. Each first participant has a client model, and the second participant has a server model. The device includes:

[0231] The forward intermediate result sending module 1102 is configured to allocate sample data blocks to different training processes for forward propagation based on the client model, and send the forward intermediate results obtained from the forward propagation of different training processes to the second participant; entity alignment between sample data blocks allocated to the corresponding training processes by different first participants;

[0232] The reverse intermediate result receiving module 1104 is configured to receive the reverse intermediate result sent by the second participant, and allocate it to the corresponding training process for backpropagation based on the client model. After calculating the gradient data during the backpropagation process, it synchronizes the gradient data with other training processes and updates the client model parameters.

[0233] The reverse intermediate result is obtained by the second participant distributing the forward intermediate results sent by different first participants to the corresponding training processes and then performing forward and backward propagation based on the server-side model.

[0234] The third embodiment of the distributed model training device provided in this specification is as follows:

[0235] In the above embodiments, a third distributed model training method is provided, and correspondingly, a third distributed model training device is also provided, which will be described below with reference to the accompanying drawings.

[0236] Reference Figure 12 This illustration shows a schematic diagram of an embodiment of a distributed model training device provided in this embodiment.

[0237] Since the apparatus embodiments correspond to the method embodiments, the descriptions are relatively simple. For relevant parts, please refer to the corresponding descriptions of the method embodiments provided above. The apparatus embodiments described below are merely illustrative.

[0238] This embodiment provides a distributed model training device that runs on a first participant having multiple training processes. The training processes of each first participant correspond to each other and also correspond to the training processes of a second participant. Each first participant has a client model, and the second participant has a server model. The device includes:

[0239] The forward intermediate result receiving module 1202 is configured to receive the forward intermediate results sent by each of the first participants; the forward intermediate results are obtained by each of the first participants distributing sample data blocks to different training processes and performing forward propagation based on the client model, and the sample data blocks distributed by different first participants to the corresponding training processes are entity aligned.

[0240] The forward intermediate result allocation module 1204 is configured to allocate the forward intermediate result to the corresponding training process and perform forward and backward propagation sequentially based on the server-side model;

[0241] The reverse intermediate result sending module 1206 is configured to send the reverse intermediate results obtained in the backpropagation to each of the first participants, and after calculating the gradient data in the backpropagation process, synchronize the gradient data with other training processes and update the server-side model parameters.

[0242] The first example of a distributed model training device provided in this specification is as follows:

[0243] Corresponding to the first distributed model training method described above, based on the same technical concept, one or more embodiments of this specification also provide a first distributed model training device, which is used to execute the first distributed model training method provided above. Figure 13 This is a schematic diagram of the structure of a distributed model training device provided for one or more embodiments of this specification.

[0244] This embodiment provides a distributed model training device, comprising:

[0245] like Figure 13As shown, distributed model training devices can vary significantly due to differences in configuration or performance. They may include one or more processors 1301 and memory 1302, with memory 1302 storing one or more application programs or data. Memory 1302 can be temporary or persistent storage. The application programs stored in memory 1302 may include one or more modules (not shown), each module including a series of computer-executable instructions from the distributed model training device. Furthermore, processor 1301 may be configured to communicate with memory 1302, executing the series of computer-executable instructions in memory 1302 on the distributed model training device. The distributed model training device may also include one or more power supplies 1303, one or more wired or wireless network interfaces 1304, one or more input / output interfaces 1305, one or more keyboards 1306, etc.

[0246] In one specific embodiment, the distributed model training device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the distributed model training device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0247] Each first participant allocates sample data blocks to different training processes for forward propagation based on the client model, and sends the forward intermediate results obtained from the forward propagation of different training processes to the second participant; entity alignment is performed between sample data blocks allocated to corresponding training processes by different first participants.

[0248] The second participant receives forward intermediate results sent by different first participants and allocates them to the corresponding training process. Based on the server-side model, forward propagation and backward propagation are performed sequentially. The reverse intermediate results obtained in the backward propagation are sent to each of the first participants. In addition, after calculating the gradient data during the backward propagation process, the gradient data is synchronized with other training processes and the server-side model parameters are updated.

[0249] Each of the first participants receives the reverse intermediate results from the second participant and assigns them to the corresponding training process to continue backpropagation based on the client model. During the backpropagation process, after calculating the gradient data, the gradient data is synchronized with other training processes and the client model parameters are updated.

[0250] The second example of a distributed model training device provided in this specification is as follows:

[0251] Corresponding to the second distributed model training method described above, based on the same technical concept, one or more embodiments of this specification also provide a second distributed model training device, which is used to execute the second distributed model training method provided above. Figure 14 This is a schematic diagram of the structure of a distributed model training device provided for one or more embodiments of this specification.

[0252] This embodiment provides a distributed model training device, comprising:

[0253] like Figure 14 As shown, distributed model training devices can vary significantly due to differences in configuration or performance. They may include one or more processors 1401 and memory 1402, with memory 1402 storing one or more application programs or data. Memory 1402 can be temporary or persistent storage. The application programs stored in memory 1402 may include one or more modules (not shown), each module including a series of computer-executable instructions from the distributed model training device. Furthermore, processor 1401 may be configured to communicate with memory 1402, executing the series of computer-executable instructions in memory 1402 on the distributed model training device. The distributed model training device may also include one or more power supplies 1403, one or more wired or wireless network interfaces 1404, one or more input / output interfaces 1405, one or more keyboards 1406, etc.

[0254] In one specific embodiment, the distributed model training device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the distributed model training device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0255] Sample data blocks are allocated to different training processes and forward propagated based on the client model. The forward intermediate results obtained from the forward propagation of different training processes are sent to the second participant. Entity alignment is performed between sample data blocks allocated to the corresponding training processes by different first participants.

[0256] Receive the reverse intermediate results sent by the second participant and assign them to the corresponding training process for backpropagation based on the client model. After calculating the gradient data during the backpropagation process, synchronize the gradient data with other training processes and update the client model parameters.

[0257] The reverse intermediate result is obtained by the second participant distributing the forward intermediate results sent by different first participants to the corresponding training processes and then performing forward and backward propagation based on the server-side model.

[0258] The third example of a distributed model training device provided in this specification is as follows:

[0259] Corresponding to the third distributed model training method described above, based on the same technical concept, one or more embodiments of this specification also provide a third distributed model training device, which is used to execute the third distributed model training method provided above. Figure 15 This is a schematic diagram of the structure of a distributed model training device provided for one or more embodiments of this specification.

[0260] This embodiment provides a distributed model training device, comprising:

[0261] like Figure 15 As shown, distributed model training devices can vary significantly due to differences in configuration or performance. They may include one or more processors 1501 and memory 1502, with memory 1502 storing one or more application programs or data. Memory 1502 can be temporary or persistent storage. The application programs stored in memory 1502 may include one or more modules (not shown), each module including a series of computer-executable instructions from the distributed model training device. Furthermore, processor 1501 may be configured to communicate with memory 1502, executing the series of computer-executable instructions in memory 1502 on the distributed model training device. The distributed model training device may also include one or more power supplies 1503, one or more wired or wireless network interfaces 1504, one or more input / output interfaces 1505, one or more keyboards 1506, etc.

[0262] In one specific embodiment, the distributed model training device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the distributed model training device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0263] Receive the forward intermediate results sent by each first participant; the forward intermediate results are obtained by each first participant allocating sample data blocks to different training processes and performing forward propagation based on the client model, and the entity alignment between the sample data blocks allocated by different first participants to the corresponding training processes.

[0264] The positive intermediate results are distributed to the corresponding training processes, and forward and backward propagation are performed sequentially based on the server-side model.

[0265] The reverse intermediate results obtained in the backpropagation are sent to each of the first participants, and the gradient data is calculated during the backpropagation process. The gradient data is then synchronized with other training processes and the server-side model parameters are updated.

[0266] The first embodiment of the computer-readable storage medium provided in this specification is as follows:

[0267] Corresponding to the first distributed model training method described above, and based on the same technical concept, one or more embodiments of this specification also provide a first computer-readable storage medium.

[0268] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed, perform the following steps:

[0269] Each first participant allocates sample data blocks to different training processes for forward propagation based on the client model, and sends the forward intermediate results obtained from the forward propagation of different training processes to the second participant; entity alignment is performed between sample data blocks allocated to corresponding training processes by different first participants.

[0270] The second participant receives forward intermediate results sent by different first participants and allocates them to the corresponding training process. Based on the server-side model, forward propagation and backward propagation are performed sequentially. The reverse intermediate results obtained in the backward propagation are sent to each of the first participants. In addition, after calculating the gradient data during the backward propagation process, the gradient data is synchronized with other training processes and the server-side model parameters are updated.

[0271] Each of the first participants receives the reverse intermediate results from the second participant and assigns them to the corresponding training process to continue backpropagation based on the client model. During the backpropagation process, after calculating the gradient data, the gradient data is synchronized with other training processes and the client model parameters are updated.

[0272] It should be noted that the embodiments of the first computer-readable storage medium in this specification and the embodiments of the first distributed model training method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0273] The second embodiment of the computer-readable storage medium provided in this specification is as follows:

[0274] Corresponding to the second distributed model training method described above, and based on the same technical concept, one or more embodiments of this specification also provide a second computer-readable storage medium.

[0275] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed, perform the following steps:

[0276] Sample data blocks are allocated to different training processes and forward propagated based on the client model. The forward intermediate results obtained from the forward propagation of different training processes are sent to the second participant. Entity alignment is performed between sample data blocks allocated to the corresponding training processes by different first participants.

[0277] Receive the reverse intermediate results sent by the second participant and assign them to the corresponding training process for backpropagation based on the client model. After calculating the gradient data during the backpropagation process, synchronize the gradient data with other training processes and update the client model parameters.

[0278] The reverse intermediate result is obtained by the second participant distributing the forward intermediate results sent by different first participants to the corresponding training processes and then performing forward and backward propagation based on the server-side model.

[0279] It should be noted that the embodiments of the second type of computer-readable storage medium in this specification and the embodiments of the second type of distributed model training method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0280] The third embodiment of the computer-readable storage medium provided in this specification is as follows:

[0281] Corresponding to the third distributed model training method described above, and based on the same technical concept, one or more embodiments of this specification also provide a third computer-readable storage medium.

[0282] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed, perform the following steps:

[0283] Receive the forward intermediate results sent by each first participant; the forward intermediate results are obtained by each first participant allocating sample data blocks to different training processes and performing forward propagation based on the client model, and the entity alignment between the sample data blocks allocated by different first participants to the corresponding training processes.

[0284] The positive intermediate results are distributed to the corresponding training processes, and forward and backward propagation are performed sequentially based on the server-side model.

[0285] The reverse intermediate results obtained in the backpropagation are sent to each of the first participants, and the gradient data is calculated during the backpropagation process. The gradient data is then synchronized with other training processes and the server-side model parameters are updated.

[0286] It should be noted that the embodiments of the third type of computer-readable storage medium in this specification and the embodiments of the third type of distributed model training method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0287] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments. For example, the device embodiment, equipment embodiment and computer-readable storage medium embodiment are all similar to the method embodiment, so the description is relatively simple. When reading the relevant content of the device embodiment, equipment embodiment and computer-readable storage medium embodiment, please refer to the description of the method embodiment.

[0288] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0289] In the 1930s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0290] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0291] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0292] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0293] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0294] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable distributed model training apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable distributed model training apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0295] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable distributed model training device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0296] These computer program instructions can also be loaded onto a computer or other programmable distributed model training device, causing a series of operational steps to be executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0297] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0298] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0299] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0300] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0301] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0302] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0303] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A distributed model training method, applied to multiple participants running multiple training processes, wherein the multiple training processes of different participants correspond to each other, wherein the clients among the multiple participants have client models, and the servers among the multiple participants have server models, the method comprising: Each client allocates sample data blocks to different training processes to extract image features based on the client-side model, and sends the feature extraction results obtained from different training processes to the server; user identifiers are aligned between the sample data blocks allocated to the corresponding training processes by different clients; the client-side model of the first client is an image color processing model, and the client-side model of the second client is an image text processing model; the server-side model is an image quality detection model; the image color processing model, the image text processing model, and the image quality detection model are obtained by decomposing the model structure to be trained; The server receives feature extraction results sent by different clients and allocates them to the corresponding training process. Based on the server model, forward propagation and backward propagation are performed sequentially, and the reverse intermediate results obtained in the backward propagation are sent to each client. Furthermore, after calculating the gradient data during backpropagation, the gradient data is synchronized with other training processes and the server-side model parameters are updated. Each client receives the reverse intermediate results from the server and is assigned to the corresponding training process to continue backpropagation based on the client model. During the backpropagation process, the gradient data is calculated and then synchronized with the gradient data of other training processes and the client model parameters are updated.

2. In the distributed model training method according to claim 1, the sample data blocks of each client are obtained from the data fragments of each client; The data shards for each client are obtained in the following manner: The training data of each client is aligned with user identifiers, and the aligned data is then split into multiple data fragments for each client. The number of data fragments obtained by data splitting is equal to the number of training processes.

3. In the distributed model training method according to claim 2, multiple data shards of each client are allocated to multiple training processes of each client, and there is a one-to-one correspondence between the allocated data shards and the training processes.

4. The distributed model training method according to claim 2, wherein the user identifier alignment processing of the training data of each client includes: The intersection of the training data of each client is calculated based on the user identifier in the training data to obtain the intersection data of each client; The user identifiers of the intersection data of the various clients are aligned according to the user identifiers in the intersection data.

5. The distributed model training method according to claim 2, wherein each client runs at least one preprocessing process for data preprocessing, and performs data preprocessing on the aligned data through the at least one preprocessing process before data segmentation.

6. The distributed model training method according to claim 5, wherein the data preprocessing of the aligned data through the at least one preprocessing process includes: The aligned data is split according to the number of preprocessing processes; The number of datasets obtained from data splitting is equal to the number of preprocessing processes. At least one dataset obtained from data splitting is assigned to different preprocessing processes for data preprocessing.

7. The distributed model training method according to claim 1, wherein synchronizing gradient data with other training processes and updating server-side model parameters includes: The gradient data of the server-side model parameters is synchronized with other training processes through inter-process communication algorithms, and the gradient data of each process is aggregated with the gradient data synchronized with other training processes. The server-side model parameters are updated based on the aggregated gradient data obtained from the aggregation.

8. The distributed model training method according to claim 1, wherein synchronizing gradient data with other training processes and updating client model parameters includes: The gradient data of the client model parameters is synchronized with the gradient data of other training processes through inter-process communication algorithms, and the gradient data of each process is aggregated with the gradient data synchronized with the gradient data of other training processes. The client model parameters are updated based on the aggregated gradient data obtained from the aggregation.

9. A distributed model training method, applied to a client running multiple training processes, wherein the multiple training processes of each client correspond to each other and also correspond to multiple training processes of a server, wherein each client has a client model and the server has a server model, the method comprising: Sample data blocks are allocated to different training processes for image feature extraction based on the client-side model, and the feature extraction results obtained from different training processes are sent to the server. User identifiers are aligned between sample data blocks allocated to corresponding training processes by different clients. The client-side model of the first client is an image color processing model, and the client-side model of the second client is an image text processing model. The server-side model is an image quality detection model. The image color processing model, the image text processing model, and the image quality detection model are obtained by decomposing the model structure of the model to be trained. The system receives the reverse intermediate results sent by the server and assigns them to the corresponding training process for backpropagation based on the client model. During the backpropagation process, the gradient data is calculated and then synchronized with the gradient data of other training processes and the client model parameters are updated. The reverse intermediate result is obtained by the server distributing the feature extraction results sent by different clients to the corresponding training processes and then performing forward and backward propagation based on the server model.

10. A distributed model training method, applied to a server running multiple training processes, wherein the multiple training processes on the server correspond to the multiple training processes on each client, each client has a client model, and the server has a server model, the method comprising: Receive the feature extraction results sent by each client; The feature extraction result is obtained by each client allocating sample data blocks to different training processes and performing image feature extraction based on the client model. User identifiers are aligned between the sample data blocks allocated to the corresponding training processes by different clients. The client model of the first client is an image color processing model, and the client model of the second client is an image text processing model. The server model is an image quality detection model. The image color processing model, the image text processing model, and the image quality detection model are obtained by decomposing the model structure of the model to be trained. The feature extraction results are allocated to the corresponding training processes, and forward and backward propagation are performed sequentially based on the server-side model. The inverse intermediate results obtained during backpropagation are sent to each client. After calculating the gradient data during backpropagation, the gradient data is synchronized with other training processes and the server-side model parameters are updated.

11. A distributed model training system, comprising multiple participants running multiple training processes, wherein the multiple training processes of different participants correspond to each other, wherein the client among the multiple participants has a client model, and the server among the multiple participants has a server model. in, Each client is configured to allocate sample data blocks to different training processes for image feature extraction based on the client model, and send the feature extraction results obtained from different training processes to the server; and receive the reverse intermediate results from the server, allocate them to the corresponding training processes for backpropagation based on the client model, calculate gradient data during backpropagation, synchronize gradient data with other training processes, and update client model parameters; user identifiers are aligned between sample data blocks allocated to corresponding training processes by different clients; the client model of the first client is an image color processing model, and the client model of the second client is an image text processing model; the server model is an image quality detection model; the image color processing model, the image text processing model, and the image quality detection model are obtained by decomposing the model structure of the model to be trained; The server is configured to receive feature extraction results sent by different clients and allocate them to the corresponding training process. Based on the server model, forward propagation and backward propagation are performed sequentially, and the reverse intermediate results obtained in the backward propagation are sent to each client. Furthermore, after calculating the gradient data during backpropagation, the gradient data is synchronized with other training processes and the server-side model parameters are updated.

12. A distributed model training device, running on multiple participants with multiple training processes, wherein the training processes of different participants correspond to each other, wherein the clients among the multiple participants have client models, and the servers among the multiple participants have server models, the device comprising: The data block allocation module is configured to allow each client to allocate sample data blocks to different training processes for image feature extraction based on the client model, and to send the feature extraction results obtained from different training processes to the server; user identifiers are aligned between the sample data blocks allocated to the corresponding training processes by different clients; the client model of the first client is an image color processing model, and the client model of the second client is an image text processing model; the server model is an image quality detection model; the image color processing model, the image text processing model, and the image quality detection model are obtained by decomposing the model structure of the model to be trained; The intermediate result receiving module is configured to receive feature extraction results sent by different clients on the server side, and allocate them to the corresponding training process to perform forward propagation and backward propagation in sequence based on the server-side model, and send the reverse intermediate results obtained in the backward propagation to each client. Furthermore, after calculating the gradient data during backpropagation, the gradient data is synchronized with other training processes and the server-side model parameters are updated. The intermediate result allocation module is configured so that each client receives the reverse intermediate result from the server and allocates it to the corresponding training process to continue backpropagation based on the client model. After calculating the gradient data during the backpropagation process, the gradient data is synchronized with other training processes and the client model parameters are updated.

13. A distributed model training device, running on clients with multiple training processes, wherein the training processes on each client correspond to each other and also correspond to multiple training processes on a server, wherein each client has a client model and the server has a server model, the device comprising: The forward intermediate result sending module is configured to allocate sample data blocks to different training processes for image feature extraction based on the client model, and send the feature extraction results obtained from the image feature extraction in different training processes to the server; the user identifiers are aligned between the sample data blocks allocated to the corresponding training processes by different clients; the client model of the first client is an image color processing model, and the client model of the second client is an image text processing model; the server model is an image quality detection model; the image color processing model, the image text processing model, and the image quality detection model are obtained by decomposing the model structure of the model to be trained; The reverse intermediate result receiving module is configured to receive the reverse intermediate result sent by the server, and allocate it to the corresponding training process for backpropagation based on the client model. After calculating the gradient data during the backpropagation process, it synchronizes the gradient data with other training processes and updates the client model parameters. The reverse intermediate result is obtained by the server distributing the feature extraction results sent by different clients to the corresponding training processes and then performing forward and backward propagation based on the server model.

14. A distributed model training device, running on a server with multiple training processes, wherein the multiple training processes on the server correspond to the multiple training processes on each client, each client having a client model, and the server having a server model, the device comprising: The forward intermediate result receiving module is configured to receive the feature extraction results sent by each client; The feature extraction result is obtained by each client allocating sample data blocks to different training processes and performing image feature extraction based on the client model. User identifiers are aligned between the sample data blocks allocated to the corresponding training processes by different clients. The client model of the first client is an image color processing model, and the client model of the second client is an image text processing model. The server model is an image quality detection model. The image color processing model, the image text processing model, and the image quality detection model are obtained by decomposing the model structure of the model to be trained. The forward intermediate result allocation module is configured to allocate the feature extraction results to the corresponding training process and perform forward and backward propagation sequentially based on the server-side model; The reverse intermediate result sending module is configured to send the reverse intermediate results obtained in the backpropagation to each client, and after calculating the gradient data during the backpropagation process, synchronize the gradient data with other training processes and update the server-side model parameters.

15. A distributed model training device, comprising: processor; And a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method of claim 1.

16. A distributed model training device, comprising: processor; And a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the steps of the method of claim 9.

17. A distributed model training device, comprising: processor; And a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the steps of the method of claim 10.

18. A computer-readable storage medium for storing computer-executable instructions that, when executed, implement the steps of the method of claim 1.

19. A computer-readable storage medium for storing computer-executable instructions that, when executed, implement the steps of the method of claim 9.

20. A computer-readable storage medium for storing computer-executable instructions that, when executed, implement the steps of the method of claim 10.

Images