Methods and apparatus for multi-party collaborative model training

Abstract

This specification provides a method and apparatus for multi-party collaborative model training. The multi-party components include a labeling party possessing sample labels and several data parties possessing sample features. The method is applied to any one of the data parties and includes: processing a first batch of local features corresponding to a first batch of sample identifiers agreed upon by the multi-party components using a locally deployed first model to obtain a first output, and sending the first output to the labeling party; performing a first update on the first model using contrastive learning based on several second batches of local features; receiving a returned gradient from the labeling party, which is determined based on the first outputs of each data party, the sample labels corresponding to the first batch of sample identifiers, and the target model deployed in the labeling party; and performing a second update on the first model after the first update based on the returned gradient. This achieves privacy protection for the data of all parties and effectively improves training efficiency.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and more particularly to a privacy-preserving split learning method and apparatus. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, Deep Neural Networks (DNNs) have been gradually applied to various fields such as product recommendation, risk assessment, speech recognition, facial recognition, and natural language processing. The network structure of DNNs is relatively fixed across different application scenarios, and more training data is needed to achieve better model performance.

[0003] In many fields such as healthcare and finance, different companies or institutions possess diverse data samples. Jointly training these data can significantly improve model accuracy, thereby effectively enhancing business quality and user satisfaction. However, this raw training data contains substantial amounts of user privacy and trade secrets; any leakage could cause irreparable negative consequences. Therefore, while addressing the data silo problem through multi-party collaborative training, protecting data privacy and data value has become a key research focus in recent years.

[0004] Otkrist et al. proposed the concept of split learning, which enables deep neural networks to be trained on multiple data sources without directly sharing the original data from the data sources.

[0005] However, current methods for joint training using split-learning techniques are limited and cannot meet the higher demands of practical applications. Therefore, a joint training scheme is needed to better meet practical application requirements, such as improving training efficiency. Summary of the Invention

[0006] This specification describes a method and apparatus for multi-party collaborative model training to protect privacy. Using these methods and apparatus, during the split learning process, training members also utilize contrastive learning algorithms to perform unsupervised training on the local model, eliminating the need to idle while waiting for gradient backpropagation, thereby effectively improving the overall training efficiency.

[0007] According to a first aspect, a method for multi-party joint model training is provided, wherein the multi-party includes a labeling party possessing sample labels and several data parties possessing sample features; the method is applied to any one of the data parties, comprising: processing the first batch of local features corresponding to a first batch of sample identifiers agreed upon by the multi-party using a locally deployed first model to obtain a first output, and sending the first output to the labeling party; performing a first update on the first model using contrastive learning based on several second batches of local features; receiving a backpropagation gradient from the labeling party, which is determined based on the first outputs of each data party, the sample labels corresponding to the first batch of sample identifiers, and the target model deployed in the labeling party; and performing a second update on the first model after the first update based on the backpropagation gradient.

[0008] In one embodiment, the labeler also possesses partial features corresponding to the identifiers of the first batch of samples, and the target model is used to fuse the first output of the labeler and the first outputs of each data party, and obtain a prediction result based on the fusion result; the first output of the labeler is obtained by processing the partial features using its first model.

[0009] In one embodiment, the plurality of data parties are multiple data parties, and the target model is used to fuse the first outputs of each data party and obtain a prediction result based on the fusion result.

[0010] In one embodiment, a first computing node and a second computing node are deployed in a distributed manner in any of the data parties; the first update in the method is performed on the first computing node, and the other steps are performed on the second computing node.

[0011] In one specific embodiment, the method further includes: synchronizing the model parameters updated by the first update on the first computing node to the second computing node.

[0012] In one embodiment, the first update includes: performing data augmentation on any batch of features from the plurality of second batches of local features to obtain augmented features; processing a feature set including the augmented features using at least the first model to obtain representation outputs for each feature; calculating a contrast loss using the representation outputs; and updating at least the first model based on the contrast loss.

[0013] In one specific embodiment, data augmentation processing is performed on any batch of features in the plurality of second batches of local features to obtain augmented features, including: for the local features corresponding to a single sample identifier, multiple feature values ​​are randomly masked to obtain the corresponding augmented features.

[0014] In one specific embodiment, at least the first model is used to process a feature set including the augmented features to obtain the representation output of each feature, including: processing each feature using the first model to obtain a first output of each feature; processing the first output of each feature using a locally deployed second model to obtain a second output of each feature as the representation output; and updating at least the first model according to the contrastive loss, including: updating the first model and the second model using the contrastive loss.

[0015] In one embodiment, the first model includes several layers of neural networks, and the method further includes: storing the input of each layer of the neural network when processing the first batch of local features using the first model; wherein the second update includes: determining the parameter gradient of each layer of the neural network based on the backpropagation gradient and the input of each layer of the neural network, and updating the model parameters in each layer of the neural network according to the parameter gradient.

[0016] According to a second aspect, a method for multi-party joint model training is provided, wherein multiple data parties among the multiple parties each possess different labeled training samples, and the multiple parties also include a neutral party; the method is applied to any one of the data parties, comprising: processing a first batch of local training samples using a locally deployed first model to obtain a first output, and sending the first output to the neutral party; performing a first update on the first model using contrastive learning based on several second batches of local training samples; receiving a backpropagation gradient from the neutral party, the backpropagation gradient being determined based on the first output, sample labels, and a target model deployed in the neutral party; and performing a second update on the first model after the first update based on the backpropagation gradient.

[0017] In one embodiment, the method further includes: sending the second updated model parameters to one of the other data parties among the plurality of data parties, so that the data party uses the second updated model parameters as the initial parameters of its model, thereby training its model.

[0018] In one embodiment, a first computing node and a second computing node are deployed in a distributed manner in any of the data parties, and the first update in the method is performed on the first computing node, while the other steps are performed on the second computing node.

[0019] In one specific embodiment, the method further includes: synchronizing the model parameters updated by the first update on the first computing node to the second computing node.

[0020] In one embodiment, the first update includes: performing data augmentation on any batch of training samples from the plurality of second batches of training samples to obtain augmented samples; processing a sample set including the augmented samples using at least the first model to obtain representation outputs for each sample therein; calculating a contrastive loss using the representation outputs; and updating at least the first model based on the contrastive loss.

[0021] In one specific embodiment, data augmentation processing is performed on any batch of training samples from the plurality of second batches of training samples to obtain augmented samples, including: adding noise to each training sample to obtain the corresponding augmented sample.

[0022] In one specific embodiment, at least the first model is used to process the sample set including the augmented samples to obtain the representation output of each sample, including: processing each sample using the first model to obtain a first output of each sample; processing the first output of each sample using a locally deployed second model to obtain a second output of each sample as the representation output; and updating at least the first model according to the contrastive loss, including: updating the first model and the second model using the contrastive loss.

[0023] In one embodiment, the first model includes a multi-layer neural network, and the method further includes: storing the inputs of each layer of the neural network when processing the first batch of local training samples using the first model; wherein the second update includes: determining the parameter gradients of each layer of the neural network based on the backpropagation gradients and the inputs of each layer of the neural network, and updating the model parameters in each layer of the neural network according to the parameter gradients.

[0024] According to a third aspect, an apparatus for multi-party joint model training is provided, wherein the multi-party includes a labeling party possessing sample labels and several data parties possessing sample features; the apparatus is integrated into any one of the data parties and includes: a first model processing unit configured to process a first batch of local features corresponding to a first batch of sample identifiers agreed upon by the multi-party using a locally deployed first model, to obtain a first output; a sending unit configured to send the first output to the labeling party; a first update unit configured to perform a first update on the first model based on several second batches of local features using contrastive learning; a receiving unit configured to receive a feedback gradient from the labeling party, which is determined based on the first outputs of each data party, the sample labels corresponding to the first batch of sample identifiers, and the target model deployed in the labeling party; and a second update unit configured to perform a second update on the first model after the first update based on the feedback gradient.

[0025] According to a fourth aspect, an apparatus for multi-party joint model training is provided, wherein multiple data parties among the multiple parties each possess different labeled training samples, and the multiple parties also include a neutral party; the apparatus is integrated into any one of the data parties and includes: a first model processing unit configured to process a first batch of local training samples using a locally deployed first model to obtain a first output; a sending unit configured to send the first output to the neutral party; a first update unit configured to perform a first update on the first model based on several second batches of local training samples using contrastive learning; a receiving unit configured to receive a feedback gradient from the neutral party, the feedback gradient being determined based on the first output, sample labels, and a target model deployed in the neutral party; and a second update unit configured to perform a second update on the first model after the first update based on the feedback gradient.

[0026] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first or second aspect.

[0027] According to a sixth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of the first aspect or the second aspect.

[0028] In the methods and apparatus provided in the embodiments of this specification, unsupervised contrastive learning is introduced into the data side to update the locally deployed first model, so that the data side is not in an idle state before receiving the backpropagation gradient, thereby improving the overall training efficiency; and, at certain time intervals, the backpropagation gradient is used to update the first model to prevent the parameters of the first model from diverging, thereby improving the overall convergence speed and thus improving the training efficiency. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This diagram illustrates a scenario architecture for multi-party collaborative model training according to one embodiment.

[0031] Figure 2 A schematic diagram illustrating a method for multi-party collaborative model training according to one embodiment is shown.

[0032] Figure 3This diagram illustrates a method flow chart for multi-party collaborative model training according to another embodiment.

[0033] Figure 4 A schematic diagram of a device structure for multi-party joint model training is shown according to one embodiment;

[0034] Figure 5 A schematic diagram of a device structure for multi-party joint model training according to another embodiment is shown. Detailed Implementation

[0035] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0036] As mentioned earlier, different enterprises or organizations possess different data. To fully leverage the value of this data and improve business quality and user satisfaction, joint training using this data is proposed. In joint training, based on how the complete sample is segmented among multiple training members, it can be divided into horizontal and vertical scenarios. In a horizontal scenario, multiple training members possess complete samples of different business objects; that is, they share the same feature space but differ in their sample space. For example, in image recognition, different training members possess labeled data for different images. In a vertical scenario, multiple training members possess different feature parts of the same sample; that is, they share the same sample space but differ in their feature space. For example, in recommendation systems, one training member possesses user features from user behavior samples, while another training member possesses behavioral object features and behavioral labels from user behavior samples.

[0037] Split learning is applicable to both the horizontal and vertical scenarios described above. In the horizontal split learning model, each training member sequentially trains locally using private data, obtaining the output of the first few layers of the network model. This output is then transmitted to the server for forward propagation of the remaining layers, yielding the server model's output. The server then transmits this output to the training member, continuing forward propagation, calculating the loss function, and performing backpropagation to update both the server and training member models. The training member then sends its local model to the next training member to update their local model, continuing subsequent computations.

[0038] During the vertical split learning model process, the training members simultaneously perform forward propagation of their local models and send them to the label holder. The label holder receives all the hidden layer representations, fuses them, and then continues forward and backward propagation to update the local models of all training members.

[0039] However, regardless of whether it is a horizontal or vertical split learning model, because forward and backward propagation are sequential, the training members need to wait for the gradient to be backpropagated after calculating the output of their local model before they can update their local model, resulting in low training efficiency.

[0040] Based on the above observations and analysis, the inventors proposed that, in addition to calculating the output of the local model, the training data provider uses contrastive learning to perform unsupervised training on the local model, eliminating the need to wait for the backpropagation of training gradients, thereby effectively improving the overall training efficiency. At the same time, at certain training intervals, the backpropagation gradients of the split learning are used to update the local model, preventing the divergence of each local model and improving the overall convergence speed, thereby improving training efficiency and effectiveness.

[0041] Next, we will introduce the joint training scheme proposed by the inventor for vertical splitting scenarios.

[0042] Figure 1 This diagram illustrates a scenario architecture for multi-party collaborative model training according to one embodiment, where the multiple parties include a label party possessing sample labels and several data parties possessing sample features. It should be understood that "several" in this text refers to one or more; furthermore, the label party may contain only sample labels or may also include some of the sample features.

[0043] The following section provides an example of the data distribution among the aforementioned parties, using specific business scenarios as examples.

[0044] In recommendation scenarios, taking the example of two parties among the aforementioned multiple parties, these two parties include the user data party and the tag party. The user data party possesses the user data in the recommendation samples, while the tag party possesses the object data and sample tags in the recommendation samples.

[0045] In one embodiment, user data includes static attribute features of the user, such as gender, age, occupation, interests, place of residence, and asset information. For example, these features can be extracted from user registration information. In another embodiment, user data also includes dynamic behavioral features of the user. In one example, the user data provider is a social media platform; in this case, dynamic behavioral features may include social interaction duration, frequency of social interaction, number of friends, number of personal updates, number of likes, etc. In another example, the user data provider is a payment platform; in this case, dynamic behavioral features may include payment frequency, payment amount, and payment time period, etc.

[0046] In one embodiment, the tagger can be an e-commerce platform, payment platform, or content platform, and the business objects involved in the object data it possesses can be products, services, public accounts, subscription accounts, applications, content information, advertisements, etc. In one example, the object data is product data, i.e., product characteristics such as category, place of origin, cost, selling price, sales volume, and target audience. In another example, the object data is public account characteristics, such as category, number of followers, frequency of published content, number of likes, and number of reposts. In yet another example, the object data is advertising characteristics, such as advertising medium (including images, text, video, or audio), advertising content, and spokesperson information.

[0047] It is understood that in recommendation scenarios, the labels on recommended samples indicate whether a user has made a pre-order action regarding a business object, and the pre-order action is appropriate to the business object. In one embodiment, the business object is a product or service, and the pre-order action could be purchasing or using it. In another embodiment, the business object is a public account or subscription account, and the pre-order action could be following or subscribing. In yet another embodiment, the business object is an application, and the pre-order action could be downloading or registering. In still another embodiment, the business object is an advertisement or content information, and the pre-order action could be clicking.

[0048] The above describes the distribution of multi-party data in recommendation scenarios.

[0049] In risk control scenarios, multiple parties can each hold different behavioral data of the same user, and the tagging party also holds the user's risk label. For example, these multiple parties can include banks, payment platforms, credit service platforms, and credit reporting agencies. Furthermore, banks hold the user's bank statements, such as deposit and withdrawal records; payment platforms hold the user's payment data, such as payment amount and frequency; credit service platforms hold the user's credit service usage data, such as records of deposit-free rentals of books and bicycles; and credit reporting agencies hold the user's credit data, such as the number of credit cards the user has and their repayment default records.

[0050] In one embodiment, the credit reporting agency can be a labeler, and the risk label it holds can be a risk category label, such as low risk, medium risk, and high risk, or a risk score label, such as 30 points or 90 points in a percentage system.

[0051] The above describes the distribution of multi-party data in risk control scenarios. It's understandable that the business scenarios for vertically splitting learning are not limited to the recommendation and risk control scenarios mentioned above; they can also include other business scenarios, which will not be exhaustively listed here.

[0052] Back Figure 1 The label P is shown in the figure. y and m (m is a positive integer) data squares holding sample features, wherein any one of the data squares is denoted as P. i ;P y Deploying target model M in the middle t P i Deploy the first model M in the middle i .

[0053] Based on the above, during the joint training process, P i The party will deliver a batch of goods as agreed by multiple parties. Figure 1 The local features corresponding to the sample identifiers in batch b1 are input into the first model M. i The first output o is obtained. i And send it to P yFang, and P i Fang received P y Before the gradient is returned by the square, a contrastive learning algorithm is used, utilizing several batches ( Figure 1 The batch number is designated as b2 to b. n The local features of the first model M) i Perform several iterations of training to achieve the desired results for the first model M. i The model parameters (or weight parameters) are updated in several iterations. Figure 1 The iterative change process of the parameters is illustrated in the diagram: w0→w1→…→w n-1 After that, P i Fang can utilize the received backpropagation gradient g i For the first model M i The model parameter w n-1 Further updates are being made, now updated to w n Therefore, P can be made i Fang calculates the first output o i and receive the back-transmitted gradient g i The time intervals between these periods are not idle; rather, contrastive learning is used to train the first model M. i Continuous training effectively improves overall training efficiency.

[0054] in addition, Figure 1 China to P y Calculate the backpropagation gradient g i This is illustrated using one of several methods. For example... Figure 1 As shown, P y Fang uses target model M t The fusion results of the m first outputs received from m data sources are processed to obtain the prediction result. Therefore, based on the prediction results Given the corresponding sample label Y, calculate the m backpropagation gradients for the m data sides, including those backpropagated to P. i The backpropagation gradient g of the square i .

[0055] The specific implementation steps of the above solution are described below with reference to more embodiments. Figure 2 This diagram illustrates a method flow for multi-party joint model training according to one embodiment, wherein the multi-party includes Figure 1 The diagram shows the label side and several (denoted as m) data sides. It can be understood that each of these sides can be implemented as any device, server, platform, or cluster of devices with computing and processing capabilities.

[0056] Figure 2 The execution subject of the method shown is any one of the several data parties, P. i .like Figure 2 As shown, the method includes the following steps:

[0057] Step S210: Use the locally deployed first model to process the first batch of local features corresponding to the first batch of sample identifiers agreed upon by multiple parties, and obtain the first output.

[0058] In one embodiment, P i The first batch of sample identifiers is randomly selected from the training sample identifier set and sent to the other parties in the multi-party group. In another embodiment, P i Fang can receive the first batch of sample identifiers selected by one of the other parties from the training sample identifier set.

[0059] It is understandable that the training sample identifier set is shared by multiple parties. In one embodiment, P i Based on its local sample identifier set, Fang performs a Privacy Set Intersection (PSI) with the sample identifier sets held by other parties in the multi-party group to obtain the sample identifier intersection, which serves as the aforementioned training sample identifier set. In a specific embodiment, the execution process of the PSI protocol is as follows: P i Fang first hashes each sample identifier in its local sample identifier set, then sends the resulting set of identifier hash values ​​to the server. After receiving the intersection of the identifier hash values ​​obtained by the server from the intersection of multiple sets of identifier hash values ​​sent by multiple parties, Fang determines the sample identifier intersection based on the mapping relationship between identifier hash values ​​and sample identifiers.

[0060] For example, in recommendation or risk control scenarios, the sample identifier can be a user identifier, such as a user's mobile phone number, ID card number, passport number, etc.

[0061] Based on the first batch of sample identifiers obtained above, P i Fang uses the first model to process the first batch of local features and obtains the first output o. i It's important to understand that the first model can be derived from P. i The model is defined based on local features, and its parameters can be initialized automatically after definition. Furthermore, the first model includes several hidden layers, and its output is essentially the hidden layer representation. In one example, the first model includes several fully connected layers and several activation layers. Accordingly, the first model can be used to perform linear and nonlinear transformations on the first batch of local features to obtain the first output o. i In another example, the first model includes multiple convolutional layers. Accordingly, the first model can be used to convolve the first batch of local features to obtain the first output o. i .

[0062] The first output o can be obtained from the above. i Next, in step S220, the first output o i Send to tag P y At this time, P i Fang Zhong uses the first batch of local features to update the first model M i The process is interrupted (or paused), and it is necessary to receive P. y The process continues only after the gradient is returned.

[0063] In step S230, based on several second batches of local features, contrastive learning is used to refine the first model M. i Perform the first update. It's important to understand that the first model M... i Model parameters θ before the first update u The first model M used in step S210 above i Model parameters θ i Since they are the same, they can both be denoted as w0.

[0064] Regarding the timing of this step, in one implementation, it can be performed after the aforementioned step S220. In another implementation, the timing of this step is independent of step S220. Specifically, split learning and contrastive learning can be considered as two relatively independent tasks, thereby employing distributed training. The operators performing these two tasks are distributed across multiple hardware devices for concurrent computation, thus improving computational efficiency. In one example, P... i The method deploys a first computing node and a second computing node. This step is executed on the first computing node, while other steps in the method, such as the aforementioned steps S210 and S220, are executed on the second computing node. It should be noted that the computing node in this text can be a computing chip or a cluster of devices, etc.

[0065] It's important to understand that contrastive learning is a self-supervised algorithm, or rather, an unsupervised algorithm. Therefore, the aforementioned second batch of local features can be derived from P. i It is obtained by randomly sampling the local feature set.

[0066] In this step, for any batch of features from several second batches of local features, data augmentation is first performed to obtain augmented features. Specifically, for the local features corresponding to a single sample identifier in any batch of features, in one embodiment, multiple feature values ​​are randomly masked to obtain the corresponding augmented features. It should be understood that masking refers to modifying the original feature values ​​to default values ​​(such as null), and the number of masked feature values ​​can be predetermined, such as 10 or 15. In another embodiment, a noise vector of the same dimension is superimposed on the local features, thereby using the noise-added result as the corresponding augmented features. For example, the noise vector can be a Gaussian noise vector or a Laplacian noise vector. In addition, it should be noted that for the same local feature, it is usually subjected to two augmentation processes to obtain two different augmented features.

[0067] Next, at least the first model M is used. i The feature set including the augmented features is processed to obtain the representation output of each feature. The feature set may include only the augmented features or it may include the original local features of the corresponding batch. In one embodiment, a first model M is used. i Each feature in the feature set is processed to obtain the first output of each feature, which serves as the corresponding representation output. It should be understood that each feature refers to a set of features corresponding to a single sample identifier, such as a local feature set or an augmented feature set. In another embodiment, after obtaining the first output of each feature, a locally deployed second model is used to process it to obtain the second output of each feature, which serves as the corresponding representation output. The second model can be P... i The second model can be customized to further process the hidden layer representations output by the first model, resulting in deeper representations. For example, the first model can be called the basic encoding model, and the second model the projection model. Furthermore, the second model can be designed to be more complex than the first model, for example, by including more hidden layers.

[0068] Then, the contrastive loss L is calculated using the representation outputs of each feature in the feature set. con It should be understood that the mathematical form of the output representation is generally a vector, and the feature set corresponds to the output representation set. In one embodiment, the contrastive loss L... con The correlation is positively correlated with the distance between vectors corresponding to the same local features in the representation output set, and negatively correlated with the distance between vectors corresponding to different local features. For example, representation output pairs corresponding to the same local features and representation output pairs corresponding to different local features can be constructed first based on the representation output set. Then, the contrastive loss described above is calculated based on the two constructed representation output pairs. It is understood that the distance between vectors described in this paper can be measured using Euclidean distance, Chebyshev distance, etc.

[0069] Then, based on the comparative loss L con At least update the first model M i In one embodiment, the above representation output is the second output generated by the second model, in which case the second model is also updated.

[0070] The above describes how to update the first model using any batch of local features from several second batches of local features. Similarly, by using several second batches of local features, the first model M can be updated. i The process involves several updates, which are collectively referred to as the first update. Intuitively, the first model M in the first update can be represented as... i Model parameters θ i The process of change is denoted as: w0→w1→…→w n-1 where w refers to θ u , where n is an integer greater than 1.

[0071] Therefore, unsupervised contrastive learning can be used to locally improve the performance of the first model M. i The first update, the corresponding model parameters θ u From w0 to w n-1 .

[0072] Then, in step S240, from the tag P y Receive the returned gradient g i It is based on the first output of each data source, the sample label corresponding to the first batch of sample identifiers, and the label source P. y The target model M deployed in China t And that's certain.

[0073] For ease of understanding, let's talk about P. y Fang determines the backpropagation gradient g i This will be briefly introduced in a certain way. Specifically, P y Fang first bases the first output of each data source on the target model M. t Determine the prediction result. In one implementation scenario, the labeler P... y There are no sample features in the data. Furthermore, when the aforementioned data points are single data points, P... y Fang can utilize the target model M t Process the first output from a single data square to obtain the prediction result. In the case where the aforementioned data parties are multiple data parties, P y Fang can first perform fusion processing on multiple first outputs from multiple data sources, such as concatenation, summation, and averaging, and then use the target model M. t The fusion results are processed to obtain the prediction results.

[0074] In another implementation scenario, P y Fang also possesses some features corresponding to the identifiers of the first batch of samples mentioned above. It can be understood that these features are P. y The local or private characteristics of the labeler; in this case, the labeler P y The system can first use its locally deployed first model to process some features and obtain the first output. Then, it can fuse this first output with several first outputs received from several data sources, such as by concatenation, summation, or averaging. Finally, it can use the target model M. t The fusion results are processed to obtain the prediction results.

[0075] Furthermore, P y Based on the prediction results Calculate the split learning loss L with the corresponding sample label Y. sp Then, based on the training loss L sp The backpropagation method is used to calculate the value returned to P. i gradient g of the square i In one embodiment, P y Fang also utilized training loss L sp Update target model M t In another embodiment, P y Fang also utilized training loss L sp Update the first model deployed locally.

[0076] The above introduces P. y Fang determines the backpropagation gradient g i In this way, P i Fang can come from P y Fang receives the returned gradient g i .

[0077] Therefore, in step S250, based on the backpropagated gradient g i For the first model M after the first update mentioned above i Perform the second update. As mentioned before, the model parameters of the first model after the first update are denoted as w. n-1 Therefore, the parameter change sent in the second update can be denoted as w. n-1 →w n .

[0078] In one embodiment, P i The first and second computing nodes are deployed as described above. At this time, steps S240 and this step can also be executed on the second computing node. In one example, the second computing node responds to the backpropagated gradient g. i The system receives the latest model parameters from the first computing node, which correspond to the model parameters w updated in the first iteration. n-1In another example, the number of iterations involved in the first update is pre-set. In this case, the second compute node can receive the model parameters w automatically synchronized by the first compute node. n-1 .

[0079] On the other hand, the first model M i Includes several layers of neural networks, P i The received return gradient g i In reality, the gradient is backpropagated to the last layer of the neural network. In the case of a multi-layer neural network, the gradient backpropagation needs to continue.

[0080] Furthermore, for each layer in a multi-layered neural network, it is necessary to first determine the gradient (or parameter gradient) of the weight parameters in that layer using the propagation gradient and the input of that layer, and then use the parameter gradient of that layer to update the model parameters in that layer. This can be written as:

[0081]

[0082]

[0083] In equation (1), Let x represent the gradient of the parameters of the j-th layer of the neural network. j This represents the input to the j-th layer of the neural network. It should be understood that this input is obtained in step S210 using the first model M. i (At this point, the model parameters are w0) This is the input to the i-th layer of the neural network when processing the first batch of local features. In fact, the execution of step S210 can also include storing these inputs, thus facilitating computation in this step. It can be called directly at that time. j Let M represent the propagation gradient to the j-th layer of the neural network. i The number of neural network layers included in g is denoted as k. At this point, g... k =g i .

[0084] In equation (2), w j This refers to the model parameters in the j-th layer of the neural network; α represents the learning rate, which is a hyperparameter, and can be manually set to 0.002.

[0085] It should be noted that for the first term w on the right side of the equal sign in equation (2) j Its parameters w are derived from the model parameters after the first update mentioned above. n-1 The corresponding value is obtained from the middle. For w on the left side of the equal sign in equation (2), j This forms the model parameters w after the second update. n .

[0086] From the above, we can utilize the backpropagation gradient g i For the first model M after the first update i A second update is performed to prevent the parameters of the first model deployed locally by each party from diverging during the comparative learning process, thereby improving the overall convergence speed and training efficiency.

[0087] According to another embodiment, when the first update and the second update are performed in a distributed manner on the first computing node and the second computing node, the second computing node can also transfer the second updated model parameters w n Synchronize with the first compute node, then you can start accessing it. Figure 2 The text indicates the next execution of the method, w. n This is used as the starting parameter w0 in the next execution. It is understandable that this is achieved through repeated iterative executions. Figure 2 The method shown in the diagram continues until the parameters of each model converge, resulting in the final trained model. In practical applications, the first model from each party and the target model from the label party are required. For cases involving a second model during training, since the second model is used to assist training, it can be discarded after training is complete.

[0088] In summary, the multi-party collaborative model training method disclosed in the embodiments of this specification introduces unsupervised contrastive learning to update the locally deployed first model in the data party. This ensures that the data party is not idle during the period between calculating the first output corresponding to the sample identifier agreed upon by the multi-party parties and receiving the backpropagated gradient, thereby improving the overall training efficiency. Furthermore, the first model is updated at regular intervals using the gradient returned by the label party to prevent the parameters of the first model from diverging, thereby improving the overall convergence speed and thus improving the training efficiency.

[0089] The above introduces the joint training scheme applied to vertical splitting scenarios. Next, we will introduce the joint training scheme applied to horizontal splitting scenarios. Figure 3 The diagram illustrates a method for multi-party joint model training according to another embodiment. The multi-party includes multiple data parties and a neutral party. Each of the multiple data parties has different labeled training samples. It can be understood that the neutral party is neutral and trustworthy compared to the multiple data parties. For example, it may be an authoritative institution.

[0090] For training samples in horizontal scenarios, in image classification scenarios, the training samples can be the original images, and their corresponding sample labels are the image category labels. Furthermore, in more fine-grained image classification scenarios, such as object recognition scenarios, the sample labels include the bounding box and object category (e.g., car, tree, pedestrian) for the target object in the original image; similarly, in face recognition scenarios, the original image is a face image, and the image category label is the user's identity identifier (e.g., phone number).

[0091] In text classification scenarios, training samples can be raw text, and their corresponding sample labels are text categories. For example, in content recommendation scenarios, text categories can include technology, music, social news, etc.

[0092] Figure 3 The method shown applies to any data square (denoted as P). j Fang), such as Figure 4 As shown, the method includes the following steps:

[0093] First, in step S310, the first batch of local training samples is processed using the locally deployed first model to obtain the first output. It should be noted that the first model can be defined through negotiation among multiple data providers. For example, after defining the first model, one data provider sends the definition information, including the model structure, to other data providers.

[0094] It is understandable that the first batch in this step can be sampled by the data provider themselves.

[0095] In one embodiment, the local training samples are image samples, and the first model is implemented based on Convolutional Neural Networks (CNN). In this case, the first output can be the feature map of the image.

[0096] In one embodiment, the local training samples are text, and the first model is implemented based on DNN networks or recurrent neural networks (RNNs), etc. In this case, the first output can be a text representation.

[0097] From the above, we can obtain the first output.

[0098] Next, in step S320, the first output is sent to the neutral party. At this time, P j The process of updating the first model using the first batch of samples is paused and needs to be resumed only after receiving the gradient returned by the neutral square.

[0099] Step S330: Based on several second batches of local training samples, the first model is updated using contrastive learning. In one embodiment, first and second computing nodes are distributed across any data provider. Figure 3 The first update in the method is performed on the first computing node, and the other steps are performed on the second computing node.

[0100] In this step, data augmentation can be performed on any batch of training samples from several second batches to obtain augmented samples. In one embodiment, the data augmentation process can be noise addition, where the added noise can be obtained by randomly sampling the noise distribution. In another embodiment, the training samples are image samples, and in this case, the data augmentation process can also be rotation, cropping, grayscale processing, and blurring, etc.

[0101] Next, the sample set including the augmented sample is processed using at least the first model to obtain the representation output of each sample. In one embodiment, the sample set also includes any of the batches of training samples mentioned above. In one embodiment, the first model can be used to process each sample to obtain a first output for each sample, which serves as the corresponding representation output. In another embodiment, a locally deployed second model can be used to further process the first output of each sample to obtain a second output for each sample, which serves as the corresponding representation output.

[0102] Then, the contrastive loss is calculated based on the representation output. It should be understood that this contrastive loss is negatively correlated with the vector distance between representation output pairs corresponding to different training samples, and positively correlated with the vector distance between representation output pairs corresponding to the same training sample.

[0103] Then, the first model is updated at least based on the contrastive loss. In one embodiment, the second model is also updated using the contrastive loss.

[0104] In summary, comparative learning can be used to achieve the first update of the first model.

[0105] In step S340, a return gradient is received from the neutral party, the return gradient being determined based on the first output, sample labels, and the target model deployed in the neutral party.

[0106] In one embodiment, prior to this step, the method may further include sending sample labels corresponding to the first batch of local training samples to the neutral party. For example, in step S320 above, in addition to sending the first output to the neutral party, the corresponding sample labels are also sent. Thus, the neutral party can process the first output using the target model to obtain a prediction result, then calculate the training loss based on the prediction result and the sample labels, and finally calculate the backpropagation gradient based on the training loss.

[0107] In another embodiment, prior to this step, the method may further include: receiving a prediction result from a neutral party, the prediction result being obtained by processing a first output using a target model; calculating a training loss based on the prediction result and the corresponding sample labels, and sending this training loss to the neutral party so that the neutral party can calculate a backpropagation gradient based on this training loss.

[0108] Understandably, the neutral party also calculates the parameter gradient of the target model based on the training loss, and then updates the target model according to this parameter gradient.

[0109] Then, in step S350, based on the backpropagated gradient, the first model that has undergone the first update is updated a second time.

[0110] In one embodiment, the first model includes a multi-layer neural network, and step S310 further includes: storing the input of each layer of the neural network when processing the first batch of local training samples using the first model; based on this, this step may include: determining the parameter gradient of each layer of the neural network based on the backpropagation gradient and the input of each layer of the neural network, and updating the model parameters in each layer of the neural network according to the parameter gradient.

[0111] According to another embodiment, P j The updated model parameters are sent to one of the other data providers among multiple data providers, so that the data provider uses the updated model parameters as the initial parameters for its model, thereby training its model. For example, the data provider can use the updated model parameters as the model parameters for its locally deployed first model, thereby executing... Figure 3 The method shown in the figure.

[0112] It should be noted that, for Figure 3 For an introduction to the Chinese method, please refer to the section on Figure 2 Description of the method.

[0113] In summary, the multi-party joint model training method disclosed in the embodiments of this specification employs contrastive learning in any data party to perform unsupervised training on the locally deployed first model. This eliminates the need for the data party to idly wait for gradients returned from the neutral party, thereby improving overall training efficiency. Furthermore, at regular intervals, the first model is updated using gradients returned from the neutral party to prevent model parameters from diverging, thus improving overall convergence speed and training efficiency.

[0114] Corresponding to the methods described above, this specification also discloses a training device in its embodiments. Figure 4The diagram illustrates a device structure for multi-party joint model training according to one embodiment, wherein the multi-party includes a label party possessing sample labels and several data parties possessing sample features; the device 400 is integrated into any one of the data parties. Figure 4 As shown, the device 400 includes:

[0115] The first model processing unit 410 is configured to process the first batch of local features corresponding to the first batch of sample identifiers agreed upon by multiple parties using the locally deployed first model to obtain a first output; the sending unit 420 is configured to send the first output to the labeling party; the first update unit 430 is configured to perform a first update on the first model based on several second batch of local features using contrastive learning; the receiving unit 440 is configured to receive the feedback gradient from the labeling party, which is determined based on the first output of each data party, the sample label corresponding to the first batch of sample identifiers, and the target model deployed in the labeling party; the second update unit 450 is configured to perform a second update on the first model after the first update based on the feedback gradient.

[0116] In one embodiment, the labeler also possesses partial features corresponding to the identifiers of the first batch of samples, and the target model is used to fuse the first output of the labeler and the first outputs of each data party, and obtain a prediction result based on the fusion result; the first output of the labeler is obtained by processing the partial features using its first model.

[0117] In one embodiment, the plurality of data parties are multiple data parties, and the target model is used to fuse the first outputs of each data party and obtain a prediction result based on the fusion result.

[0118] In one embodiment, the device 400 has a first computing node and a second computing node deployed in a distributed manner; the first update unit 430 in the device 400 is integrated in the first computing node, and the other units are integrated in the second computing node.

[0119] In one specific embodiment, the device 400 further includes a synchronization unit 460 configured to synchronize the model parameters updated by the first update on the first computing node to the second computing node.

[0120] In one embodiment, the first update unit 430 includes: an augmentation subunit 431 configured to perform data augmentation processing on any batch of features in the plurality of second batches of local features to obtain augmented features; a prediction subunit 432 configured to process a feature set including the augmented features using at least the first model to obtain a representation output of each feature therein; a loss calculation subunit 433 configured to calculate a contrastive loss using the representation output; and an update subunit 434 configured to update at least the first model based on the contrastive loss.

[0121] In a specific embodiment, the augmentation subunit 431 is specifically configured to: perform random masking on multiple feature values ​​of the local feature corresponding to a single sample identifier to obtain the corresponding augmented feature.

[0122] In a specific embodiment, the prediction subunit 432 is specifically configured to: process each feature using the first model to obtain a first output for each feature; process the first output of each feature using a locally deployed second model to obtain a second output for each feature, which serves as the representation output; and update at least the first model according to the contrast loss, including: updating the first model and the second model using the contrast loss.

[0123] In one embodiment, the first model includes several layers of neural networks, and the first model processing unit 410 is further configured to: store the input of each layer of the neural network when processing the first batch of local features using the first model. The second update unit 450 is specifically configured to: determine the parameter gradient of each layer of the neural network based on the backpropagation gradient and the input of each layer of the neural network, and update the model parameters in each layer of the neural network according to the parameter gradient.

[0124] Figure 5 A schematic diagram of an apparatus for multi-party joint model training according to another embodiment is shown, wherein multiple data parties among the multiple parties each possess different labeled training samples, and the multiple parties also include a neutral party; the apparatus 500 is integrated into any one of the data parties. Figure 5 As shown, the device 500 includes:

[0125] The first model processing unit 510 is configured to process the first batch of local training samples using a locally deployed first model to obtain a first output; the sending unit 520 is configured to send the first output to the neutral party; the first update unit 530 is configured to perform a first update on the first model using contrastive learning based on several second batches of local training samples; the receiving unit 540 is configured to receive a backpropagation gradient from the neutral party, the backpropagation gradient being determined based on the first output, sample labels, and the target model deployed in the neutral party; and the second update unit 550 is configured to perform a second update on the first model after the first update based on the backpropagation gradient.

[0126] In one embodiment, the apparatus 500 further includes a transmission unit 560 configured to send the second updated model parameters to one of the plurality of data parties, so that the data party uses the second updated model parameters as the initial parameters of its model, thereby training its model.

[0127] In one embodiment, a first computing node and a second computing node are deployed in a distributed manner in any of the data parties, and the first update in the method is performed on the first computing node, while the other steps are performed on the second computing node.

[0128] In one specific embodiment, the device 500 further includes a synchronization unit 570, configured to synchronize the model parameters updated by the first update on the first computing node to the second computing node.

[0129] In one embodiment, the first update unit 530 includes: an augmentation subunit 531 configured to perform data augmentation processing on any batch of training samples in the plurality of second batches of training samples to obtain augmented samples; a prediction subunit 532 configured to process at least the sample set including the augmented samples using the first model to obtain the representation output of each sample therein; a loss calculation subunit 533 configured to calculate the contrastive loss using the representation output; and an update subunit 534 configured to update at least the first model based on the contrastive loss.

[0130] In a specific embodiment, the augmentation subunit 531 is specifically configured to: add noise to each training sample to obtain the corresponding augmented sample.

[0131] In a specific embodiment, the prediction subunit 532 is specifically configured to: process each sample using the first model to obtain a first output for each sample; process the first output of each sample using a locally deployed second model to obtain a second output for each sample, which serves as the representation output; and update at least the first model according to the contrast loss, including: updating the first model and the second model using the contrast loss.

[0132] In one embodiment, the first model includes a multi-layer neural network, and the first model processing unit 510 is further configured to: store the inputs of each layer of the neural network when processing the first batch of local training samples using the first model. The second update unit 550 is specifically configured to: determine the parameter gradients of each layer of the neural network based on the backpropagation gradients and the inputs of each layer of the neural network, and update the model parameters in each layer of the neural network according to the parameter gradients.

[0133] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 or Figure 3 The method described.

[0134] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 or Figure 3 The methods described herein. Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0135] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for joint training of a model by multiple parties, wherein the multiple parties include a label party that possesses sample labels in the recommended samples and several data parties that possess user features in the recommended samples, wherein the sample labels indicate whether a user has performed a predetermined action on a business object; The method is applied to any one of the data parties, including: The local user features corresponding to the first batch of sample identifiers agreed upon by multiple parties are processed using the first model deployed locally to obtain the first output, and the first output is sent to the tagging party. Based on several second-batch local user features, the first model is updated using contrastive learning. The gradient is received from the tag provider and is determined based on the first output of each data provider, the sample label corresponding to the first batch of sample identifiers, and the target model deployed in the tag provider. Based on the backpropagated gradient, the first model that has undergone the first update is updated a second time. The labeler also possesses a batch of object features corresponding to the first batch of sample identifiers. The target model is used to fuse the first output of the labeler and the first output of each data party, and obtain a prediction result based on the fusion result. The first output of the labeler is obtained by processing the batch of object features using its first model.

2. The method according to claim 1, wherein, The data sources are multiple data sources, and the target model is used to fuse the first outputs of each data source and obtain a prediction result based on the fusion result.

3. The method according to claim 1, wherein, In any one of the data parties, a first computing node and a second computing node are deployed in a distributed manner; in the method, the first update is performed on the first computing node, and the other steps are performed on the second computing node.

4. The method according to claim 3, further comprising: The updated model parameters on the first computing node are synchronized to the second computing node.

5. The method according to claim 1, wherein, The first update includes: Data augmentation processing is performed on any batch of user features from the aforementioned second batches of local user features to obtain augmented user features; At least the first model is used to process the user feature set including the augmented user features to obtain the representation output of each user feature therein; The contrast loss is calculated using the aforementioned characterization output; Based on the comparative loss, at least the first model is updated.

6. The method according to claim 5, wherein, Data augmentation processing is performed on any batch of user features from the aforementioned second batches of local user features to obtain augmented user features, including: For the local user features corresponding to a single sample identifier, multiple user feature values ​​are randomly masked to obtain the corresponding augmented user features.

7. The method according to claim 5, wherein, At least the first model is used to process the user feature set including the augmented user features to obtain the representation output of each user feature, including: The first model is used to process each user feature to obtain the first output of each user feature; The first output of each user feature is processed using a locally deployed second model to obtain the second output of each user feature, which is used as the representation output. Updating at least the first model based on the contrastive loss includes updating the first model and the second model using the contrastive loss.

8. The method according to claim 1, wherein, The first model includes several layers of neural networks, and the method further includes: When using the first model to process the first batch of local user features, the input of each neural network layer is stored. The second update includes: Based on the backpropagation gradient and the input of each neural network layer, the parameter gradient of each neural network layer is determined, and the model parameters in each neural network layer are updated according to the parameter gradient.

9. A method for joint model training by multiple parties, wherein each of the multiple data parties possesses different training images with image category labels, and the multiple parties also include a neutral party; the method is applied to any one of the data parties, comprising: The first batch of locally trained images is processed using the locally deployed first model to obtain a first output, and the first output is sent to the neutral party. Based on several second batches of local training images, the first model is updated using contrastive learning. The return gradient is received from the neutral party, and the return gradient is determined based on the first output, the image category label, and the target model deployed in the neutral party; Based on the backpropagated gradient, the first model that has undergone the first update is updated a second time.

10. The method of claim 9, further comprising: The updated model parameters are sent to one of the other data providers among the plurality of data providers, so that the data provider uses the updated model parameters as the initial parameters of its model and trains its model.

11. The method according to claim 9, wherein, In any one of the data parties, a first computing node and a second computing node are deployed in a distributed manner. In the method, the first update is performed on the first computing node, and the other steps are performed on the second computing node.

12. The method of claim 11, further comprising: The updated model parameters on the first computing node are synchronized to the second computing node.

13. The method according to claim 9, wherein, The first update includes: Data augmentation processing is performed on any batch of training images from the plurality of second batches of training images to obtain augmented samples; At least the first model is used to process the sample set including the augmented samples to obtain the representation output of each sample therein; The contrast loss is calculated using the aforementioned characterization output; Based on the comparative loss, at least the first model is updated.

14. The method according to claim 13, wherein, Data augmentation processing is performed on any batch of training images from the plurality of second batches to obtain augmented samples, including: For each training image, noise is added to it to obtain the corresponding augmented sample.

15. The method according to claim 13, wherein, At least the first model is used to process the sample set including the augmented samples to obtain the representation output of each sample therein, including: The first model is used to process each sample to obtain the first output of each sample; The first output of each sample is processed using a locally deployed second model to obtain the second output of each sample, which is used as the representation output. Updating at least the first model based on the contrastive loss includes updating the first model and the second model using the contrastive loss.

16. The method according to claim 9, wherein, The first model includes a multi-layer neural network, and the method further includes: When processing the first batch of local training images using the first model, the inputs of each neural network layer are stored. The second update includes: Based on the backpropagation gradient and the input of each neural network layer, the parameter gradient of each neural network layer is determined, and the model parameters in each neural network layer are updated according to the parameter gradient.

17. An apparatus for joint training of a model by multiple parties, the multiple parties including a label party having sample labels in the recommended samples and several data parties having user features in the recommended samples, the sample labels indicating whether a user has performed a predetermined action on a business object; The device is integrated into any one of the data sides, including: The first model processing unit is configured to use the locally deployed first model to process the first batch of local user features corresponding to the first batch of sample identifiers agreed upon by multiple parties, and obtain the first output. The sending unit is configured to send the first output to the tag. The first update unit is configured to perform a first update on the first model based on several second batches of local user features and using comparative learning. The receiving unit is configured to receive the returned gradient from the tag party, which is determined based on the first output of each data party, the sample label corresponding to the first batch of sample identifiers, and the target model deployed in the tag party; The second update unit is configured to perform a second update on the first model that has undergone the first update based on the backpropagated gradient. The labeler also possesses a batch of object features corresponding to the first batch of sample identifiers. The target model is used to fuse the first output of the labeler and the first output of each data party, and obtain a prediction result based on the fusion result. The first output of the labeler is obtained by processing the batch of object features using its first model.

18. An apparatus for joint model training by multiple parties, wherein multiple data parties among the multiple parties each possess different training images with image category labels, and the multiple parties further include a neutral party; the apparatus is integrated into any one of the data parties, comprising: The first model processing unit is configured to process the first batch of locally trained images using the locally deployed first model to obtain the first output; The sending unit is configured to send the first output to the neutral party; The first update unit is configured to perform a first update on the first model based on several second batches of local training images, using contrastive learning. The receiving unit is configured to receive a return gradient from the neutral party, the return gradient being determined based on the first output, sample labels, and the target model deployed in the neutral party; The second update unit is configured to perform a second update on the first model that has undergone the first update, based on the backpropagated gradient.

19. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method of any one of claims 1-16.

20. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-16.

Images