A method for training a longitudinal federated learning model and related apparatuses

Abstract

The application discloses a training method of a longitudinal federated learning model and a related device, relates to the technical field of data security, and aims to improve the utilization capability of a longitudinal federated learning model for feature data and guarantee the overall performance of the longitudinal federated learning model. The method comprises the following steps: firstly, obtaining first data samples of a task initiator and second data samples of a data provider, and obtaining common data samples of the task initiator and the data provider based on the first data samples and the second data samples; then, obtaining first data features corresponding to the task initiator and second data features corresponding to the data provider based on the common data samples respectively; further, performing feature fusion on the first data features and the second data features to obtain fused data features; finally, performing multiple times of training on a longitudinal federated learning model to be trained based on the first data features, the second data features and the fused data features, so as to obtain a target longitudinal federated learning model.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of data security, and particularly relates to a training method of a vertical federated learning model and a related device. BACKGROUND

[0002] With the development of the field of privacy protection computing, federated learning gradually becomes a research focus. The federated learning allows data to be distributed in different devices and networks and invisible to each other, and solves the security problems such as privacy leakage existing in data cooperation tasks.

[0003] The vertical federated learning models different data types of data, and therefore, the vertical federated learning model can contain data of different data types. In addition, in the vertical federated learning model, one participant provides task label information of data samples, which is called a task initiator, and participants other than the task initiator are called data providers. Through the vertical federated learning model, the task initiator can more comprehensively utilize the feature information of the data of the task initiator itself and the feature information of the data owned by the data providers under the premise of ensuring data security, and improve the prediction accuracy of the vertical federated learning model.

[0004] In related technologies, the structure of the vertical federated learning model can be divided into three layers of upper, middle and lower, and the data provided by the task initiator and the data provided by the data providers are directly aggregated after being calculated by the lower layer model. However, the data of the task initiator and the data of the data providers do not interact in the shallow network of the lower layer model, which will cause the lower layer model to lose part of the available information during training.

[0005] Specifically, the data provided by the task initiator and the data provided by the data providers are calculated in the respective corresponding lower layer models, and then the feature fusion is performed in the middle layer model to obtain fusion feature data, and then the fusion feature data is aggregated by the upper layer model.

[0006] However, in this process, the latent vectors of the respective feature data of the task initiator and the task participant will be lost with the increase of the number of layers of the upper layer model when the fusion feature data is transmitted downward level by level, which will cause part of the latent vectors of the fusion feature data to be lost. Therefore, the available information in the fusion feature data will be less and less, and thus the feature data available for the vertical federated learning model obtained finally is incomplete, which cannot guarantee the overall performance of the vertical federated learning model. SUMMARY

[0007] The present application provides a training method of a vertical federated learning model and a related device to improve the utilization ability of the vertical federated learning model for feature data, and thus guarantee the overall performance of the vertical federated learning model.

[0008] Firstly, this application provides a method for training a longitudinal federated learning model, including:

[0009] Obtain the first data sample from the task initiator and the second data sample from the data provider, and based on the first and second data samples, obtain the shared data sample between the task initiator and the data provider;

[0010] Based on the shared data samples, the first data feature corresponding to the task initiator and the second data feature corresponding to the data provider are obtained respectively; wherein, the first data feature and the second data feature respectively represent different feature information of the shared data samples;

[0011] The first data feature and the second data feature are fused to obtain the fused data feature.

[0012] Based on the first data features, the second data features, and the fused data features, the longitudinal federated learning model to be trained is trained multiple times to obtain the target longitudinal federated learning model.

[0013] In one optional implementation, the longitudinal federated learning model to be trained is trained multiple times based on the first data features, the second data features, and the fused data features to obtain the target longitudinal federated learning model, including:

[0014] Calculate the first loss value corresponding to the first data feature, the second loss value corresponding to the second data feature, and the fusion loss value corresponding to the fused data feature, respectively.

[0015] Partial derivatives are performed on the first loss value, the second loss value, and the fusion loss value to obtain the first initial deviation value of the task initiator, the second initial deviation value of the data provider, and the fusion deviation value.

[0016] The first weighted deviation value of the task initiator is obtained by weighting the fusion loss value and the first initial deviation value, and the second weighted deviation value of the data provider is obtained by weighting the fusion loss value and the second initial deviation value.

[0017] Based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value, the parameters of the longitudinal federated learning model to be trained are updated to obtain the target longitudinal federated learning model.

[0018] In one optional implementation, a first weighted deviation value for the task initiator is obtained by weighting the fusion loss value and the first initial deviation value, and a second weighted deviation value for the data provider is obtained by weighting the fusion loss value and the second initial deviation value, including:

[0019] Based on the fusion loss value and the first loss value, calculate the first loss weight corresponding to the first data feature, and based on the fusion loss value and the second loss value, calculate the second loss weight corresponding to the second data feature;

[0020] A first weighted deviation value is obtained based on a first initial deviation value and a first loss weight, and a second weighted deviation value is obtained based on a second initial deviation value and a second loss weight.

[0021] In one optional implementation, the parameters of the longitudinal federated learning model to be trained are updated based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value to obtain the target longitudinal federated learning model, including:

[0022] The parameters of the first underlying model in the longitudinal federated learning model to be trained are updated based on the first weighted bias value; where the first underlying model is set for the task initiator.

[0023] The parameters of the second-layer model in the longitudinal federated learning model to be trained are updated based on the second weighted bias value; whereby the second-layer model is set for the data provider.

[0024] The parameters of the middle-layer model of the longitudinal federated learning model to be trained are updated based on the fusion bias value, and the parameters of the upper-layer model of the longitudinal federated learning model to be trained are updated based on the fusion loss value, so as to obtain the target longitudinal federated learning model.

[0025] In one optional implementation, updating the parameters of the longitudinal federated learning model to be trained to obtain the target longitudinal federated learning model includes:

[0026] Update the parameters of the longitudinal federated learning model to be trained, and obtain the fusion loss value after parameter update;

[0027] If the fusion loss value after parameter update converges to the preset loss threshold, then the parameter-updated longitudinal federated learning model is taken as the target longitudinal federated learning model.

[0028] Secondly, this application provides a training apparatus for a longitudinal federated learning model, comprising:

[0029] The processing module is used to obtain the first data sample from the task initiator and the second data sample from the data provider, and based on the first data sample and the second data sample, obtain the common data sample between the task initiator and the data provider.

[0030] The feature conversion module is used to obtain the first data feature corresponding to the task initiator and the second data feature corresponding to the data provider based on the shared data samples; wherein the first data feature and the second data feature respectively represent different feature information of the shared data samples;

[0031] The feature fusion module is used to fuse the first data feature and the second data feature to obtain the fused data feature.

[0032] The training module is used to train the longitudinal federated learning model to be trained multiple times based on the first data feature, the second data feature, and the fused data feature, so as to obtain the target longitudinal federated learning model.

[0033] In one optional implementation, when the longitudinal federated learning model to be trained is trained multiple times based on the first data features, the second data features, and the fused data features to obtain the target longitudinal federated learning model, the training module is specifically used for:

[0034] Calculate the first loss value corresponding to the first data feature, the second loss value corresponding to the second data feature, and the fusion loss value corresponding to the fused data feature, respectively.

[0035] Partial derivatives are performed on the first loss value, the second loss value, and the fusion loss value to obtain the first initial deviation value of the task initiator, the second initial deviation value of the data provider, and the fusion deviation value.

[0036] The first weighted deviation value of the task initiator is obtained by weighting the fusion loss value and the first initial deviation value, and the second weighted deviation value of the data provider is obtained by weighting the fusion loss value and the second initial deviation value.

[0037] Based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value, the parameters of the longitudinal federated learning model to be trained are updated to obtain the target longitudinal federated learning model.

[0038] In an optional implementation, when performing a weighted calculation based on the fusion loss value and the first initial bias value to obtain the first weighted bias value for the task initiator, and performing a weighted calculation based on the fusion loss value and the second initial bias value to obtain the second weighted bias value for the data provider, the training module is specifically used for:

[0039] Based on the fusion loss value and the first loss value, calculate the first loss weight corresponding to the first data feature, and based on the fusion loss value and the second loss value, calculate the second loss weight corresponding to the second data feature;

[0040] A first weighted deviation value is obtained based on a first initial deviation value and a first loss weight, and a second weighted deviation value is obtained based on a second initial deviation value and a second loss weight.

[0041] In one optional implementation, when updating the parameters of the longitudinal federated learning model to be trained based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value to obtain the target longitudinal federated learning model, the training module is specifically used for:

[0042] The parameters of the first underlying model in the longitudinal federated learning model to be trained are updated based on the first weighted bias value; where the first underlying model is set for the task initiator.

[0043] The parameters of the second-layer model in the longitudinal federated learning model to be trained are updated based on the second weighted bias value; whereby the second-layer model is set for the data provider.

[0044] The parameters of the middle-layer model of the longitudinal federated learning model to be trained are updated based on the fusion bias value, and the parameters of the upper-layer model of the longitudinal federated learning model to be trained are updated based on the fusion loss value, so as to obtain the target longitudinal federated learning model.

[0045] In one optional implementation, when updating the parameters of the longitudinal federated learning model to be trained to obtain the target longitudinal federated learning model, the training module is specifically used for:

[0046] Update the parameters of the longitudinal federated learning model to be trained, and obtain the fusion loss value after parameter update;

[0047] If the fusion loss value after parameter update converges to the preset loss threshold, then the parameter-updated longitudinal federated learning model is taken as the target longitudinal federated learning model.

[0048] Thirdly, this application provides an electronic device, comprising:

[0049] Memory, used to store computer programs;

[0050] When the processor executes the computer program stored in the memory, it implements the steps of the training method for a vertical federated learning model described above.

[0051] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the training method for a longitudinal federated learning model described above.

[0052] Through the technical solutions in one or more of the above embodiments of this application, the embodiments of this application have at least the following beneficial effects:

[0053] In the training method of the vertical federated learning model provided in this application embodiment, firstly, a first data sample from the task initiator and a second data sample from the data provider are obtained. Based on the first and second data samples, a shared data sample between the task initiator and the data provider is obtained. This ensures that each party can only obtain the correct shared data and will not obtain any data from the other party outside the shared data, thus guaranteeing data security. Then, based on the shared data samples, a first data feature corresponding to the task initiator and a second data feature corresponding to the data provider are obtained respectively. Further, the first and second data features are fused to obtain fused data features. Finally, based on the first data features, the second data features, and the fused data features, the vertical federated learning model to be trained is trained multiple times to obtain the target vertical federated learning model.

[0054] In this approach, auxiliary learning models are added to the underlying models of both the task initiator and the data provider. The data features in the underlying models can interact fully without losing usable data features. This provides the vertical federated learning model with the ability to utilize data features. The underlying models can learn label information from multiple links, enhancing the ability to mine data and thus ensuring the overall performance of the vertical federated learning model.

[0055] For the various aspects of the second to fourth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect and the various possible solutions in the first aspect. They will not be repeated here. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0057] Figure 1 A schematic diagram illustrating the implementation process of a training method for a vertical federated learning model provided in this application embodiment;

[0058] Figure 2 A schematic diagram of the structure of a vertical federated learning model provided in an embodiment of this application;

[0059] Figure 3 A schematic diagram illustrating the implementation process of obtaining weighted bias values ​​for a training method of a longitudinal federated learning model provided in this application embodiment;

[0060] Figure 4A schematic diagram illustrating the implementation process of obtaining loss weights in a training method for a vertical federated learning model provided in this application embodiment;

[0061] Figure 5 A schematic diagram illustrating the implementation process of updating model parameters in a training method for a vertical federated learning model provided in this application embodiment;

[0062] Figure 6 A schematic diagram of the structure of a training device for a vertical federated learning model provided in an embodiment of this application;

[0063] Figure 7 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0065] It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0066] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0067] In related technologies, the structure of a vertical federated learning model can be divided into three layers: upper, middle, and lower. The data provided by the task initiator and the data provider are directly aggregated after being calculated by the lower-layer model. However, in the shallow network of the lower-layer model, the data from the task initiator and the data provider do not interact, which will cause the lower-layer model to lose some usable information during training.

[0068] Specifically, the data provided by the task initiator and the data provided by the data provider are calculated in their respective lower-level models, and then feature fusion is performed in the middle-level model to obtain fused feature data. Finally, the upper-level model directly aggregates the fused feature data.

[0069] However, during this process, as the number of upper-level model layers increases, the latent vectors of the various corresponding feature data of the task initiator and task participants will be lost when transmitting the fused feature data down level by level. As a result, the available information in the fused feature data will become less and less. Thus, the feature data that the final vertical federated learning model can use is incomplete, and the overall performance of the vertical federated learning model cannot be guaranteed.

[0070] To address the aforementioned technical problems, this application provides a training method for a vertical federated learning model. The method includes: first, acquiring a first data sample from the task initiator and a second data sample from the data provider, and obtaining shared data samples between the task initiator and the data provider based on the first and second data samples; then, obtaining a first data feature corresponding to the task initiator and a second data feature corresponding to the data provider based on the shared data samples; wherein the first and second data features respectively represent different feature information of the shared data samples; further, fusing the first and second data features to obtain fused data features; finally, training the vertical federated learning model to be trained multiple times based on the first, second, and fused data features to obtain the target vertical federated learning model. This method strengthens the information interaction of the underlying model, improves the vertical federated learning model's ability to utilize feature data, and thus ensures the overall performance of the vertical federated learning model. Compared to existing technologies, the data from the task initiator and the data provider will not lose usable information after being processed by the lower-level model.

[0071] It should be noted that the preferred embodiments of this application are described below with reference to the accompanying drawings. The preferred embodiments described herein are only for illustration and explanation and are not intended to limit this application. Furthermore, the embodiments of this application and the features thereof can be combined with each other without conflict.

[0072] See Figure 1 The diagram shown illustrates the implementation flow of a training method for a vertical federated learning model provided in this application. The specific implementation flow of this method is as follows:

[0073] S1: Obtain the first data sample from the task initiator and the second data sample from the data provider, and based on the first data sample and the second data sample, obtain the shared data sample between the task initiator and the data provider.

[0074] In this embodiment, the participants in the task and the data provider are included. For example, the data provider can be different organizations such as banks, supermarkets, and enterprises.

[0075] Since the data samples from the task participants and the data providers each have different feature spaces, the first data sample from the task initiator and the second data sample from the data provider are obtained separately. After obtaining the first data sample from the task participants and the second data sample from the data providers, the shared data sample between the task initiator and the data providers is determined through Privacy Set Intersection (PSI).

[0076] S2: Based on the shared data samples, obtain the first data feature corresponding to the task initiator and the second data feature corresponding to the data provider.

[0077] In the embodiments of this application, see Figure 2 As shown, the vertical federated learning model mainly consists of five parts: the underlying model of the task initiator, the underlying model of the data provider, the middle model, the upper model, the auxiliary learning model corresponding to the underlying model of the task initiator, and the auxiliary learning model corresponding to the underlying model of the data provider.

[0078] It should be noted that the underlying model of the task initiator and the underlying model of the data provider are of the same structure. They are primarily used for feature transformation based on shared data samples from both parties, resulting in the first data feature corresponding to the task initiator and the second data feature of the data provider. Furthermore, the first and second data features respectively represent different characteristic information of the shared data samples. For example, the first data feature of company A represents user characteristic information, while the second data feature of company B represents product characteristic information.

[0079] Specifically, the shared data samples are input into the underlying models of the task initiator and the data provider, respectively, to obtain the transformed first data features and second data features.

[0080] For example, the first data feature and the second data feature can be calculated using the following formula:

[0081]

[0082]

[0083] in, The first data feature corresponding to the task initiator. The second data feature corresponding to the data provider. For activation function, , These are a specific feature in the shared data sample, representing the characteristics of the task initiator and the data provider. and The feature extraction for the underlying neural networks of the task initiator and data provider is performed using a pre-defined set of weight parameters.

[0084] For example, by finding the intersection of privacy-preserving sets, we obtain a shared data sample I between the task initiator and the data provider. Among the shared data sample I, sample i has some data features owned by the task initiator. The data provider possesses another set of characteristics. .

[0085] Therefore, sample i from the shared data sample I is input into the underlying model of the task initiator and the underlying model of the data provider respectively for feature transformation to obtain the first data feature of the task initiator. and the second data features of the data provider .

[0086] S3: Perform feature fusion on the first data feature and the second data feature to obtain the fused data feature.

[0087] In this embodiment of the application, the first data feature of the task initiator and the second data feature of the data provider are fused to obtain the fused data feature.

[0088] Specifically, the order of the data features of the task initiator and the data provider is determined separately. The order of the first data features of the task initiator corresponds to the order of the second data features of the data provider. The first data features of the task initiator and the second data features of the data provider are then fused to obtain the fused data features.

[0089] For example, the first data feature of the task initiator is The second data feature of the second data provider is ,in, i This indicates the order of data features. Merging the first and second data features of the same order yields the merged data features.

[0090] Specifically, the fused data features can be calculated using the following formula:

[0091]

[0092] in, Indicates the fusion of data features, The concatenated vector representing the first data feature and the second data feature. This represents the preset weight parameter set used by the neural network in the intermediate layer model for feature extraction.

[0093] S4: Based on the first data feature, the second data feature, and the fused data feature, the vertical federated model to be trained is trained multiple times to obtain the target vertical federated learning model.

[0094] In one optional implementation, the longitudinal federated learning model to be trained is trained multiple times to obtain the target longitudinal federated learning model, see [reference]. Figure 3 As shown, this can be achieved through the following steps:

[0095] S301: Calculate the first loss value corresponding to the first data feature, the second loss value corresponding to the second data feature, and the fusion loss value corresponding to the fused data feature, respectively.

[0096] S302: Perform partial derivative operations on the first loss value, the second loss value, and the fusion loss value respectively to obtain the first initial deviation value of the task initiator, the second initial deviation value of the data provider, and the fusion deviation value.

[0097] S303: Based on the fusion loss value and the first initial deviation value, a weighted calculation is performed to obtain the first weighted deviation value of the task initiator, and based on the fusion loss value and the second initial deviation value, a weighted calculation is performed to obtain the second weighted deviation value of the data provider.

[0098] S304: Based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value, update the parameters of the longitudinal federated learning model to be trained to obtain the target longitudinal federated learning model.

[0099] Firstly, in practical implementation, the first data feature is input into the auxiliary learning model corresponding to the task initiator, and the predicted value of the first data feature is calculated. Then, the first loss value corresponding to the first data feature is determined based on the true value of the first data feature. Similarly, the second data feature is input into the auxiliary learning model corresponding to the data provider, and the predicted value of the second data feature is calculated. Then, the second loss value corresponding to the second data feature is determined based on the true value of the second data feature. Finally, the fused data feature is input into the upper-layer model, the predicted value of the fused data feature is calculated, and the fusion loss value of the fused data feature is determined.

[0100] Specifically, the predicted values ​​of the first and second data features of the task are calculated using the following formulas:

[0101]

[0102]

[0103] in, This represents the predicted value of the first data feature. This represents the predicted value of the second data feature, where sigmoid() is the activation function. This represents the preset weight parameter set of the auxiliary learning model corresponding to the task initiator when performing feature extraction. This represents the preset weight parameter set of the auxiliary learning model corresponding to the data provider when performing feature extraction.

[0104] Furthermore, based on the predicted values ​​of the first data feature and the second data feature, a first loss value corresponding to the first data feature and a second loss value corresponding to the second data feature are calculated.

[0105] In this embodiment, loss functions are set for the auxiliary learning layer of the task initiator, the auxiliary learning layer of the data provider, and the upper layer of the model, respectively. The first loss value of the first data feature, the second loss value of the second data feature, and the fusion loss value of the fused data feature are calculated by the corresponding loss functions.

[0106] Specifically, the first loss value of the first data feature, the second loss value of the second data feature, and the fusion loss value of the fused data feature are calculated using the following formula:

[0107]

[0108]

[0109]

[0110] in, Represents the actual value. This represents the first loss value representing the first data feature. The second loss value represents the second data feature. This represents the fusion loss value, which indicates the fusion features of the fused data.

[0111] Furthermore, partial derivatives are performed on the first loss value, the second loss value, and the fusion loss value to obtain the first initial deviation value of the task initiator, the second initial deviation value of the data provider, and the fusion deviation value.

[0112] In this embodiment, the first initial deviation value, the second initial deviation value, and the fusion deviation value are calculated using the following formula:

[0113]

[0114]

[0115]

[0116] in, Indicates the fusion deviation value. This represents the first initial bias value of the task initiator's auxiliary learning model. This represents the second initial bias value of the data provider's auxiliary learning model.

[0117] In some embodiments, the first initial deviation value of the task initiator's underlying model and the second initial deviation value of the data provider's underlying model can be calculated using the following formula:

[0118]

[0119]

[0120] in, This represents the first initial deviation value of the underlying model of the task initiator. This represents the second initial deviation value of the underlying model provided by the data provider.

[0121] In an optional implementation, in step S303, the process of performing a weighted calculation based on the fusion loss value and the first initial deviation value to obtain the first weighted deviation value of the task initiator, and performing a weighted calculation based on the fusion loss value and the second initial deviation value to obtain the second weighted deviation value of the data provider, is described in the following reference. Figure 4 As shown, this can be achieved through the following steps:

[0122] S401: Calculate the first loss weight corresponding to the first data feature based on the fusion loss value and the first loss value, and calculate the second loss weight corresponding to the second data feature based on the fusion loss value and the second loss value.

[0123] S402: Based on the first initial deviation value and the first loss weight, obtain the first weighted deviation value, and based on the second initial deviation value and the second loss weight, obtain the second weighted deviation value.

[0124] In practice, the calculation of the first loss weight and the second loss weight is mainly based on the comparison between the fusion loss and the first loss value of the task initiator's auxiliary learning model and the second loss value of the data provider's auxiliary learning model.

[0125] In this embodiment, the auxiliary learning model of the task initiator, the auxiliary model of the data provider, and the longitudinal federated learning model have the same training objective. If the loss value of the auxiliary learning model of one of the task initiators or the data provider is less than the loss value of the longitudinal federated learning model, the loss weight of the auxiliary learning model of the party with the smaller loss value is increased. If the loss value of the auxiliary learning model of one of the task initiators or the data provider is greater than the loss value of the longitudinal federated learning model, the loss weight of the auxiliary learning model of the party with the larger loss value is decreased.

[0126] By dynamically determining the loss weights, it is possible to maintain a balance among the layers in a longitudinal federated learning model when updating parameters.

[0127] First, calculate the difference between the first loss value and the fusion loss value, and the difference between the second loss value and the fusion loss value, respectively.

[0128] For example, the difference between the first loss value and the fusion loss value, and the difference between the second loss value and the fusion loss value, are calculated using the following formulas:

[0129]

[0130]

[0131] in, This represents the difference between the first loss value and the fusion loss value. This represents the difference between the second loss value and the fusion loss value, where T represents the preset range of the control loss value difference. This represents the fusion loss value. This represents the first loss value. This represents the second loss value.

[0132] Then, based on the difference between the first loss value and the fusion loss value, and the difference between the second loss value and the fusion loss value, the first loss weight and the second loss weight are calculated respectively.

[0133] For example, the first loss weight and the second loss weight are calculated using the following formulas:

[0134]

[0135]

[0136] Therefore, based on the first loss weight and the first initial deviation value, the first weighted deviation value of the task initiator is obtained, and based on the second loss weight and the second initial deviation value, the second weighted deviation value of the data provider is obtained. The calculation of the first weighted deviation value of the task initiator can then be as follows:

[0137]

[0138] The second weighted bias value of the data provider can be calculated as follows:

[0139]

[0140] in, This represents the first weighted deviation value of the task initiator. This represents the second weighted bias value from the data provider.

[0141] See Figure 5As shown, after obtaining the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value, the parameters of the longitudinal federated learning model to be trained are updated to obtain the target longitudinal federated learning model. This can be achieved through the following steps:

[0142] S501: Update the parameters of the first underlying model in the longitudinal federated learning model to be trained based on the first weighted bias value.

[0143] S502: Update the parameters of the second underlying model in the longitudinal federated learning model to be trained based on the second weighted bias value.

[0144] S503: Update the parameters of the middle-layer model of the longitudinal federated learning model to be trained based on the fusion bias value, and update the parameters of the upper-layer model of the longitudinal federated learning model to be trained based on the fusion loss value to obtain the target longitudinal federated learning model.

[0145] In this embodiment of the application, the first underlying model is set for the task initiator, and the second underlying model is set for the second underlying model.

[0146] In practice, after obtaining the first weighted average bias and the second weighted average bias, the gradients of the first and second bottom-level models are calculated by partial derivative operations. Then, the parameters of the first and second bottom-level models in the longitudinal federated learning model to be trained are updated.

[0147] The gradient of the intermediate model of the longitudinal federated learning model to be trained is obtained by performing partial derivative operation based on the fusion bias value. The parameters of the intermediate model of the longitudinal federated learning model to be trained are updated based on the gradient of the intermediate model.

[0148] In addition, the gradient of the upper-level model of the longitudinal federated learning model to be trained is obtained by calculating the partial derivative of the fusion loss value, and the parameters of the upper-level model of the longitudinal federated learning model to be trained are updated based on the gradient of the upper-level model.

[0149] In this embodiment, the gradients of the first bottom-layer model, the second bottom-layer model, the middle-layer model, and the upper-layer model are calculated using the following formulas:

[0150]

[0151]

[0152]

[0153]

[0154] in, This represents the gradient of the first layer model (model parameters / network parameters). This represents the gradient of the second-layer model (model parameters / network parameters). This represents the gradient of the mid-level model (model parameters / network parameters). This represents the gradient of the upper-level model (model parameters / network parameters).

[0155] Specifically, the gradient is the change in parameters, so after calculating the gradient of each layer of the model, the parameters can be updated.

[0156] In one optional implementation, after updating the parameters of the longitudinal federated learning model to be trained, the updated fusion loss value is obtained; if the updated fusion loss value converges to a preset loss threshold, the updated longitudinal federated learning model is used as the target longitudinal federated learning model.

[0157] Based on the method in steps S1-S4, this application adopts the idea of ​​relay supervision to add an auxiliary learning model to the underlying model of the longitudinal federated learning model. The loss values ​​are calculated for the underlying model of the task initiator and the underlying model of the data initiator, respectively, so as to obtain the first weighted bias value of the underlying model of the task initiator and the second weighted bias value of the data provider. This method can ensure the normal update of the parameters of the underlying model and solve the problem of gradient vanishing when the bias value of the upper model is updated through multiple backpropagation layers when the parameters of the entire model are updated directly.

[0158] Based on the same inventive concept, this application also provides a training device for a vertical federated learning model, see below. Figure 6 As shown, the device includes: a processing module 601, a feature conversion module 602, a feature fusion module 603, and a training module 604; wherein:

[0159] The processing module 601 is used to obtain a first data sample from the task initiator and a second data sample from the data provider, and to obtain a shared data sample between the task initiator and the data provider based on the first data sample and the second data sample.

[0160] The feature conversion module 602 is used to obtain the first data feature corresponding to the task initiator and the second data feature corresponding to the data provider based on the shared data samples; wherein the first data feature and the second data feature respectively represent different feature information of the shared data samples;

[0161] The feature fusion module 603 is used to fuse the first data feature and the second data feature to obtain the fused data feature.

[0162] Training module 604 is used to train the longitudinal federated learning model to be trained multiple times based on the first data feature, the second data feature, and the fused data feature to obtain the target longitudinal federated learning model.

[0163] In some optional embodiments, when the longitudinal federated learning model to be trained is trained multiple times based on the first data feature, the second data feature, and the fused data feature to obtain the target longitudinal federated learning model, the training module 604 is specifically used for:

[0164] Calculate the first loss value corresponding to the first data feature, the second loss value corresponding to the second data feature, and the fusion loss value corresponding to the fused data feature, respectively.

[0165] Partial derivatives are performed on the first loss value, the second loss value, and the fusion loss value to obtain the first initial deviation value of the task initiator, the second initial deviation value of the data provider, and the fusion deviation value.

[0166] The first weighted deviation value of the task initiator is obtained by weighting the fusion loss value and the first initial deviation value, and the second weighted deviation value of the data provider is obtained by weighting the fusion loss value and the second initial deviation value.

[0167] Based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value, the parameters of the longitudinal federated learning model to be trained are updated to obtain the target longitudinal federated learning model.

[0168] In some optional embodiments, when performing a weighted calculation based on the fusion loss value and the first initial deviation value to obtain the first weighted deviation value of the task initiator, and performing a weighted calculation based on the fusion loss value and the second initial deviation value to obtain the second weighted deviation value of the data provider, the training module 604 is specifically used for:

[0169] Based on the fusion loss value and the first loss value, calculate the first loss weight corresponding to the first data feature, and based on the fusion loss value and the second loss value, calculate the second loss weight corresponding to the second data feature;

[0170] A first weighted deviation value is obtained based on a first initial deviation value and a first loss weight, and a second weighted deviation value is obtained based on a second initial deviation value and a second loss weight.

[0171] In some optional implementations, when updating the parameters of the longitudinal federated learning model to be trained based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value to obtain the target longitudinal federated learning model, the training module 604 is specifically used for:

[0172] The parameters of the first underlying model in the longitudinal federated learning model to be trained are updated based on the first weighted bias value; where the first underlying model is set for the task initiator.

[0173] The parameters of the second-layer model in the longitudinal federated learning model to be trained are updated based on the second weighted bias value; whereby the second-layer model is set for the data provider.

[0174] The parameters of the middle-layer model of the longitudinal federated learning model to be trained are updated based on the fusion bias value, and the parameters of the upper-layer model of the longitudinal federated learning model to be trained are updated based on the fusion loss value, so as to obtain the target longitudinal federated learning model.

[0175] In some optional implementations, when updating the parameters of the longitudinal federated learning model to be trained to obtain the target longitudinal federated learning model, the training module 604 is specifically used for:

[0176] Update the parameters of the longitudinal federated learning model to be trained, and obtain the fusion loss value after parameter update;

[0177] If the fusion loss value after parameter update converges to the preset loss threshold, then the parameter-updated longitudinal federated learning model is taken as the target longitudinal federated learning model.

[0178] It should be noted that the apparatus provided in this application embodiment can implement all the method steps in the above-described training method embodiment of the vertical federated learning model and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0179] Based on the same inventive concept, this application also provides an electronic device that can implement the function of the aforementioned training method for the vertical federated learning model, referring to... Figure 7 As shown, the electronic device includes:

[0180] At least one processor 701 and a memory 702 connected to at least one processor 701. In this embodiment, the specific connection medium between the processor 701 and the memory 702 is not limited. Figure 7 The example shown is the connection between processor 701 and memory 702 via bus 700. Bus 700 is... Figure 7 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 700 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 7 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 701 can also be called a controller; there is no restriction on the name.

[0181] In this embodiment, memory 702 stores instructions executable by at least one processor 701. By executing the instructions stored in memory 702, at least one processor 701 can perform the training method for the longitudinal federated learning model discussed above. Processor 701 can implement... Figure 6 The functions of each module in the device shown.

[0182] The processor 701 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 702 and calling data stored in memory 702, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0183] In one possible design, processor 701 may include one or more processing units. Processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 701. In some embodiments, processor 701 and memory 702 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0184] The processor 701 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the model training method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0185] Memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 702 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 702 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 702 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0186] By designing and programming the processor 701, the code corresponding to the training method of the vertical federated learning model described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the code during runtime. Figure 1 The steps of the model training method in the illustrated embodiment are as follows. How to design and program the processor 701 is a technique well-known to those skilled in the art and will not be described further here.

[0187] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the training method for the longitudinal federated learning model described above.

[0188] In some possible implementations, various aspects of the training method for the longitudinal federated learning model provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the training method for the longitudinal federated learning model according to the various exemplary embodiments of this application described above.

[0189] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for a longitudinal federated learning model, characterized in that, include: Obtain a first data sample from the task initiator and a second data sample from the data provider, and based on the first data sample and the second data sample, obtain a shared data sample between the task initiator and the data provider; Based on the shared data samples, a first data feature corresponding to the task initiator and a second data feature corresponding to the data provider are obtained respectively; wherein, the first data feature and the second data feature respectively represent different feature information of the shared data samples; The first data feature and the second data feature are fused to obtain the fused data feature; Based on the first data feature, the second data feature, and the fused data feature, the longitudinal federated learning model to be trained is trained multiple times to obtain the target longitudinal federated learning model.

2. The method as described in claim 1, characterized in that, The process of training the vertical federated learning model to be trained multiple times based on the first data feature, the second data feature, and the fused data feature to obtain the target vertical federated learning model includes: Calculate the first loss value corresponding to the first data feature, the second loss value corresponding to the second data feature, and the fusion loss value corresponding to the fused data feature, respectively. Partial derivatives are performed on the first loss value, the second loss value, and the fusion loss value respectively to obtain the first initial deviation value of the task initiator, the second initial deviation value of the data provider, and the fusion deviation value; A weighted calculation is performed based on the fusion loss value and the first initial deviation value to obtain the first weighted deviation value of the task initiator, and a weighted calculation is performed based on the fusion loss value and the second initial deviation value to obtain the second weighted deviation value of the data provider; Based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value, the parameters of the longitudinal federated learning model to be trained are updated to obtain the target longitudinal federated learning model.

3. The method as described in claim 2, characterized in that, The step of performing a weighted calculation based on the fusion loss value and the first initial deviation value to obtain the first weighted deviation value of the task initiator, and performing a weighted calculation based on the fusion loss value and the second initial deviation value to obtain the second weighted deviation value of the data provider, includes: Based on the fusion loss value and the first loss value, calculate the first loss weight corresponding to the first data feature, and based on the fusion loss value and the second loss value, calculate the second loss weight corresponding to the second data feature; Based on the first initial deviation value and the first loss weight, the first weighted deviation value is obtained, and based on the second initial deviation value and the second loss weight, the second weighted deviation value is obtained.

4. The method as described in claim 2, characterized in that, The step of updating the parameters of the longitudinal federated learning model to be trained based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value to obtain the target longitudinal federated learning model includes: The parameters of the first underlying model in the longitudinal federated learning model to be trained are updated based on the first weighted bias value; wherein, the first underlying model is set for the task initiator; The parameters of the second bottom-level model in the longitudinal federated learning model to be trained are updated based on the second weighted bias value; wherein, the second bottom-level model is set for the data provider; The parameters of the middle layer model of the longitudinal federated learning model to be trained are updated based on the fusion bias value, and the parameters of the upper layer model of the longitudinal federated learning model to be trained are updated based on the fusion loss value, so as to obtain the target longitudinal federated learning model.

5. The method according to any one of claims 2-4, characterized in that, The step of updating the parameters of the longitudinal federated learning model to be trained to obtain the target longitudinal federated learning model includes: Update the parameters of the longitudinal federated learning model to be trained, and obtain the fusion loss value after parameter update; If the fusion loss value after parameter update converges to a preset loss threshold, then the parameter-updated longitudinal federated learning model is taken as the target longitudinal federated learning model.

6. A training device for a longitudinal federated learning model, characterized in that, include: The processing module is used to obtain a first data sample from the task initiator and a second data sample from the data provider, and based on the first data sample and the second data sample, obtain a shared data sample between the task initiator and the data provider. The feature conversion module is used to obtain a first data feature corresponding to the task initiator and a second data feature corresponding to the data provider based on the shared data samples; wherein the first data feature and the second data feature respectively represent different feature information of the shared data samples; The feature fusion module is used to fuse the first data feature and the second data feature to obtain fused data features; The training module is used to train the longitudinal federated learning model to be trained multiple times based on the first data feature, the second data feature, and the fused data feature to obtain the target longitudinal federated learning model.

7. The apparatus as claimed in claim 6, characterized in that, When the vertical federated learning model to be trained is trained multiple times based on the first data feature, the second data feature, and the fused data feature to obtain the target vertical federated learning model, the training module is specifically used for: Calculate the first loss value corresponding to the first data feature, the second loss value corresponding to the second data feature, and the fusion loss value corresponding to the fused data feature, respectively. Partial derivatives are performed on the first loss value, the second loss value, and the fusion loss value respectively to obtain the first initial deviation value of the task initiator, the second initial deviation value of the data provider, and the fusion deviation value; A weighted calculation is performed based on the fusion loss value and the first initial deviation value to obtain the first weighted deviation value of the task initiator, and a weighted calculation is performed based on the fusion loss value and the second initial deviation value to obtain the second weighted deviation value of the data provider; Based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value, the parameters of the longitudinal federated learning model to be trained are updated to obtain the target longitudinal federated learning model.

8. The apparatus as claimed in claim 7, characterized in that, When the first weighted deviation value of the task initiator is obtained by weighted calculation based on the fusion loss value and the first initial deviation value, and the second weighted deviation value of the data provider is obtained by weighted calculation based on the fusion loss value and the second initial deviation value, the training module is specifically used for: Based on the fusion loss value and the first loss value, calculate the first loss weight corresponding to the first data feature, and based on the fusion loss value and the second loss value, calculate the second loss weight corresponding to the second data feature; Based on the first initial deviation value and the first loss weight, the first weighted deviation value is obtained, and based on the second initial deviation value and the second loss weight, the second weighted deviation value is obtained.

9. The apparatus as claimed in claim 7, characterized in that, When updating the parameters of the longitudinal federated learning model to be trained based on the first weighted bias value, the second weighted bias value, the fusion loss value, and the fusion bias value to obtain the target longitudinal federated learning model, the training module is specifically used for: The parameters of the first underlying model in the longitudinal federated learning model to be trained are updated based on the first weighted bias value; wherein, the first underlying model is set for the task initiator; The parameters of the second bottom-level model in the longitudinal federated learning model to be trained are updated based on the second weighted bias value; wherein, the second bottom-level model is set for the data provider; The parameters of the middle layer model of the longitudinal federated learning model to be trained are updated based on the fusion bias value, and the parameters of the upper layer model of the longitudinal federated learning model to be trained are updated based on the fusion loss value, so as to obtain the target longitudinal federated learning model.

10. The apparatus according to any one of claims 7-9, characterized in that, When updating the parameters of the longitudinal federated learning model to be trained to obtain the target longitudinal federated learning model, the training module is specifically used for: Update the parameters of the longitudinal federated learning model to be trained, and obtain the fusion loss value after parameter update; If the fusion loss value after parameter update converges to a preset loss threshold, then the parameter-updated longitudinal federated learning model is taken as the target longitudinal federated learning model.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the steps of the method according to any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-5.

Images