Longitudinal federated learning model training method, device, storage medium and program product

Abstract

The application discloses a longitudinal federated learning model training method, device, storage medium and program product. The method comprises the following steps: obtaining first training sample data, performing feature extraction on the first training sample data through a first-party feature extractor to obtain first training sample features; receiving second training sample features sent by a second device; updating a first-party main task prediction model and the first-party feature extractor based on the first training sample features and the second training sample features by calculating a first main task model prediction loss, and updating a first-party correlation relationship prediction model based on the second training sample features by calculating a first correlation relationship model prediction loss; and determining a first gradient according to the first main task model prediction loss, the first correlation relationship model prediction loss and the second training sample features, and sending the first gradient to the second device. Thus, the technical problem that the effect of user behavior prediction through longitudinal federated learning modeling in the prior art is poor is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology in financial technology (Fintech), and in particular to a method, device, storage medium and program product for training a vertical federated learning model. Background Technology

[0002] With the continuous development of fintech, especially internet fintech, more and more technologies (such as distributed systems and artificial intelligence) are being applied in the financial field, but the financial industry is also placing higher demands on technology.

[0003] In e-commerce scenarios, vertical federated learning is often used to predict user behavior. In this scenario, individual user data and the relationship features between users have a significant impact on the accuracy of prediction. However, in vertical federated learning scenarios, individual user data and relationship data between users are distributed among multiple participants, and both individual data and relationship data are key privacy data that cannot be shared. Currently, in scenarios where the proportion of individual data among participants is large, with a small amount of relational data, modeling is typically performed using logistic regression, tree models, or fully connected neural networks based on the individual data. However, this approach does not utilize relational data for modeling, thus failing to effectively leverage the relational data distributed across various participants, resulting in poor performance of joint modeling. In scenarios where both individual and relational data are abundant, graph neural networks are used as the backbone network to effectively capture the relationships between users. Graph neural networks can leverage the structural relationships of nodes by aggregating the feature vectors of adjacent users based on relational data (edges). However, since the relational relationships are distributed across various participants and cannot be shared, each party can only train its graph neural network based on its own relational features and use its own node representations for federated modeling, without being able to utilize the relational data of other parties. Compared to the performance of data-centralized graph network modeling, federated modeling suffers a significant performance loss in this case.

[0004] In other words, current vertical federated modeling cannot effectively utilize the relational data distributed among various participants, resulting in significant performance loss in federated modeling and poor performance in predicting user behavior through vertical federated learning modeling. Summary of the Invention

[0005] The main purpose of this application is to provide a method, device, storage medium and program product for training a vertical federated learning model, which aims to solve the technical problem that the existing technology has poor performance in predicting user behavior through vertical federated learning modeling.

[0006] To achieve the above objectives, this application provides a vertical federated learning model training method. The method is applied to a first device, which is equipped with a first-party feature extractor to be trained, a first-party main task prediction model to be trained, and a first-party association prediction model to be trained. The vertical federated learning model training method includes the following steps: First training sample data is obtained, and features are extracted from the first training sample data using a first-party feature extractor to obtain first training sample features, wherein the first training sample data includes first association data; The second training sample features are received from the second device, wherein the second training sample features are obtained by the second device through a second third-party feature extractor by extracting features from the first matching sample data corresponding to the first training sample data, and the first matching sample data includes second association data. Based on the features of the first training sample and the features of the second training sample, the first main task prediction loss of the first main task prediction model is determined, and based on the features of the second training sample, the first association prediction loss of the first association prediction model is determined. A first gradient is determined based on the prediction loss of the first main task model, the prediction loss of the first association model, and the features of the second training samples. The first gradient is then sent to the second device so that the second device can update the second-party feature extractor based on the first gradient.

[0007] This application also provides a vertical federated learning prediction optimization method, which is applied to a third device. The third device is equipped with a trained first-party feature extractor, a trained first-party main task prediction model, and a trained first-party association prediction model. The vertical federated learning prediction optimization method includes the following steps: Obtain the sample data to be predicted, and extract features from the sample data to be predicted using a first-party feature extractor to obtain first-party intermediate sample features. The first-party feature extractor is trained using the longitudinal federated learning model training method described above. The second device receives second-party intermediate sample features sent by the second device, wherein the second-party intermediate sample features are obtained by the second device through feature extraction of the third matching sample data corresponding to the sample data to be predicted by the second-party feature extractor, and the second-party feature extractor is trained using the longitudinal federated learning model training method described above. By inputting the first-party intermediate sample features and the second-party intermediate sample features into the first-party main task prediction model, the target federated prediction result is obtained. The first-party main task prediction model is trained using the longitudinal federated learning model training method described above.

[0008] This application also provides a vertical federated learning model training device, which is applied to a first device. The first device is equipped with a first-party feature extractor to be trained, a first-party main task prediction model to be trained, and a first-party association prediction model to be trained, including: The first feature extraction module is used to acquire first training sample data, and to extract features from the first training sample data using a first-party feature extractor to obtain first training sample features, wherein the first training sample data includes first association data. The first receiving module is used to receive the second training sample features sent by the second device, wherein the second training sample features are obtained by the second device through a second feature extractor to extract features from the first matching sample data corresponding to the first training sample data, and the first matching sample data includes second association data. The loss determination module is used to determine the first main task model prediction loss of the first main task prediction model based on the first training sample features and the second training sample features, and to determine the first association model prediction loss of the first association prediction model based on the second training sample features. The sending module is used to determine a first gradient based on the prediction loss of the first main task model, the prediction loss of the first association model, and the features of the second training samples, and send the first gradient to the second device so that the second device can update the second-party feature extractor based on the first gradient.

[0009] This application also provides a vertical federated learning prediction optimization device, wherein the vertical federated learning model training device is applied to a third device, and the third device is equipped with a trained first-party feature extractor, a trained first-party main task prediction model, and a trained first-party association prediction model. The vertical federated learning prediction optimization device includes: The second feature extraction module is used to acquire the sample data to be predicted, and to extract features from the sample data to be predicted by the first-party feature extractor to obtain the first-party intermediate sample features. The first-party feature extractor is trained using the longitudinal federated learning model training method described above. The second receiving module is used to receive the second intermediate sample features sent by the second device. The second intermediate sample features are obtained by the second device through feature extraction of the third matching sample data corresponding to the sample data to be predicted by the second feature extractor. The second feature extractor is trained using the longitudinal federated learning model training method described above. The prediction module is used to obtain the target federated prediction result by inputting the first party intermediate sample features and the second party intermediate sample features into the first party main task prediction model, wherein the first party main task prediction model is trained using the longitudinal federated learning model training method described above.

[0010] This application also provides an electronic device, which is a physical device, comprising: a memory, a processor, and a program of the vertical federated learning model training method stored in the memory and executable on the processor. When the program of the vertical federated learning model training method is executed by the processor, it can implement the steps of the vertical federated learning model training method as described above or the vertical federated learning prediction optimization method as described above.

[0011] This application also provides a storage medium, which is a computer-readable storage medium, storing a program for implementing a longitudinal federated learning model training method. When the program for the longitudinal federated learning model training method is executed by a processor, it implements the steps of the longitudinal federated learning model training method or the longitudinal federated learning prediction optimization method as described above.

[0012] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the longitudinal federated learning model training method or the longitudinal federated learning prediction optimization method as described above.

[0013] This application provides a vertical federated learning model training method, device, storage medium, and program product. The vertical federated learning model training method is applied to a first device, which is equipped with a first-party feature extractor to be trained, a first-party main task prediction model to be trained, and a first-party association prediction model to be trained. The method obtains first training sample data and extracts features from the first training sample data using the first-party feature extractor to obtain first training sample features. The first training sample data includes first association data, thus achieving the acquisition of features from one's own samples. Then, the method receives second training sample features sent by a second device. The second training sample features are obtained by the second device through a second-party feature extractor extracting features from first matching sample data corresponding to the first training sample data. The first matching sample data includes second association data, thus achieving the acquisition of features from other parameters. By obtaining the sample features of the participating party, and then determining the first main task model prediction loss of the first party's main task prediction model based on the first training sample features and the second training sample features, and determining the first association model prediction loss of the first party's association prediction model based on the second training sample features, the purpose of training and updating the local first party feature extractor, the first party's main task prediction model, and the first party's association prediction model using the second training sample features of the participating party is achieved. Furthermore, by determining the first gradient based on the first main task model prediction loss, the first association model prediction loss, and the second training sample features, the first gradient is sent to the second device so that the second device can update the second party feature extractor based on the first gradient, thus achieving the purpose of training and updating the participating party's feature extractor based on its own first main task model prediction loss and the first association model prediction loss. Compared to approaches that do not incorporate association features or only use one's own association features in federated learning modeling, this application introduces an association prediction model on the part of the participant. By optimizing the feature extractor of the participants through the association prediction model, the extracted sample features of the participants incorporate similar association information between the participants. In other words, it can implicitly obtain some association information of the participants, improve the discriminativeness and accuracy of the second-party intermediate sample features extracted by the feature extractor of the participants. Therefore, it overcomes the technical defect of vertical federated modeling, which is difficult to effectively utilize the association data distributed among the participants, resulting in poor vertical federated prediction performance, and improves the performance of vertical federated prediction. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating an embodiment of the training method for the vertical federated learning model in this application. Figure 2 This is a schematic diagram of one possible implementation of the overall framework of the vertical federated learning model in this application. Figure 3 This is a schematic diagram of a data structure used for vertical federated modeling in a recommended scenario of this application; Figure 4 This is a flowchart illustrating another embodiment of the training method for the vertical federated learning model in this application. Figure 5 A schematic diagram of an implementation method for the unsupervised learning framework of the longitudinal federated learning model in this application; Figure 6 This is a flowchart illustrating an embodiment of the longitudinal federated learning prediction optimization method in this application. Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the vertical federated learning model training method in this application embodiment.

[0017] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 This application provides a method for training a vertical federated learning model. In the first embodiment of this method, refer to... Figure 1 The vertical federated learning model training method is applied to a first device, on which a first-party feature extractor to be trained, a first-party main task prediction model to be trained, and a first-party association prediction model to be trained are deployed. The vertical federated learning model training method includes the following steps: Step S10: Obtain first training sample data, and extract features from the first training sample data using a first-party feature extractor to obtain first training sample features, wherein the first training sample data includes first association data. In this embodiment, it should be noted that vertical federated learning involves extracting users and data from participants with similar user characteristics but different user data characteristics when there is little overlap in participant data features but significant overlap in user data. This is then used for joint machine learning training. In e-commerce scenarios such as shopping recommendations and advertising recommendations, multiple participants with similar users but different user data can be trained together to expand the user data feature space of the model training and improve the accuracy of model predictions. User data includes individual data and relational data. Relational data refers to the data related to the relationships between users. Both individual data and relational data have a significant impact on prediction accuracy. Users with similar behaviors (shopping, browsing, etc.) are more likely to be interested in the same product / video / advertisement. Therefore, effectively utilizing the relationships between users can help the model capture the correlation between users, learn better user representations, and thus improve the joint modeling performance of the model. However, both individual user data and relational data are distributed among multiple participants and are considered critical privacy data that cannot be shared.

[0020] In scenarios where the proportion of individual data among various participants is relatively large, while there is a small amount of relational data, the relational data is usually modeled using common networks (such as logistic regression models, tree models, fully connected neural networks, etc.) based only on the individual data, rather than graph neural networks. Therefore, the relational data is not used for federated modeling.

[0021] In scenarios where both parties possess a large amount of individual and relational data, existing technologies typically employ graph neural networks (GNNs) as the backbone network for modeling to effectively capture the characteristics of user relationships. GNNs can leverage the structural relationships between nodes by aggregating feature vectors from adjacent users based on relational data (edges). However, relational data is distributed across various participants, and each party can only train its GNN based on its own relational data, using its node representations for supervised modeling, without utilizing the relational data of other parties. Compared to centralized graph network modeling, federated modeling suffers a significant performance loss in this scenario. To address this issue, current methods primarily transform relational data or rely on cryptography and privacy-preserving computation techniques: 1) Transforming relational information and sharing it, for example, sharing only the number of other nodes connected to a node, without directly sharing the original structural information. However, this still carries the risk of relational information leakage and increases communication overhead; 2) Sharing individual data from each party using cryptographic methods, and then each party models based on its full individual data and local relational data. This method carries the risk of individual data leakage, and the cryptographic approach not only increases computational overhead but also incurs substantial communication overhead. Compared to the two methods mentioned above, this application does not increase communication overhead, and the additional computational overhead is negligible compared to cryptographic computation. At the same time, it can effectively improve federated modeling performance and increase the accuracy of model prediction.

[0022] In this embodiment, it should be noted that the vertical federated learning model training method is applied to a vertical federated learning scenario. Both the first device and the second device are participants in the vertical federated learning scenario. The first device is the predictor, which can be a server, computer, laptop, smartphone, or other terminal device used to perform the prediction task. The second device is the auxiliary data provider, which can be a server, computer, laptop, smartphone, or other terminal device used to provide more training sample features to the first device, thereby improving the prediction accuracy of the first device. The predictor has a need to predict user behavior, therefore it deploys a main task prediction model to predict user behavior. The auxiliary data provider may also have a need to predict user behavior, in which case the main task prediction model can be deployed on the auxiliary data provider; alternatively, the auxiliary data provider may not have a need to predict user behavior, in which case it does not need to deploy a main task prediction model.

[0023] The first device also deploys a first-party feature extractor and a first-party association prediction model. The first-party feature extractor is used to extract its own sample features, and the first-party association prediction model is used to predict the association relationship of its own party based on the training sample features of other participants. The first-party association prediction model can predict association relationships based on node features, or it can predict association relationships by constructing subgraphs, such as based on graph neural networks, or it can predict association relationships by calculating the similarity of node features. The specific choice can be made according to the actual situation, and this embodiment does not impose any restrictions. Since the input of the first-party association prediction model is the training sample features of other participants, which implicitly carry the association relationship information of other parties, for example, if user A recommends user B to become a member of a certain store, then user A and user B may both have made purchases in the same store, and user A and user B may have recently purchased similar types of products, watched similar types of videos, etc. However, not all data has a significant correlation with the relationship information. Therefore, using the training sample features that implicitly carry the relationship information of other parties to predict one's own relationship can further filter the training sample features of other participants based on the parts of the relationship information of other parties that are similar to one's own relationship. In this way, training sample features of other participants carrying more effective relationship information can be obtained.

[0024] In one feasible approach, refer to Figure 2 , Figure 2 This is a structural diagram illustrating one possible implementation of the overall framework of a vertical federated learning model. Figure 2 In this scenario, participant A is the predictor, i.e., the first device, and participant B is the auxiliary data provider, i.e., the second device. Participant B deploys a second-party feature extractor M. B , to transfer individual data X B and related data e B Input to the second-party feature extractor M B Output the second-party feature h b Participant B will use the second-party feature h b Send to participant A. Participant A has deployed a first-party feature extractor M. A First-party primary task prediction model H A and the first-party association prediction model P A , to transfer individual data X A and related data e A Input to the first-party feature extractor M A Output the first-party feature h a , the first-party feature h a And the second-party feature h received from participant B bAfter aggregation, the data is input into the first-party primary task prediction model H. A In the first party's main task prediction model H A It can be a one- or multi-layer fully connected neural network, based on the prediction results of the first main task model and the label Y. A Calculate the prediction loss L of the first main task model. supervised ; the second-party feature h b Input into the first-party association prediction model P A In the first-party relationship prediction model P A It can be a one-layer or multi-layer fully connected neural network, based on the prediction results of the obtained first association model and the association data e. A Calculate the prediction loss L of the first association model. structure Participant A, based on the total loss function L... total =L supervised +L structure Calculate the gradient and update the model M. A P A and H A And calculate for h B The first gradient g B , the first gradient g B Transmitted to participant B, so that participant B can base its decision on the first gradient g. B Update participant B's model M B .

[0025] As an example, when a vertical federated learning model is applied to the e-commerce field, the participants in the vertical federated learning can be various types of e-commerce platforms, such as shopping platforms, video platforms, gaming platforms, mobile app stores, and financial platforms. In this scenario, the main task prediction model can be a user behavior prediction model, such as a user consumption behavior prediction model. The training samples used to train the vertical federated learning model refer to users, and the training sample data refers to user data. User data includes individual data and relational data. Individual data includes, but is not limited to, basic user information (e.g., name, gender, age, education, occupation, address, income data, etc.), business data (e.g., loan data, consumption data, etc.), and behavioral data (e.g., overdue repayment behavior, types of videos recently watched, frequency of payment operations over a period of time, etc.). Relational data refers to data on relationships between users, including but not limited to friendly relationship data (e.g., likes, friend relationships, etc.), recommendation relationship data (e.g., sharing links, recommending to become a member, etc.), and behavioral similarity relationship data (e.g., being members of the same store, having the same preference tags, etc.).

[0026] As an example, step S10 includes: obtaining a batch of first training sample data from the first-party raw data; extracting features from the batch of first training sample data using a first-party feature extractor to be trained deployed on the first device to obtain first training sample features. The first training sample data can be user data, and can include first-party volume data and first-party relational data, etc. The first-party feature extractor can be selected according to the data format. For example, for image data, the first-party feature extractor can be a convolutional neural network; for table data, the first-party feature extractor can be a fully connected neural network; for data with a large number of structural relationships, the first-party feature extractor can be a graph neural network. This embodiment does not limit the model structure of the first-party feature extractor, so it is applicable to various vertical federated learning scenarios.

[0027] Step S20: Receive second training sample features sent by the second device, wherein the second training sample features are obtained by the second device through a second third-party feature extractor by extracting features from the first matching sample data corresponding to the first training sample data, and the first matching sample data includes second association data. In this embodiment, it should be noted that the second device, as an auxiliary data provider, is equipped with at least a second-party feature extractor. This second-party feature extractor is used to extract second training sample features from first matching sample data stored in the second device that is aligned with the first training sample data. In one feasible implementation, if the second device also has a need to predict user behavior, it can also act as a predictor. Therefore, it can also be equipped with a main task prediction model and a relationship prediction model. In this case, the model training and prediction processes on the second device are similar to those on the first device, which will not be elaborated upon further.

[0028] As an example, step S20 includes: after the first device and the second device achieve data alignment through encryption, the second device extracts features from the first matching sample data aligned with the first training sample data using a second-party feature extractor to obtain second training sample features, and sends the second training sample features to the first device. The first device receives the second training sample features sent by the second device. The first matching sample data may be user data, and the first matching sample data may include second individual data and second relationship data from the second party, etc.

[0029] Step S30: Determine the first main task model prediction loss of the first main task prediction model based on the first training sample features and the second training sample features, and determine the first association model prediction loss of the first association prediction model based on the second training sample features. As an example, step S30 includes two loss calculation processes: the calculation process of the loss predicted by the first main task model and the calculation process of the loss predicted by the first association model. This embodiment does not restrict the order of the two loss calculation processes. In one implementable manner, the loss predicted by the first main task model can be cross-entropy loss, and the loss predicted by the first association model can be binary classification loss.

[0030] In one feasible approach, the calculation of the prediction loss of the first primary task model includes: concatenating the features of the first training samples and the features of the second training samples into a first training sample feature vector or a first training sample feature matrix; inputting the first training sample feature vector or the first training sample feature matrix into the first primary task prediction model to obtain the prediction result of the first primary task model; and then training the model through supervised learning or unsupervised learning to obtain the prediction loss of the first primary task prediction model.

[0031] In one feasible approach, the calculation of the prediction loss of the first association relationship model includes: concatenating two or more second training sample features into an initial training sample feature vector or an initial training sample feature matrix; then concatenating each of the initial training sample feature vectors or initial training sample feature matrices into a second training sample feature vector or a second training sample feature matrix; inputting the second training sample feature vector or second training sample feature matrix into the first-party association relationship prediction model to obtain the prediction result of the first association relationship model; and then training the model through supervised learning or unsupervised learning to obtain the first association relationship prediction loss of the first-party association relationship prediction model.

[0032] In one feasible approach, after the step of determining the first primary task model prediction loss of the first primary task prediction model based on the features of the first training samples and the features of the second training samples, the process may further include a first-party model update process. The first-party model update process may update the model based on each loss function separately, or it may determine the total first-party model loss based on each loss function and update the model based on the total first-party model loss.

[0033] As an example, when determining whether a model has converged based on the loss, the judgment can be made separately based on the prediction loss of the first main task model and the prediction loss of the first association model, and each model can be trained separately. Alternatively, the prediction loss of the first main task model and the prediction loss of the first association model can be combined into the total loss of the first-party model, and the first-party feature extractor, the first-party main task prediction model, and the first-party association prediction model can be trained as a whole model. The convergence of the whole model can be determined based on the total loss of the first-party model. This embodiment does not impose any restrictions on this approach.

[0034] In one feasible approach, the method of training the model based on the prediction loss of the first primary task model includes: determining whether the prediction loss of the first primary task model has converged; if the prediction loss of the first primary task model has converged, then determining that the training of the first primary task prediction model is complete; if the prediction loss of the first primary task model has not converged, then updating the first primary task prediction model and the first feature extractor in one round based on the model gradient calculated by the prediction loss of the first primary task model, and returning to the step of obtaining the first training sample data to obtain another batch of the first training sample data, until the calculated prediction loss of the first primary task model converges.

[0035] In one feasible approach, the method of training the model based on the prediction loss of the first association model includes: determining whether the prediction loss of the first association model has converged; if the prediction loss of the first association model has converged, then determining that the training of the first association model has been completed; if the prediction loss of the first association model has not converged, then updating the first association model based on the model gradient calculated by the prediction loss of the first association model, and returning to the step of obtaining the first training sample data to obtain another batch of the first training sample data, until the calculated prediction loss of the first association model converges.

[0036] In one feasible approach, model training based on the first-party model total loss includes: determining whether the first-party model total loss has converged; if the first-party model total loss has converged, then determining that the training of the first-party feature extractor, the first-party main task prediction model, and the first-party association prediction model is complete; if the first-party model total loss has not converged, then updating the first-party feature extractor, the first-party main task prediction model, and the first-party association prediction model based on the model gradient calculated by the first-party model total loss, and returning to the step of obtaining the first training sample data to obtain another batch of the first training sample data, until the calculated first-party model total loss converges.

[0037] Furthermore, after the step of determining the first primary task model prediction loss of the first primary task prediction model based on the first training sample features and the second training sample features, the method further includes: Step A10: Obtain the main task label corresponding to the first training sample data; In this embodiment, it should be noted that when the sample labels of the training sample data are available, supervised learning can be used for model training.

[0038] As an example, step A10 includes: obtaining the main task label corresponding to the first training sample data, wherein the first-party main task prediction model can be a classification model, and the main task label can be a pre-labeled user label. In one implementable manner, when the first-party main task prediction model is a user consumption behavior prediction model, the main task label can be a classification label that identifies the probability of a user making a purchase, or a classification label that identifies the user's consumption level, etc.

[0039] Step A20: By inputting the features of the first training sample and the features of the second training sample into the first main task prediction model, the prediction result of the first main task model is obtained; As an example, step A20 includes: concatenating the first training sample features and the second training sample features into a first training sample feature vector or a first training sample feature matrix, and inputting the first training sample feature vector or the first training sample feature matrix into a first main task prediction model to obtain the prediction result of the first main task model.

[0040] Step A30: Based on the prediction results of the first main task model and the main task label, update the first main task prediction model and the first feature extractor by calculating the prediction loss of the first main task prediction model.

[0041] As an example, step A30 includes: calculating the first main task prediction loss of the first main task prediction model based on the difference between the prediction result of the first main task model and the main task label; determining whether the first main task prediction loss has converged; if the first main task prediction loss has converged, determining that the training of the first main task prediction model has been completed; if the first main task prediction loss has not converged, updating the first main task prediction model and the first feature extractor based on the model gradient calculated by the first main task prediction loss, and returning to the step of obtaining the first training sample data to obtain another batch of the first training sample data until the calculated first main task prediction loss has converged.

[0042] Furthermore, after the step of determining the first association relationship prediction model prediction loss of the first association relationship prediction model based on the features of the second training samples, the method further includes: Step B10: Obtain the first association label corresponding to the first training sample data; In this embodiment, it should be noted that when the sample labels of the training sample data are available, supervised learning can be used for model training.

[0043] As an example, step B10 includes: obtaining a first association label corresponding to the first training sample data, wherein the first association label can be obtained by manually annotating the first training sample data, or the association data can be extracted from the first training sample data as the first association label.

[0044] In one feasible approach, refer to Figure 3 , Figure 3 This is a schematic diagram of a data structure used for vertical federated modeling in a recommendation scenario. Figure 3 In the table, the left side shows user data from the video app (Application), and the right side shows user data from the shopping app (Identity). The document (identity identifier) ​​is the user ID. Both parties have five common users: U1, U2, U3, U4, and U5. X1, X2, X3, and X4 are individual data, and E1, E2, E4, and E5 are relationship data. Y is the probability label for purchasing product XXX. Based on this user data, the first relationship label can be obtained. Among the common users, for the relationship feature E4, Y(U2,U3)=1, Y(U2,U5)=1, Y(U3,U5)=1, and other combinations are 0, that is, there is a relationship between U2 and U3, U2 and U5, and U3 and U5. For the relationship feature E5, Y(U1,U3)=1, Y(U2,U3)=1, Y(U4,U5)=1, and other combinations are 0, that is, there is a relationship between U1 and U3, U2 and U3, and U4 and U5.

[0045] Step B20: By inputting the features of the second training sample into the first association relationship prediction model, the prediction result of the first association relationship model is obtained; As an example, step B20 includes: concatenating two or more second training sample features into an initial training sample feature vector or an initial training sample feature matrix, then concatenating each of the initial training sample feature vectors or initial training sample feature matrices into a second training sample feature vector or a second training sample feature matrix, and inputting the second training sample feature vector or the second training sample feature matrix into a first-party association prediction model to obtain the prediction result of the first association model.

[0046] Further, the step of obtaining the prediction result of the first association model by inputting the features of the second training sample into the first association prediction model includes: Step B21: By querying the first association label, select at least one set of first target training sample features that have an association relationship and at least one set of second target training sample features that do not have an association relationship from the second training sample features; As an example, step B21 includes: selecting at least one set of training sample features to be queried from the second training sample features; querying from the first association label whether there is an association between the various training sample features to be queried in each set; determining at least one set of training sample features to be queried with an association as the first target training sample features; and determining at least one set of training sample features to be queried without an association as the second target training sample features. The number of training sample features to be queried in each set can be determined according to the actual situation, and this embodiment does not impose any limitation on this; for example, it can be two.

[0047] Step B22: Concatenate the features of each group of first target training samples into positive sample data, and concatenate the features of each group of second target training samples into negative sample data. As an example, step B22 includes: concatenating the features of the first target training samples belonging to the same group into a positive sample data, and concatenating the features of the second target training samples belonging to the same group into a negative sample data, thereby obtaining at least one positive sample data and at least one negative sample data.

[0048] Step B23: Select positive and negative sample data that meet the preset positive-negative sample ratio, input them into the first-party association prediction model, and obtain the prediction results of the first association model.

[0049] As an example, step B23 includes: selecting a certain number of positive sample data and a certain number of negative sample data according to a preset positive-negative sample ratio and inputting them into the first-party association relationship prediction model to obtain the prediction result of the first association relationship model. The positive-negative sample ratio can be determined according to actual needs, and this embodiment does not impose any restrictions on it.

[0050] In one feasible approach, the ratio of positive to negative samples can be 1:1. Considering that the correlation data is usually sparse and the negative sample data is much more numerous than the positive sample data, the ratio of positive to negative samples can be manually balanced to maintain it at around 1:1 when constructing the samples.

[0051] Step B30: Based on the prediction results of the first association model and the first association label, update the first association prediction model by calculating the prediction loss of the first association prediction model.

[0052] As an example, step B30 includes: calculating the first association model prediction loss of the first association model based on the difference between the prediction result of the first association model and the first association label; determining whether the first association model prediction loss has converged; if the first association model prediction loss has converged, determining that the training of the first association model has been completed; if the first association model prediction loss has not converged, updating the first association model based on the model gradient calculated by the first association model prediction loss, and returning to the step of obtaining the first training sample data to obtain another batch of the first training sample data, until the calculated first association model prediction loss converges.

[0053] Furthermore, the first association label includes at least one dimension-related association label, and the first-party association prediction model includes at least one dimension-related first-party association prediction sub-model to be trained. The step of inputting the features of the second training sample into the first association prediction model to obtain the prediction result of the first association model includes: Step C10: By inputting the features of the second training sample into each first-party association prediction sub-model, the prediction results of the first association model corresponding to each dimension are obtained. In this embodiment, it should be noted that the association can be further subdivided into different dimensions, including friendly relationships, recommendation relationships, and behavioral similarity relationships. Different association prediction sub-models can be established for different dimensions, and predictions can be made based on different association data to improve the accuracy of association prediction. When multiple association prediction sub-models are established, each dimension of the association should be labeled separately to obtain the corresponding dimension association labels.

[0054] As an example, step C10 includes: concatenating two or more second training sample features into an initial training sample feature vector or an initial training sample feature matrix, then concatenating each of the initial training sample feature vectors or initial training sample feature matrices into a second training sample feature vector or a second training sample feature matrix, and inputting the second training sample feature vectors or second training sample feature matrices into each first-party correlation prediction sub-model to obtain the first correlation model prediction results corresponding to each dimension.

[0055] The step of updating the first-party association prediction model by calculating the prediction loss of the first-party association prediction model based on the prediction result of the first association model and the first association label includes: Step C20: Based on the prediction results of the first association model corresponding to each dimension and the dimension association label, calculate the association sub-model prediction loss of each first-party association prediction sub-model. As an example, step C20 includes: calculating the model prediction loss for each dimension; and for each target dimension, calculating the association sub-model prediction loss of the first-party association prediction sub-model corresponding to the target dimension based on the difference between the prediction result of the first association model corresponding to the target dimension and the dimension association label corresponding to the target dimension.

[0056] Step C30: The average value of the prediction loss of each of the association sub-models is determined as the first association model prediction loss, and the first association prediction model is updated based on the first association model prediction loss.

[0057] As an example, step C30 includes: determining the average value of the prediction loss of each of the association sub-models as the first association model prediction loss; determining whether the first association model prediction loss has converged; if the first association model prediction loss has converged, determining that the first association prediction model training is complete; if the first association model prediction loss has not converged, updating the first association prediction model based on the model gradient calculated by the first association model prediction loss, and returning to the step of obtaining the first training sample data to obtain another batch of the first training sample data, until the calculated first association model prediction loss converges.

[0058] Step S40: Determine a first gradient based on the prediction loss of the first main task model, the prediction loss of the first association model, and the features of the second training samples, and send the first gradient to the second device so that the second device can update the second-party feature extractor based on the first gradient.

[0059] As an example, step S40 includes: merging the prediction loss of the first main task model and the prediction loss of the first association model by means of addition, multiplication, weighted summation, etc., to obtain the total loss of the first-party model; determining whether the total loss of the first-party model has converged; if the total loss of the first-party model has converged, determining that the training of the second-party feature extractor is complete; if the total loss of the first-party model has not converged, sending the first gradient of the features of the second training samples calculated based on the total loss of the first-party model to the second device, so that the second device can update the second-party feature extractor based on the first gradient. Then, returning to the step of obtaining the first training sample data, another batch of the first training sample data is obtained until the calculated total loss of the first-party model converges.

[0060] In this embodiment, the vertical federated learning model training method is applied to a first device. The first device is equipped with a first-party feature extractor to be trained, a first-party main task prediction model to be trained, and a first-party association prediction model to be trained. By acquiring first training sample data, the first-party feature extractor extracts features from the first training sample data to obtain first training sample features. The first training sample data includes first association data, thus obtaining features of one's own samples. Then, by receiving second training sample features sent by a second device, the second training sample features are obtained by the second device extracting features from first matching sample data corresponding to the first training sample data using a second-party feature extractor. The first matching sample data includes second association data, thus obtaining sample features of other participants. By determining the first main task prediction loss of the first party's main task prediction model based on the first training sample features and the second training sample features, and determining the first association prediction loss of the first party's association prediction model based on the second training sample features, the purpose of training and updating the local first-party feature extractor, the first party's main task prediction model, and the first party's association prediction model using the second training sample features of the participants is achieved. Furthermore, by determining the first gradient based on the first main task model prediction loss, the first association prediction loss, and the second training sample features, and sending the first gradient to the second device, the second device can update the second-party feature extractor based on the first gradient, thus achieving the purpose of training and updating the feature extractor of the participants based on the first main task model prediction loss and the first association prediction loss of the participants. Compared to approaches that do not incorporate association features or only use one's own association features in federated learning modeling, this application introduces an association prediction model on the part of the participant. By optimizing the feature extractor of the participants through the association prediction model, the extracted sample features of the participants incorporate similar association information between the participants. In other words, it can implicitly obtain some association information of the participants, improve the discriminativeness and accuracy of the second-party intermediate sample features extracted by the feature extractor of the participants. Therefore, it overcomes the technical defect of vertical federated modeling, which is difficult to effectively utilize the association data distributed among the participants, resulting in poor vertical federated prediction performance, and improves the performance of vertical federated prediction.

[0061] Example 2 Furthermore, referring to Figure 4Based on the above embodiments of this application, in the second embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. On this basis, before the step of obtaining the first training sample data and extracting features from the first training sample data using a first-party feature extractor to obtain the features of the first training sample, the method further includes: Step D10: Obtain the third training sample data and the second association label corresponding to the third training sample data; In this embodiment, it should be noted that for vertical federated scenarios, due to the difficulty in obtaining labeled data, there is usually a lot of alignment data for the two or more parties involved in joint modeling, but little labeled data. When there is a lot of relationship data between users, relationship prediction can be combined with unsupervised learning to make full use of the alignment data of both parties. Before training the main task prediction model, the structural prediction model of one party, the feature extractor of one party, and the feature extractors of other participating parties are updated.

[0062] For example, refer to Figure 5 , Figure 5 This is a schematic diagram illustrating one possible implementation of an unsupervised learning framework for a longitudinal federated learning model. Figure 5 In this scenario, both participant A and participant B are predictors (i.e., the first device) and also data providers (i.e., the second device). Participant A deploys the first-party feature extractor M. A First-party feature transformation model H A and the first-party association prediction model P A Participant B also deployed a second-party feature extractor M. B Second-party feature transformation model H B And the second-party association prediction model P B The training processes for participants A and B are similar. Taking participant A as an example, participant A will use individual data X... A and related data e A Input to the first-party feature extractor M A Output the first-party feature h a , the first-party feature h a And the second-party feature h received from participant B b Input into the first-party feature transformation model H A In the middle, the first-party feature h a Second-party feature h b Transform to the same feature space and calculate the unsupervised loss L. unsupervised The first-party feature transformation model H A It can be a one-layer or multi-layer fully connected neural network; the second-party feature h bInput into the first-party association prediction model P A In the first-party relationship prediction model P A It can be a one-layer or multi-layer fully connected neural network, based on the prediction results of the obtained first association model and the association data e. A Calculate the prediction loss L of the second correlation model. structure Participant A, based on the total loss function L... total =L unsupervised +L structure Calculate the gradient and update the model M. A P A and H A And calculate for h B The first gradient g B , the first gradient g B Transmitted to participant B, so that participant B can base its decision on the first gradient g. B Update participant B's model M B .

[0063] In this scenario, both the predictor and the auxiliary data provider have correlation data, and therefore both have deployed correlation prediction models. In order to integrate the feature and structural information of both parties, the predictor and the auxiliary data provider will calculate the correlation-related loss function and update each other's model parameters based on the loss function.

[0064] As an example, step D10 includes: obtaining a batch of third training sample data from the first-party original data, and determining the second association label corresponding to the third training sample data, wherein the second association label can be obtained by manually annotating the third training sample data, or the association data can be extracted from the third training sample data as the first association label.

[0065] Step D20: The third training sample data is subjected to feature extraction by the first-party feature extractor to obtain the third training sample features, and the third training sample features are sent to the second device so that the second device can perform model training based on the third training sample features. As an example, step D20 includes: extracting features from a batch of third training sample data using a first-party feature extractor deployed on the first device to obtain third training sample features, and sending the third training sample features to the second device so that the second device can perform model training based on the third training sample features. The method by which the second device performs model training based on the third training sample features can correspond to the method by which the first device performs model training based on the fourth training sample features sent by the second device. This embodiment will not elaborate further here.

[0066] Step D30: Receive the fourth training sample features sent by the second device, wherein the fourth training sample features are obtained by the second device through a second-party feature extractor by extracting features from the second matching sample data corresponding to the third training sample data; As an example, step D30 includes: after the first device and the second device achieve data alignment through encryption, the second device performs feature extraction on the second matching sample data aligned with the third training sample data through a second-party feature extractor to obtain fourth training sample features, and sends the fourth training sample features to the first device, and the first device receives the fourth training sample features sent by the second device.

[0067] Step D40: Based on the features of the third training sample and the features of the fourth training sample, perform unsupervised learning, update the first-party feature extractor by calculating the unsupervised loss, and update the first-party association prediction model by calculating the second association model prediction loss of the first-party association prediction model based on the features of the fourth training sample and the second association label. As an example, step D40 includes: performing unsupervised learning using an unsupervised model based on the features of the third and fourth training samples to obtain an unsupervised loss; concatenating two or more features of the fourth training samples into a feature vector or feature matrix; inputting this feature vector into a first-party association prediction model to obtain a second-party association prediction result; and calculating the second-party association prediction loss based on the second-party association prediction result and the second-party association label. The first-party feature extractor can be updated based on the unsupervised loss, and the first-party association prediction model can be updated based on the second-party association prediction loss; alternatively, the unsupervised loss and the second-party association prediction loss can be combined using addition, multiplication, weighted summation, etc., to obtain the total loss of the first-party unsupervised model, and the first-party feature extractor and the first-party association prediction model can be updated based on the total loss of the first-party unsupervised model.

[0068] Furthermore, a projection model is also deployed on the first device, and the step of performing unsupervised learning based on the third training sample features and the fourth training sample features includes: Step D41: By inputting the features of the fourth training sample into the projection model, the transformed features are obtained; In this embodiment, it should be noted that the unsupervised learning can be contrastive learning. The purpose of contrastive learning is to make the features of the same user as close as possible in the feature space, and the features of different users as far apart as possible in the feature space.

[0069] As an example, step D41 includes: inputting the fourth training sample features into a projection model to transform the fourth training sample features into the same feature space as the third training sample features, thereby obtaining transformed features, wherein the projection model can be a multilayer fully connected network.

[0070] Step D42: Calculate the negative cosine similarity loss based on the transformed features and the third training sample features.

[0071] As an example, step D41 includes: calculating the intermediate negative cosine similarity loss based on the transformed features, the third training sample features, and a preset similarity loss algorithm. For example, for two aligned sample features f... i and z j The preset similarity loss algorithm is as follows: .

[0072] Then, by iterating through the features of the training samples in the current training batch from both the predictor and the auxiliary data provider, and calculating the intermediate negative cosine similarity loss for each aligned sample pair in the current training batch, the negative cosine similarity loss is obtained. .

[0073] Step D50: Calculate the second gradient based on the unsupervised loss, the prediction loss of the second association model, and the features of the fourth training sample, and send the second gradient to the second device so that the second device can update the second-party feature extractor based on the second gradient.

[0074] As an example, step D50 includes: merging the unsupervised loss and the prediction loss of the second association model through addition, multiplication, weighted summation, etc., to obtain the total loss of the first-party unsupervised model; determining whether the total loss of the first-party unsupervised model has converged; if the total loss of the first-party unsupervised model has converged, then determining that the training of the second-party feature extractor is complete; if the total loss of the first-party unsupervised model has not converged, then sending the second gradient of the features of the fourth training sample calculated based on the total loss of the first-party unsupervised model to the second device, so that the second device can update the second-party feature extractor based on the second gradient. The process then returns to the step of obtaining the third training sample data and the second association label corresponding to the third training sample data, obtaining another batch of third training sample data, until the calculated total loss of the first-party unsupervised model converges.

[0075] In this embodiment, for situations where a small amount of labeled sample data and a large amount of unlabeled sample data are available, structural relationship prediction is combined with unsupervised learning in advance. By fully utilizing the aligned data of both parties, the association prediction model of one party and the feature extractors of each party are trained effectively. Subsequently, supervised learning is used to further adjust the model in combination with labeled sample data. This approach can fully utilize the sample data of each party to train the model effectively, thereby improving model performance, increasing the accuracy of vertical federated learning modeling for user behavior prediction, and enhancing the model's prediction effect.

[0076] Example 3 This application provides a vertical federated learning model training method. The vertical federated learning prediction optimization method is applied to a third device, which is equipped with a trained first-party feature extractor, a trained first-party main task prediction model, and a trained first-party association prediction model. The vertical federated learning prediction optimization method includes the following steps: Step E10: Obtain the sample data to be predicted, and extract features from the sample data to be predicted using a first-party feature extractor to obtain first-party intermediate sample features. The first-party feature extractor is trained using the longitudinal federated learning model training method described in Example 1 and / or Example 2. It should be noted that the vertical federated learning prediction optimization method is applied to a third device, which may be the same as or different from the first device. That is, model training and model application can be performed on the same device or on different devices. This embodiment does not impose any restrictions on this. The third device is equipped with a trained first-party feature extractor, a trained first-party main task prediction model, and a trained first-party association prediction model. The training process of the first-party feature extractor, the first-party main task prediction model, and the first-party association prediction model can be referred to Embodiment 1 and / or Embodiment 2, which will not be elaborated on here.

[0077] As an example, step E10 includes: acquiring sample data to be predicted, concatenating the sample data to be predicted into a feature vector or feature matrix to be extracted, inputting the feature vector or feature matrix to be extracted into a trained first-party feature extractor for feature extraction, and obtaining first-party intermediate sample features, wherein the first-party feature extractor is trained using the longitudinal federated learning model training method as described in Embodiment 1 and / or Embodiment 2.

[0078] Step E20: Receive second-party intermediate sample features sent by the second device, wherein the second-party intermediate sample features are obtained by the second device through feature extraction of the third matching sample data corresponding to the sample data to be predicted by the second-party feature extractor, and the second-party feature extractor is trained using the longitudinal federated learning model training method as described in Embodiment 1 and / or Embodiment 2. As an example, step E20 includes: after the first device and the second device achieve data alignment through encryption, the second device extracts features from the third matching sample data aligned with the sample data to be predicted using a trained second-party feature extractor to obtain second-party intermediate sample features, and sends the second-party intermediate sample features to the first device. The first device receives the second-party intermediate sample features sent by the second device, wherein the second-party feature extractor is trained using the vertical federated learning model training method described in Embodiment 1 and / or Embodiment 2.

[0079] Step E30: By inputting the first-party intermediate sample features and the second-party intermediate sample features into the first-party main task prediction model, the target federated prediction result is obtained. The first-party main task prediction model is trained using the longitudinal federated learning model training method described in Example 1 and / or Example 2.

[0080] As an example, step E30 includes: concatenating the first intermediate sample features and the second intermediate sample features into a feature vector or feature matrix to be predicted, inputting the feature vector or feature matrix to be predicted into the first main task prediction model, and obtaining the target federated prediction result, wherein the first main task prediction model is trained using the vertical federated learning model training method as described in Embodiment 1 and / or Embodiment 2.

[0081] This application provides a vertical federated learning prediction optimization method. By acquiring the sample data to be predicted, a first-party feature extractor extracts features from the sample data to obtain first-party intermediate sample features, thus extracting features from the user's own sample data. Then, by receiving second-party intermediate sample features sent by a second device, where the second-party intermediate sample features are obtained by the second device extracting features from the third matching sample data corresponding to the sample data to be predicted using the second-party feature extractor, features from other participants (the second device) are acquired. Since the second-party feature extractor is obtained through iterative optimization based on the first main task prediction model prediction loss of the first-party main task prediction model and the first association prediction model prediction loss of the first-party association prediction model, the second-party intermediate sample features extracted by the second-party feature extractor achieve the fusion of association features and individual data without revealing the user's own association features. Furthermore, by inputting the first-party intermediate sample features and the second-party intermediate sample features into the first main task prediction model, the target federated prediction result is obtained. This achieves implicit prediction using the association data of each participant, improving prediction accuracy. Compared to training graph neural networks based on their own relational features and using the node representations obtained by each party for federated modeling, this application introduces a relational prediction model. By optimizing the feature extractors of the participants through the relational prediction model, similar relations among the participants can be extracted. Based on similar relations, the discriminativeness and accuracy of the second-party intermediate sample features extracted by the feature extractors of the participants are improved. Using more accurate features for model prediction can effectively improve prediction accuracy. Therefore, this overcomes the technical defect of vertical federated modeling, which is difficult to effectively utilize the relational data distributed among the participants, resulting in poor vertical federated prediction performance, and improves the performance of vertical federated prediction.

[0082] Example 4 Furthermore, embodiments of this application also provide a vertical federated learning model training device, which is applied to a first device. The first device is equipped with a first-party feature extractor to be trained, a first-party main task prediction model to be trained, and a first-party association prediction model to be trained. The vertical federated learning model training device includes: The first feature extraction module is used to acquire first training sample data, and to extract features from the first training sample data using a first-party feature extractor to obtain first training sample features, wherein the first training sample data includes first association data. The first receiving module is used to receive the second training sample features sent by the second device, wherein the second training sample features are obtained by the second device through a second feature extractor to extract features from the first matching sample data corresponding to the first training sample data, and the first matching sample data includes second association data. The loss determination module is used to determine the first main task model prediction loss of the first main task prediction model based on the first training sample features and the second training sample features, and to determine the first association model prediction loss of the first association prediction model based on the second training sample features. The sending module is used to determine a first gradient based on the prediction loss of the first main task model, the prediction loss of the first association model, and the features of the second training samples, and send the first gradient to the second device so that the second device can update the second-party feature extractor based on the first gradient.

[0083] Optionally, after determining the first primary task model prediction loss of the first primary task prediction model based on the first training sample features and the second training sample features, the longitudinal federated learning model training device further includes a first model update module, which is used to: Obtain the main task label corresponding to the first training sample data; By inputting the features of the first training sample and the features of the second training sample into the first main task prediction model, the prediction result of the first main task model is obtained; Based on the prediction results of the first main task model and the main task label, the first main task prediction model and the first feature extractor are updated by calculating the prediction loss of the first main task prediction model.

[0084] Optionally, after the operation of determining the first association model prediction loss of the first association prediction model based on the features of the second training samples, the longitudinal federated learning model training device further includes a second model update module, which is used to: Obtain the first association label corresponding to the first training sample data; By inputting the features of the second training sample into the first association prediction model, the prediction result of the first association model is obtained; Based on the prediction results of the first association relationship model and the first association relationship label, the first association relationship prediction model is updated by calculating the prediction loss of the first association relationship prediction model.

[0085] Optionally, the first association label includes at least one dimension-related association label, the first-party association prediction model includes at least one dimension-related first-party association prediction sub-model to be trained, and the second model update module is further configured to: By inputting the features of the second training sample into each first-party association prediction sub-model, the prediction results of the first association model corresponding to each dimension are obtained; Based on the prediction results of the first association model corresponding to each dimension and the dimension association label, calculate the association sub-model prediction loss of each first-party association prediction sub-model. The average value of the prediction loss of each of the association sub-models is determined as the first association model prediction loss, and the first association prediction model is updated based on the first association model prediction loss.

[0086] Optionally, the second model update module is further configured to: By querying the first association label, at least one set of first target training sample features that have an association relationship and at least one set of second target training sample features that do not have an association relationship are selected from the second training sample features; The features of each group of first target training samples are concatenated into positive sample data, and the features of each group of second target training samples are concatenated into negative sample data. Select positive and negative sample data that meet the preset positive and negative sample ratio, input them into the first-party association prediction model, and obtain the prediction results of the first association model.

[0087] Optionally, the longitudinal federated learning model training device further includes an unsupervised learning module, which is used for: Obtain the third training sample data and the second association label corresponding to the third training sample data; The third training sample data is processed by a first-party feature extractor to obtain the third training sample features. The third training sample features are then sent to the second device so that the second device can train the model based on the third training sample features. The device receives a fourth training sample feature sent by a second device, wherein the fourth training sample feature is obtained by the second device through a second feature extractor extracting features from the second matching sample data corresponding to the third training sample data; Based on the features of the third training sample and the features of the fourth training sample, unsupervised learning is performed. The first-party feature extractor is updated by calculating the unsupervised loss. Based on the features of the fourth training sample and the second association label, the first-party association prediction model is updated by calculating the second association model prediction loss of the first-party association prediction model. The second gradient is calculated based on the unsupervised loss, the prediction loss of the second association model, and the features of the fourth training sample. The second gradient is then sent to the second device so that the second device can update the second-party feature extractor based on the second gradient.

[0088] Optionally, the unsupervised learning module is further configured to: Transformed features are obtained by inputting the features of the fourth training sample into the projection model; Based on the transformation features and the third training sample features, the negative cosine similarity loss is calculated.

[0089] The vertical federated learning model training device provided by this invention employs the vertical federated learning model training method in the above embodiments, solving the technical problem that the existing technology has poor performance in predicting user behavior through vertical federated learning modeling. Compared with the prior art, the beneficial effects of the vertical federated learning model training device provided by this invention are the same as those of the vertical federated learning model training method provided in the above embodiments, and other technical features in this vertical federated learning model training device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0090] Example 5 Furthermore, embodiments of this application also provide a vertical federated learning prediction optimization device, which is applied to a third device. The third device is equipped with a trained first-party feature extractor, a trained first-party main task prediction model, and a trained first-party association prediction model. The vertical federated learning prediction optimization device includes: The second feature extraction module is used to acquire the sample data to be predicted, and to extract features from the sample data to be predicted by the first-party feature extractor to obtain the first-party intermediate sample features. The first-party feature extractor is trained using the longitudinal federated learning model training method described above. The second receiving module is used to receive the second intermediate sample features sent by the second device. The second intermediate sample features are obtained by the second device through feature extraction of the third matching sample data corresponding to the sample data to be predicted by the second feature extractor. The second feature extractor is trained using the longitudinal federated learning model training method described above. The prediction module is used to obtain the target federated prediction result by inputting the first party intermediate sample features and the second party intermediate sample features into the first party main task prediction model, wherein the first party main task prediction model is trained using the longitudinal federated learning model training method described above.

[0091] The vertical federated learning prediction optimization device provided by this invention employs the vertical federated learning prediction optimization method in the above embodiments, solving the technical problem that the existing technology has poor performance in user behavior prediction through vertical federated learning modeling. Compared with the prior art, the beneficial effects of the vertical federated learning prediction optimization device provided by this invention are the same as those of the vertical federated learning prediction optimization method provided in the above embodiments, and other technical features in this vertical federated learning prediction optimization device are the same as those disclosed in the method of the above embodiments, and will not be repeated here.

[0092] Example 6 Furthermore, embodiments of the present invention provide an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the longitudinal federated learning model training method in the above embodiments.

[0093] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as Bluetooth headsets, mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0094] like Figure 7 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and arrays required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0095] Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. Communication devices allow electronic devices to communicate wirelessly or wiredly with other devices to exchange arrays. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0096] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined above in the methods of embodiments of this disclosure.

[0097] The electronic device provided by this invention employs the vertical federated learning model training method described in the above embodiments, solving the technical problem that the existing technology performs poorly in predicting user behavior through vertical federated learning modeling. Compared with the prior art, the beneficial effects of the electronic device provided by this invention are the same as those of the vertical federated learning model training method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0098] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0100] Example 7 Furthermore, this embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, the computer-readable program instructions being used to execute the longitudinal federated learning model training method in the above embodiment.

[0101] The computer-readable storage medium provided in this embodiment of the invention may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0102] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0103] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the following: it acquires first training sample data; it extracts features from the first training sample data using a first-party feature extractor to obtain first training sample features, wherein the first training sample data includes first association data; it receives second training sample features sent by a second device, wherein the second training sample features are obtained by the second device through a second-party feature extractor from first matching sample data corresponding to the first training sample data, wherein the first matching sample data includes second association data; it determines a first main task model prediction loss of a first-party main task prediction model based on the first training sample features and the second training sample features, and determines a first association model prediction loss of a first-party association prediction model based on the second training sample features; it determines a first gradient based on the first main task model prediction loss, the first association model prediction loss, and the second training sample features, and sends the first gradient to the second device so that the second device can update the second-party feature extractor based on the first gradient.

[0104] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device can also: acquire sample data to be predicted; extract features from the sample data to be predicted using a first-party feature extractor to obtain first-party intermediate sample features, wherein the first-party feature extractor is trained using the vertical federated learning model training method described above; receive second-party intermediate sample features sent by a second device, wherein the second-party intermediate sample features are obtained by the second device through feature extraction from third matching sample data corresponding to the sample data to be predicted using a second-party feature extractor, wherein the second-party feature extractor is trained using the vertical federated learning model training method described above; and obtain a target federated prediction result by inputting the first-party intermediate sample features and the second-party intermediate sample features into a first-party main task prediction model, wherein the first-party main task prediction model is trained using the vertical federated learning model training method described above.

[0105] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0108] The computer-readable storage medium provided by this invention stores computer-readable program instructions for executing the above-described vertical federated learning model training method, thus solving the technical problem that the existing technology performs poorly in user behavior prediction through vertical federated learning modeling. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiments of this invention are the same as the beneficial effects of the vertical federated learning model training method provided in the above embodiments, and will not be repeated here.

[0109] Example 8 Furthermore, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the longitudinal federated learning model training method or the longitudinal federated learning prediction optimization method as described above.

[0110] The computer program product provided in this application solves the technical problem that the existing technology performs poorly in predicting user behavior through vertical federated learning modeling. Compared with the prior art, the beneficial effects of the computer program product provided in this embodiment are the same as the beneficial effects of the vertical federated learning model training method provided in the above embodiments, and will not be repeated here.

[0111] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A method for training a longitudinal federated learning model, characterized in that, The vertical federated learning model training method is applied to a first device, which is equipped with a first-party feature extractor to be trained, a first-party main task prediction model to be trained, and a first-party association prediction model to be trained. The first-party main task prediction model is a user consumption behavior prediction model. The vertical federated learning model training method includes the following steps: First training sample data is acquired, and features are extracted from the first training sample data using a first-party feature extractor to obtain first training sample features. The first training sample data includes first individual data and first relationship data. The first individual data includes basic information, business data, and behavioral data. The basic information includes name, gender, age, education, occupation, address, and income data. The business data includes loan data and consumption data. The behavioral data includes overdue repayment behavior, recently viewed video types, and payment operation frequency. The first relationship data includes friendly relationship data, recommendation relationship data, and behavioral similarity relationship data between the user corresponding to the first individual data and other users. The friendly relationship data includes likes and friend relationships. The recommendation relationship data includes shared links and recommendations. The behavioral similarity relationship data includes store memberships and preference tags. The second training sample features are received from the second device, wherein the second training sample features are obtained by the second device through a second feature extractor by extracting features from the first matching sample data aligned with the first training sample data, and the first matching sample data includes second volume data and second association data. Based on the features of the first training sample and the features of the second training sample, the first main task prediction loss of the first main task prediction model is determined, and based on the features of the second training sample, the first association prediction loss of the first association prediction model is determined. A first gradient is determined based on the prediction loss of the first main task model, the prediction loss of the first association model, and the features of the second training samples. The first gradient is then sent to the second device so that the second device can update the second-party feature extractor based on the first gradient.

2. The training method for the vertical federated learning model as described in claim 1, characterized in that, After the step of determining the first primary task model prediction loss of the first primary task prediction model based on the first training sample features and the second training sample features, the method further includes: Obtain the main task label corresponding to the first training sample data; By inputting the features of the first training sample and the features of the second training sample into the first main task prediction model, the prediction result of the first main task model is obtained; Based on the prediction results of the first main task model and the main task label, the first main task prediction model and the first feature extractor are updated by calculating the prediction loss of the first main task prediction model.

3. The training method for the longitudinal federated learning model as described in claim 1, characterized in that, After the step of determining the prediction loss of the first association relationship prediction model based on the features of the second training samples, the method further includes: Obtain the first association label corresponding to the first training sample data; By inputting the features of the second training sample into the first association prediction model, the prediction result of the first association model is obtained; Based on the prediction results of the first association relationship model and the first association relationship label, the first association relationship prediction model is updated by calculating the prediction loss of the first association relationship prediction model.

4. The training method for the vertical federated learning model as described in claim 3, characterized in that, The first association label includes at least one dimension-related association label, and the first-party association prediction model includes at least one dimension-related first-party association prediction sub-model to be trained. The step of inputting the features of the second training sample into the first association prediction model to obtain the prediction result of the first association model includes: By inputting the features of the second training sample into each first-party association prediction sub-model, the prediction results of the first association model corresponding to each dimension are obtained; The step of updating the first-party association prediction model by calculating the prediction loss of the first-party association prediction model based on the prediction result of the first association model and the first association label includes: Based on the prediction results of the first association model corresponding to each dimension and the dimension association label, calculate the association sub-model prediction loss of each first-party association prediction sub-model. The average value of the prediction loss of each of the association sub-models is determined as the first association model prediction loss, and the first association prediction model is updated based on the first association model prediction loss.

5. The training method for the longitudinal federated learning model as described in claim 3, characterized in that, The step of inputting the features of the second training sample into the first association prediction model to obtain the prediction result of the first association model includes: By querying the first association label, at least one set of first target training sample features that have an association relationship and at least one set of second target training sample features that do not have an association relationship are selected from the second training sample features; The features of each group of first target training samples are concatenated into positive sample data, and the features of each group of second target training samples are concatenated into negative sample data. Select positive and negative sample data that meet the preset positive and negative sample ratio, input them into the first-party association prediction model, and obtain the prediction results of the first association model.

6. The training method for a vertical federated learning model as described in claim 1, characterized in that, Before the step of obtaining the first training sample data and extracting features from the first training sample data using a first-party feature extractor to obtain the features of the first training sample, the method further includes: Obtain the third training sample data and the second association label corresponding to the third training sample data; The third training sample data is processed by a first-party feature extractor to obtain the third training sample features. The third training sample features are then sent to the second device so that the second device can train the model based on the third training sample features. The device receives a fourth training sample feature sent by a second device, wherein the fourth training sample feature is obtained by the second device through a second feature extractor extracting features from the second matching sample data corresponding to the third training sample data; Based on the features of the third training sample and the features of the fourth training sample, unsupervised learning is performed. The first-party feature extractor is updated by calculating the unsupervised loss. Based on the features of the fourth training sample and the second association label, the first-party association prediction model is updated by calculating the second association model prediction loss of the first-party association prediction model. The second gradient is calculated based on the unsupervised loss, the prediction loss of the second association model, and the features of the fourth training sample. The second gradient is then sent to the second device so that the second device can update the second-party feature extractor based on the second gradient.

7. The training method for a vertical federated learning model as described in claim 6, characterized in that, The first device is also equipped with a projection model, and the step of performing unsupervised learning based on the features of the third training sample and the features of the fourth training sample includes: Transformed features are obtained by inputting the features of the fourth training sample into the projection model; Based on the transformation features and the third training sample features, the negative cosine similarity loss is calculated.

8. A longitudinal federated learning prediction optimization method, characterized in that, The vertical federated learning prediction optimization method is applied to a third device, which is equipped with a trained first-party feature extractor, a trained first-party main task prediction model, and a trained first-party association prediction model. The vertical federated learning prediction optimization method includes the following steps: Acquire sample data to be predicted, and extract features from the sample data to be predicted using a first-party feature extractor to obtain first-party intermediate sample features, wherein the first-party feature extractor is trained using the longitudinal federated learning model training method as described in any one of claims 1-7. The second device receives second-party intermediate sample features, wherein the second-party intermediate sample features are obtained by the second device through feature extraction of the third matching sample data corresponding to the sample data to be predicted by the second-party feature extractor, and the second-party feature extractor is trained using the longitudinal federated learning model training method as described in any one of claims 1-7. By inputting the first-party intermediate sample features and the second-party intermediate sample features into the first-party main task prediction model, the target federated prediction result is obtained, wherein the first-party main task prediction model is trained using the longitudinal federated learning model training method as described in any one of claims 1-7.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the steps of the longitudinal federated learning model training method according to any one of claims 1-7 or the longitudinal federated learning prediction optimization method according to claim 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and the computer-readable storage medium stores a program for implementing a longitudinal federated learning model training method. The program for implementing the longitudinal federated learning model training method is executed by a processor using the steps of the longitudinal federated learning model training method according to any one of claims 1-7 or the longitudinal federated learning prediction optimization method according to claim 8.

11. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor using the steps of the longitudinal federated learning model training method according to any one of claims 1-7 or the longitudinal federated learning prediction optimization method according to claim 8.

Images