Model construction optimization method, device, storage medium, and program product

Abstract

The application discloses a model construction optimization method and device, a storage medium and a program product. The method comprises the following steps: a first participant device inputs first original feature data into a first feature projection model; each piece of second projection feature data sent by a second participant device is received, and each piece of first projection feature data is combined into a data pair; the data pair is input into a first encoding model to obtain first encoding feature data; the first encoding feature data is input into a first classification model to obtain a classification result corresponding to the data pair, and the classification result is used to represent whether the projection feature data in the data pair corresponds to the same sample; after at least one round of pre-training update of each model, longitudinal federated learning is performed with the second participant device to obtain a target task model. The application realizes model pre-training by using unlabeled data, so that the unlabeled data can be used for participating in longitudinal federated learning, and the prediction accuracy of the model obtained through longitudinal federated learning can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model building optimization method, device, storage medium, and program product. Background Technology

[0002] Vertical federated learning addresses the data silo problem in the financial sector, where multiple parties collaborate to build business models. Its application scenarios involve situations where participating parties have significant user overlap but limited feature overlap, with different parties possessing information about users from different domains and perspectives. A typical scenario involves parties discovering a high proportion of overlapping users through user matching, but each party possesses a low proportion of corresponding labels, resulting in limited samples usable for training. Therefore, utilizing unlabeled data is a crucial practical issue. For example, in risk control, banks and e-commerce institutions can use their respective user data across different dimensions for joint vertical federated learning to obtain a model predicting loan repayment risk. However, the number of labeled users shared by banks and e-commerce institutions is limited, meaning the amount of training data available for the vertical federated learning process is insufficient, hindering model training and leading to low accuracy in predicting loan repayment risk. Summary of the Invention

[0003] The main objective of this invention is to provide a model building optimization method, device, storage medium, and program product. It aims to propose a method for model pre-training using unlabeled data in a vertical federated learning scenario, so that unlabeled data can be used to participate in vertical federated learning, thereby helping to improve the prediction accuracy of the model obtained by vertical federated learning.

[0004] To achieve the above objectives, the present invention provides a model construction optimization method. The method is applied to a first participating device in longitudinal federated learning, wherein the first participating device deploys a first feature projection model, a first encoding model, and a first classification model, and a second participating device in longitudinal federated learning deploys a second feature projection model. The method includes the following steps:

[0005] The first original feature data of each sample in the first participating device are respectively input into the first feature projection model for projection to obtain each first projected feature data;

[0006] The system receives each second projection feature data sent by the second participating device and forms a data pair with each first projection feature data. Each second projection feature data is obtained by the second participating device by inputting the second original feature data of each sample into the second feature projection model for projection.

[0007] The data pairs are input into the first encoding model to obtain the first encoded feature data. The data pairs include a first data pair labeled with a first label or a second data pair labeled with a second label. The first data pair consists of a first projection feature data and a second projection feature data corresponding to the same sample. The second data pair consists of a first projection feature data and a second projection feature data corresponding to different samples.

[0008] The first encoded feature data is input into the first classification model for classification to obtain the classification result corresponding to the data pair, wherein the classification result is used to characterize whether the two projected feature data in the data pair correspond to the same sample;

[0009] Based on the error between the corresponding label and the classification result, the first feature projection model, the first encoding model and the first classification model are pre-trained and updated in one round.

[0010] After at least one round of pre-training updates for each model, the target task model is obtained by longitudinal federated learning based on the updated first feature projection model, the first encoding model, and the second feature projection model and the second participating device. During the pre-training update process for each model, the second feature projection model is updated by the second participating device.

[0011] Optionally, the step of receiving each second projection feature data sent by the second participating device and forming data pairs with each first projection feature data includes:

[0012] For a first target sample in each sample, calculate the sample distance between the first target sample and the other samples in each sample except the first target sample;

[0013] The sample whose sample distance is closest to the first target sample is selected as the second target sample, or the sample whose sample distance is less than a preset threshold is selected as the second target sample.

[0014] Add the first projection feature data corresponding to each second target sample to the first projection feature data corresponding to the first target sample, and obtain multiple first projection feature data corresponding to the first target sample;

[0015] The first data pairs are formed by combining multiple first projection feature data corresponding to the first target sample with the second projection feature data corresponding to the first target sample, and each first data pair is labeled with the first label.

[0016] Optionally, when the first original feature data is tabular data including the values ​​of the corresponding sample under multiple data items, the first feature projection model includes a vocabulary and a neural network, and the target original feature data is any one of the first original feature data.

[0017] The step of inputting the original target feature data into the first feature projection model to obtain the first projected feature data corresponding to the original target feature data includes:

[0018] From the embedded feature data corresponding to the various values ​​under the first data item in the vocabulary, select the embedded feature data corresponding to the values ​​under the first data item in the target original feature data as the first part of the projected feature data, wherein the first data item is a data item with discrete values;

[0019] The values ​​of each second data item in the original target feature data are input into the neural network for feature projection to obtain the second part of projected feature data, wherein the second data item is a data item with continuous values;

[0020] The first projection feature data corresponding to the original feature data of the target is obtained based on the first part of the projection feature data and the second part of the projection feature data.

[0021] Optionally, when the first encoding model is a self-attention mechanism-based encoding model, the first projected feature data and the second projected feature data each include multiple projected feature vectors, and the target data pair is any one of the data pairs. The step of inputting the target data pair into the first encoding model for encoding to obtain the first encoded feature data corresponding to the target data pair includes:

[0022] The projection feature vectors included in the first target projection feature data and the projection feature vectors included in the second target projection feature data are combined to form a feature vector sequence, wherein the first target projection feature data is the first projection feature data included in the target data pair, and the second target projection feature data is the second projection feature data included in the target data pair.

[0023] The feature vector sequence is input into the first encoding model for encoding to obtain the encoded feature vectors corresponding to each vector in the feature vector sequence.

[0024] The first encoded feature data corresponding to the target data pair is obtained based on the encoded feature vector.

[0025] Optionally, the second participating device further deploys a second encoding model and a second classification model, and the step of performing a pre-training update on the first feature projection model, the first encoding model, and the first classification model based on the error between the corresponding label and the classification result according to the data further includes:

[0026] Before updating each model in the next round of pre-training, the first target coding layer in the updated first coding model is sent to the aggregation device, so that the aggregation device can aggregate the first target coding layer and the second target coding layer received from the second participating device to obtain a third target coding layer. The first target coding layer is the nth coding layer in the updated first coding model, and the second target coding layer is the nth coding layer in the updated second coding model, where n is greater than 1.

[0027] After receiving the third target coding layer sent by the aggregation device, updating the first target coding layer in the first coding model using the third target coding layer, and then performing the next round of pre-training update for each model.

[0028] Optionally, after the step of inputting the first encoded feature data into the first classification model for classification to obtain the classification result corresponding to the data pair, the method further includes:

[0029] Based on the error between the corresponding label and the classification result, a first intermediate result is calculated to update the second feature projection model.

[0030] The first intermediate result is sent to the second participating device so that the second participating device can use the first intermediate result to perform a round of pre-training update on the second feature projection model.

[0031] Optionally, the first participating device further deploys a prediction model, and the step of obtaining the target task model by performing longitudinal federated learning with the second participating device based on the updated first feature projection model, the first encoding model, and the second feature projection model includes:

[0032] The first original feature data of each aligned sample in the first participating device are respectively input into the updated first feature projection model for projection to obtain each third projection feature data.

[0033] The device receives each fourth projection feature data sent by the second participating device, wherein each fourth projection feature data is obtained by the second participating device by projecting the second original feature data of each aligned sample into the updated second feature projection model.

[0034] The third projection feature data and the fourth projection feature data corresponding to the same alignment sample are combined and input into the updated first encoding model for encoding to obtain the second encoding feature data corresponding to each alignment sample.

[0035] Each of the second encoded feature data is input into the prediction model for prediction, and the prediction results corresponding to each of the aligned samples are obtained.

[0036] Based on the error between the predicted label corresponding to each aligned sample and the prediction result, a round of vertical federated update is performed on the first feature projection model, the first encoding model and the prediction model, and a second intermediate result for updating the second feature projection model is calculated;

[0037] The second intermediate result is sent to the second participating device so that the second participating device can perform a round of vertical federated update on the second feature projection model based on the second intermediate result;

[0038] After performing at least one round of vertical federated updates on each model, the target task model is obtained based on the updated first feature projection model, second feature projection model, first encoding model, and prediction model.

[0039] In addition, to achieve the above objectives, the present invention also provides a model building optimization device, the model building optimization device comprising: a memory, a processor, and a model building optimization program stored in the memory and executable on the processor, wherein the model building optimization program, when executed by the processor, implements the steps of the model building optimization method as described above.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a model building optimization program, which, when executed by a processor, implements the steps of the model building optimization method as described above.

[0041] Furthermore, to achieve the above objectives, the present invention also proposes a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the model building optimization method as described above.

[0042] In this invention, a first participating device in the vertical federated learning deploys a first feature projection model, a first encoding model, and a first classification model, while a second participating device deploys a second feature projection model. The first participating device inputs the first original feature data of each sample into the first feature projection model for projection to obtain each first projected feature data. It receives each second projected feature data sent by the second participating device and forms data pairs with each first projected feature data. Each second projected feature data is obtained by the second participating device inputting the second original feature data of each sample into the second feature projection model for projection. The data pairs are then input into the first encoding model for encoding to obtain first encoded feature data. Each data pair includes a first data pair labeled with a first label or a second data pair labeled with a second label. The first data pair is generated from the data of the same sample. The first data pair consists of a first projection feature data and a second projection feature data. The second data pair consists of a first projection feature data and a second projection feature data corresponding to different samples. The first encoded feature data is input into the first classification model for classification to obtain the classification result corresponding to the data pair. The classification result is used to characterize whether the two projection feature data in the data pair correspond to the same sample. The first feature projection model, the first encoding model, and the first classification model are pre-trained and updated based on the error between the label corresponding to the data pair and the classification result. After at least one round of pre-training and updating each model, the target task model is obtained by longitudinal federated learning based on the updated first feature projection model, the first encoding model, and the second feature projection model and the second participating device. During the pre-training and updating of each model, the second feature projection model is updated by the second participating device.

[0043] Compared to using labeled sample data from each participant to perform vertical federated learning on randomly initialized or manually initialized first feature projection models, second feature projection models, first encoding models, and prediction models, this invention uses pre-trained first feature projection models, second feature projection models, and first encoding models as the foundation for vertical federated learning. Since these models learn information related to sample features, during the vertical federated learning phase, they can encode more accurate prediction data, thus improving the prediction accuracy of the trained target task model and shortening the duration of vertical federated learning, thereby reducing computational resource consumption. Furthermore, it also enables the application of unlabeled sample raw feature data to model training in the vertical federated learning scenario, thereby improving model prediction accuracy based on unlabeled sample raw feature data. Additionally, during pre-training, the participating devices do not directly send the raw feature data of the samples, thus ensuring the privacy and security of data among the participants. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the hardware operating environment involved in the embodiments of the present invention;

[0045] Figure 2 This is a flowchart illustrating the first embodiment of the model construction optimization method of the present invention;

[0046] Figure 3 This is a schematic diagram of a pre-training architecture according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of a pre-training architecture according to an embodiment of the present invention.

[0048] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0051] It should be noted that the model building and optimization device in this embodiment of the invention can be a smartphone, a personal computer, a server, or other such device, and no specific limitations are imposed here.

[0052] like Figure 1 As shown, the optimized device for this model construction may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the model building optimization device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0054] like Figure 1 As shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a model building and optimization program. The operating system is a program that manages and controls the device's hardware and software resources, supporting the operation of the model building and optimization program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for the server to establish a communication connection; and the processor 1001 can be used to call the model building optimization program stored in the memory 1005 and execute the operations described in the following embodiments of the model building optimization method of the present invention.

[0055] Based on the above structure, various embodiments of the model building optimization method are proposed.

[0056] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the model construction optimization method of the present invention.

[0057] This invention provides an embodiment of a model building optimization method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order. In this embodiment, the model building optimization method is applied to a first participant device participating in longitudinal federated learning. The first participant device is deployed in the first participant of longitudinal federated learning, and devices deployed in other participants for longitudinal federated learning are referred to as second participant devices. The first and second participant devices can be devices such as smartphones, personal computers, and servers, and are not limited in this embodiment. In this embodiment, the model building optimization method includes:

[0058] Step S10: Input the first original feature data of each sample in the first participating device into the first feature projection model to project and obtain each first projected feature data;

[0059] In longitudinal federated learning, there is typically one participant that provides both feature data and label data (referred to as the "data user") and at least one participant that provides only feature data (hereinafter referred to as the "raw feature data" for distinction) (referred to as the "data provider"). In the application scenarios of longitudinal federated learning, the raw feature data from the data user and the data provider belong to different feature dimensions. For example, they might be images taken from different angles, or user data collected from different dimensions. The purpose of longitudinal federated learning is to combine the raw feature data of the same sample from various participants across different feature dimensions to perform modeling, thereby fully utilizing the features of each dimension of the sample to improve the model's prediction accuracy.

[0060] For ease of description, devices deployed on the data application side are referred to as data application side devices, and devices deployed on the data provider side are referred to as data provider side devices.

[0061] In specific application scenarios, model tasks can be set as needed, and models to be trained for each participant can be designed based on these tasks. The models to be trained are then deployed on the devices of each participant, and longitudinal federated learning is performed on these models to obtain the target task model. The model task refers to the intended use of the model after modeling, such as risk prediction or ad recommendation. That is, when the model task is risk prediction, the trained target task model can be used for risk prediction; when the model task is ad recommendation, the trained target task model can be used for ad recommendation. In this embodiment, no restrictions are placed on the model task.

[0062] To facilitate pre-training of the model to be trained using the raw feature data of unlabeled samples, thereby improving the training effect of the model to be trained in the longitudinal federated learning stage, in this embodiment, the model to be trained may include a feature projection model, an encoding model, and a prediction model deployed on the data application device, as well as a feature projection model deployed on the data provider device. The feature projection models in the data application device and the feature projection models in each data provider device are implemented using different types of models or different structures of the same type.

[0063] The feature projection model is used to project the original feature data of a sample from its own feature control to another feature space. This achieves the acquisition of feature data (hereinafter referred to as "projected feature data") that differs in data form from the original feature data without changing the specific information contained in the original feature data. When the data application device or data provider device projects its own original feature data through its own feature projection model to obtain projected feature data, and then sends the projected feature data to other parties' devices, the other parties' devices can input the projected feature data into subsequent models for processing, but cannot obtain the corresponding original feature data. This protects the privacy information carried in the original feature data of the sample from being leaked to other parties. The specific feature projection model used in this embodiment is not limited. It can be selected based on the data form of the original feature data. For example, when the original feature data is image data, the feature projection model can use a neural network, and the network parameters in the neural network can be updated during training. When the original feature data is text, the feature projection model can use a vocabulary, which records the embedded feature data corresponding to each character or word. The embedded feature data can be a vector or a matrix, and the embedded feature data can be updated during training.

[0064] An encoding model is used to encode feature data in order to extract implicit information useful for the model's prediction task. The specific model used for implementation is not limited in this embodiment; for example, it can be implemented using a multi-layer neural network, or a self-attention model (such as the encoder in a transformer model).

[0065] The type of prediction model selected can be set according to the prediction results required by the model task; for example, if the model task is binary classification, then the prediction model can use a binary classifier. Or, if the model task is multi-class classification, then the prediction model can use a multi-class classifier. Based on the output of the prediction model, it can be determined which category the sample belongs to.

[0066] To facilitate pre-training of the model using the raw feature data of unlabeled samples, in this embodiment, a classification model can be deployed in the data application device. This classification model can be implemented using a binary classifier to predict whether the raw feature data from the data application device and the raw feature data from the data provider device belong to the same sample. This classification task is used to pre-train the feature projection model and encoding model in the data application device. Alternatively, the feature projection model in the data provider device can also be pre-trained based on this classification task. In a specific implementation, no additional models need to be deployed in the data provider device. In this case, the feature projection model in the data provider device relies on the classification task in the data application device for pre-training. Alternatively, an encoding model and a classification model can also be deployed in the data provider device. This classification model is also used to predict whether the raw feature data from the data application device and the raw feature data from the data provider device belong to the same sample. Based on the data provider device's own classification task, the feature projection model in the data provider device can be pre-trained.

[0067] It should be noted that in the conventional longitudinal federated learning process, the parameters in the model are initialized randomly or based on human experience. In this case, the model's prediction accuracy is low. In fact, when the parameters are initialized randomly, the longitudinal federated learning process may take a long time to train the parameters to a state that allows the model to predict with high accuracy. Or, if the initial parameters are inaccurate in the overall direction, it may be impossible to train a model with the required prediction accuracy.

[0068] In this embodiment, it is proposed to pre-train the model before vertical federated learning. Pre-training refers to the pre-training of the feature projection model and encoding model in the data application device and the feature projection model in the data provider device before formally conducting vertical federated learning. In other words, compared to using the original feature data of labeled samples from each participant to perform longitudinal federated learning on randomly initialized or manually initialized feature projection models, encoding models, and prediction models, this embodiment uses pre-trained feature projection models and encoding models as the basis for longitudinal federated learning. The pre-trained feature projection models and encoding models can learn how to mine the relationships between the original feature data of samples from each participant. It can be figuratively understood that the randomly initialized or manually initialized feature projection models and encoding models are at the starting line, while the pre-trained feature projection models and encoding models have already run a certain distance from the starting line in the right direction (the direction of improving prediction accuracy). Thus, in the longitudinal federated learning stage, by projecting through the pre-trained feature projection model and then encoding through the pre-trained encoding model, more coded feature data that is more conducive to the prediction model to make accurate prediction results can be obtained. That is, it can improve the training effect of supervised modeling in subsequent longitudinal federated learning, thereby helping to improve the prediction accuracy of the target task model trained, and also helping to shorten the duration of longitudinal federated learning, thereby reducing the computational resource consumption of the longitudinal federated learning stage.

[0069] In this embodiment, one of the participants in the vertical federated learning is referred to as the first participant, and the other participants are referred to as the second participants. The device deployed in the first participant for vertical federated learning is referred to as the first participant device, and the device deployed in the second participant for vertical federated learning is referred to as the second participant device. In this embodiment, the first participant device can be either a data application device or a data provider device; there is no limitation in this embodiment.

[0070] The feature projection model, encoding model, and classification model deployed in the first participant's device will be referred to as the first feature projection model, the first encoding model, and the first classification model, respectively, for distinction. The feature projection model, encoding model, and classification model deployed in the second participant's device will be referred to as the second feature projection model, the second encoding model, and the second classification model, respectively, for distinction.

[0071] It is understood that, in specific implementations, when the first participating device is a data application device, the second participating device may not need to deploy the second encoding model and the second classification model; when the first participating device is a data provider device, the second participating device needs to deploy the second encoding model and the second classification model. However, when pre-training the first feature projection model and the first encoding model in the first participating device, it is not necessary to use the second encoding model and the second classification model in the second participating device.

[0072] Specifically, the pre-training of the first feature projection model, the second feature projection model, and the first encoding model may include one or more rounds of updates to each model. Hereinafter, the updates to the model during the pre-training phase will be referred to as pre-training updates, and the updates to the model during the longitudinal federated learning phase will be referred to as longitudinal federated updates, for distinction. The following explanation uses a single round of pre-training updates as an example to illustrate the pre-training process. It should also be noted that updating the model refers to updating the parameters within the model.

[0073] Before pre-training begins, the parameters in the first feature projection model, the second feature projection model, and the first encoding model can be randomly initialized or initialized based on experience.

[0074] The first participating party's device can acquire the original feature data (hereinafter referred to as "first original feature data" for distinction) of each sample owned by the first participating party, either locally or remotely. Each sample can correspond to at least one sample data entry. Each sample data entry includes the original feature data of that sample across all participating parties. The feature dimensions of the original feature data of that sample owned by each participating party are different, and can be seen as describing the sample from different perspectives. For example, in one embodiment, one participating party is a bank, and another participating party is an e-commerce institution. The bank possesses the financial activity data of the user (sample), such as deposit amount and loan amount, while the e-commerce institution possesses the user's purchase record data, such as the type of goods purchased and the amount of goods purchased. The model task can be to use the user's financial activity data and purchase record data to predict the user's loan repayment risk.

[0075] The samples used in the pre-training phase can be either aligned or unaligned. Aligned samples are those that all parties possess, but each party has different feature data for that sample. Samples without labeled data are called unlabeled samples. Since the pre-training process does not require the labeled data of the samples, the raw feature data of unlabeled samples can be used for pre-training.

[0076] After obtaining the first original feature data of each sample, the first participating device can input each first original feature data into the first feature projection model for projection to obtain the projection feature data corresponding to each sample (hereinafter referred to as "first projection feature data" for distinction).

[0077] The data format of the first projection feature data can be determined based on the data format of the input data of the first encoding model, and is not limited in this embodiment. For example, when the data format of the input data of the first encoding model is a vector, the first projection feature data can be a vector; when the data format of the input data of the first encoding model is a vector sequence (composed of multiple vectors), the first projection feature data can be a vector sequence.

[0078] For example, in one embodiment, when the first original feature data is image data, the first feature projection model can be implemented using a neural network. When the input data of the first encoding model is in the form of a vector, the output data of the neural network can also be set to a vector. The first participating device inputs the first original feature data into the neural network, and after the action of each neuron of the neural network, outputs a vector, which is the first projected feature data.

[0079] Step S20: Receive each second projection feature data sent by the second participating device and form a data pair with each first projection feature data, wherein each second projection feature data is obtained by the second participating device by inputting the second original feature data of each sample into the second feature projection model for projection;

[0080] The second participating device can acquire the original feature data (hereinafter referred to as "second original feature data") of each sample owned by the second participating device, either locally or remotely. It is understood that the feature dimensions of the second original feature data differ from those of the first original feature data. After acquiring the second original feature data of each sample, the second participating device can input each piece of second original feature data into the second feature projection model for projection, obtaining the projected feature data (hereinafter referred to as "second projected feature data") corresponding to each sample. It should be noted that the intersection of each sample from the first participating device and each sample from the second participating device is not empty; that is, there exist first and second projected feature data from the same sample. The second feature projection model and the first feature projection model have different model types, or the parameters in the model differ when sampling the same type of model. For example, when both the first and second feature projection models are neural networks, the parameters in the two neural networks are different. Similarly, when both the first and second feature projection models sample vocabulary lists, the embedded feature data corresponding to the same character or word in the two vocabulary lists are different. The specific implementation method of inputting the second original feature data into the second feature projection model to obtain the second projected feature data can refer to the specific implementation method of inputting the first original feature data into the first feature projection model to obtain the first projected feature data, which will not be repeated in this embodiment.

[0081] After obtaining the second projection feature data corresponding to each sample, the second participating device can send each second projection feature data to the first participating device.

[0082] Understandably, since the second projected feature data is obtained by projecting the second original feature data onto the second feature projection model through the second feature projection model, although the second projected feature data still carries the information carried by the second original feature data, the data format of the second projected feature data has changed compared to the second original feature data. Therefore, the first participating device cannot obtain the privacy information carried in the second original feature data by analyzing the second projected feature data. This ensures that the privacy data of the samples held by each party will not be leaked to other participating parties during the pre-training process, thereby guaranteeing the data security during the pre-training process.

[0083] Step S30: Input the data pair into the first encoding model to encode and obtain the first encoded feature data. The data pair includes a first data pair labeled with a first label or a second data pair labeled with a second label. The first data pair consists of a first projection feature data and a second projection feature data corresponding to the same sample. The second data pair consists of a first projection feature data and a second projection feature data corresponding to different samples.

[0084] The first participating device can combine each first projection feature data with each second projection feature data to obtain multiple data pairs, where each data pair consists of one first projection feature data and one second projection feature data. Based on the sample attribution of the two projection feature data in a data pair, the data pair is divided into a first data pair and a second data pair. In the first data pair, the first and second projection feature data correspond to the same sample (i.e., from aligned samples), while in the second data pair, the first and second projection feature data correspond to different samples (i.e., from unaligned samples). The first participating device assigns a first label to the first data pair and a second label to the second data pair. The first label indicates that the two projection feature data in the data pair come from the same sample, and the second label indicates that the two projection feature data in the data pair come from different samples.

[0085] It should be noted that the first and second participating devices can perform sample alignment in advance, for example, by using encrypted sample alignment technology to determine the aligned samples in each sample. The feature data of the aligned samples can be identified using the same ID, allowing the first participating device to identify whether the feature data comes from the same sample based on the ID of the feature data.

[0086] In a specific implementation, a sample may have one or more first projection feature data and second projection feature data. When a sample corresponds to multiple first projection feature data or multiple second projection feature data, the first projection feature data and second projection feature data can be combined in a permutation and combination manner to obtain multiple first data pairs.

[0087] The method for obtaining multiple first projection feature data corresponding to a sample is not limited here. For example, in one embodiment, the first participating device can use data augmentation to obtain multiple first projection feature data corresponding to a sample: for a target sample among the samples, the first participating device can perform data augmentation on the first original feature data of the target sample to obtain multiple augmented first original feature data corresponding to the target sample, and input each augmented first original feature data into the first feature projection model for projection to obtain multiple first projection feature data corresponding to the target sample; the data augmentation method is not limited in this embodiment. For example, when the first original feature data is image data, data augmentation can be performed by rotating the image. Similarly, multiple second projection feature data corresponding to a sample can also be obtained by the second participating device using data augmentation, which will not be elaborated here.

[0088] The first participant inputs each data pair into the first coding model for encoding, and obtains the corresponding coded feature data for each data pair (hereinafter referred to as "first coded feature data" for distinction).

[0089] In a specific implementation, when the first participating device inputs a data pair into the first encoding model, it can concatenate, calculate, or perform other processing operations on the first and second projection feature data in the data pair to obtain data in the form of input data conforming to the first encoding model, and then input it into the first encoding model for encoding. For example, in one implementation, when the first and second projection feature data are in vector form, and the input data of the first encoding model is in the form of a vector of length n, the first and second projection feature data can be concatenated or weighted to obtain a vector, which can then be padded or clipped to convert it to a length of n before being input into the first encoding model.

[0090] Step S40: Input the first encoded feature data into the first classification model for classification to obtain the classification result of the data pair, wherein the classification result is used to characterize whether the two projected feature data in the data pair correspond to the same sample;

[0091] After obtaining each first encoded feature data, the first participating device can input each first encoded feature data into the first classification model for classification, obtaining the classification result corresponding to each data pair. This classification result is used to characterize whether the two projected feature data in the corresponding data pair correspond to the same sample. It should be noted that at the beginning of pre-training, the classification result may not be accurate. The purpose of pre-training is to update each model through at least one round of pre-training so that the error between the classification result of the data pair and the label of the data pair becomes smaller and smaller, that is, to make the classification result more and more accurate. In this process, each model can learn how to accurately mine the related features between two projected feature data, and then accurately determine whether two projected feature data come from the same sample based on the features.

[0092] Step S50: Based on the error between the corresponding label and the classification result, perform a pre-training update on the first feature projection model, the first encoding model and the first classification model.

[0093] After obtaining the classification results corresponding to each data pair, the first participating device can calculate the error between the label of the data pair and the classification result, and update the parameters in the first feature projection model, the first encoding model and the first classification model with the aim of reducing the error, so as to update the first feature projection model, the first encoding model and the first classification model.

[0094] When the first participating device performs one round of pre-training updates on the first feature projection model, the first encoding model, and the first classification model, the second participating device also performs at least one round of pre-training updates on the second feature projection model. This embodiment does not limit the method by which the second participating device performs pre-training updates on the second feature projection model. For example, in one implementation, the second participating device may deploy a second encoding model and a second classifier. The first participating device sends the first projection feature data of each sample to the second participating device, and the second participating device updates the second feature projection model, the second encoding model, and the second classification model in the same way as the first participating device; specific details will not be elaborated here.

[0095] The method for calculating and updating parameters can specifically employ the gradient descent algorithm. Specifically, a loss function representing the error between the label and classification data of the data pair can be calculated. This loss function can be either the cross-entropy loss function or the binary cross-entropy loss function; no specific limitation is imposed in this embodiment. The gradient values ​​of the loss function relative to each parameter in the first feature projection model, the first encoding model, and the first classification model are calculated, and these gradient values ​​are used to update the parameters in each of these models.

[0096] Step S60: After performing at least one round of pre-training updates on each model, the target task model is obtained by longitudinal federated learning based on the updated first feature projection model, the first encoding model, and the second feature projection model and the second participating device. During the pre-training update process of each model, the second feature projection model is updated by the second participating device.

[0097] After one round of pre-training updates to each model (including the first feature projection model, the second feature projection model, the first encoding model, and the first classification model), it is checked whether the conditions for ending pre-training are met. If the conditions for ending pre-training are not met, another round of pre-training updates can be performed based on the updated models. If the conditions for ending pre-training are met, the first participating device can perform vertical federated learning with the second participating device based on the updated first feature projection model, the first encoding model, and the second feature projection model to obtain the target task model.

[0098] The conditions for ending pre-training can be that the number of pre-training update rounds reaches a preset number of rounds, the pre-training duration reaches a preset duration, or the error between the labels and classification results of each data pair converges. In this embodiment, there are no restrictions on the conditions for ending pre-training. In specific implementations, these conditions can be set according to specific needs.

[0099] The process of vertical federated learning based on the updated first feature projection model, first encoding model, and second feature projection model can refer to the conventional vertical federated learning process, and will not be elaborated in this embodiment.

[0100] In this embodiment, the principle underlying the pre-training process is analyzed as follows: In a vertical federated learning scenario, the original feature data of the same sample from different participants are correlated. Specifically, the original feature data of a sample from each participant are often collected in different scenarios, representing different aspects of the sample's characteristics, or different behaviors of the sample (when the sample is a user). The features and behaviors of a sample are consistent across different scenarios, while the features and behaviors of different samples in different scenarios are discriminative. Therefore, by determining whether each piece of original feature data from different participants belongs to the same sample, and using this as a pre-training task, each model in the pre-training process can learn highly discriminative representations (features). For example, for sample 1 and sample 2, with labels 0 and 1 respectively, both parties A and B have original feature data. Through pre-training, the model can, to some extent, distinguish that the original feature data of sample 1 in party A and the original feature data of sample 2 in party B do not belong to the same sample. Therefore, the model can learn some information related to the features of the samples, rather than having no feature extraction capability at all. That is, compared to models initialized randomly or based on human experience, the pre-trained model has already taken a certain distance in the right direction from the starting line. Using the pre-trained model for subsequent longitudinal federated learning can improve the training effect of supervised modeling in subsequent longitudinal federated learning. At the same time, it can increase the number of samples that can be used for training by using a large amount of unaligned and unlabeled original feature data, which is also an advantage of pre-training.

[0101] Therefore, compared to using labeled sample data from each participant to perform vertical federated learning on randomly initialized or manually initialized first feature projection models, second feature projection models, first encoding models, and prediction models, this embodiment uses pre-trained first feature projection models, second feature projection models, and first encoding models as the foundation for vertical federated learning. Since the first feature projection models, second feature projection models, and first encoding models learn some information related to the features of the samples, during the vertical federated learning stage, the processing by these models can encode coded feature data that is more conducive to the prediction model obtaining accurate prediction results. This helps improve the prediction accuracy of the trained target task model and shortens the duration of vertical federated learning, thereby reducing the computational resource consumption during the vertical federated learning stage. Furthermore, it also enables the application of unlabeled sample raw feature data to participate in model training in the vertical federated learning scenario, thus improving the model prediction accuracy based on unlabeled sample raw feature data. In addition, during the pre-training process, the participating devices do not directly send the raw feature data of the samples, thus ensuring the privacy and security of the data among the participating parties.

[0102] Further, in one embodiment, step S20 includes:

[0103] Step S201: For a first target sample in each sample, calculate the sample distance between the first target sample and the other samples in each sample except the first target sample;

[0104] To further improve the training efficiency and effectiveness of the pre-training phase, shorten the pre-training time, and reduce the computational resource consumption of the pre-training phase, in this embodiment, the amount of training data for pre-training can be increased by increasing the number of the first data pairs.

[0105] Specifically, taking one sample from each sample of the first participating device as an example (hereinafter referred to as the first target sample for distinction), the sample distance between the first target sample and the other samples in each sample, excluding the first sample, can be calculated.

[0106] The method for calculating the sample distance between two samples is not limited in this embodiment. For example, in one embodiment, the similarity between the first original feature data of the two samples can be calculated as the sample distance between the two samples. In another embodiment, the similarity between the first projected feature data of the two samples can be calculated as the sample distance between the two samples. The method for calculating the similarity between feature data is also not limited in this embodiment. For example, when the feature data is a vector, Euclidean distance, Manhattan distance, etc., can be used.

[0107] Step S202: Select the sample with the closest sample distance to the first target sample as the second target sample, or select the sample with the sample distance to the first target sample that is less than a preset threshold as the second target sample.

[0108] The first participating device selects a preset number of samples from among the other samples that are closest to the first target sample as the second target samples. The preset number can be set as needed and is not limited in this embodiment; for example, it can be set to 10, meaning that the 10 samples from among the other samples that are closest to the first target sample are selected as the second target samples.

[0109] Alternatively, the first participating device may also designate samples among the other samples whose sample distance from the first target sample is less than a preset threshold as the second target sample. The preset threshold can be set as needed and is not limited in this embodiment.

[0110] Step S203: Add the first projection feature data corresponding to each second target sample as the first projection feature data corresponding to the first target sample, and obtain multiple first projection feature data corresponding to the first target sample;

[0111] Since the sample distance between the second target sample and the first target sample is small, they can be considered as the same sample to a certain extent. Their corresponding first projected feature data can be considered to originate from the same sample. Therefore, the first projected feature data corresponding to each second target sample is added as the first projected feature data corresponding to the first target sample. That is, the first target sample corresponds to multiple first projected feature data, one of which comes from its own first original feature data, while the others come from the first original feature data of the second target samples.

[0112] Step S204: Combine the multiple first projection feature data corresponding to the first target sample with the second projection feature data corresponding to the first target sample to form the first data pair, and label each first data pair with the first label.

[0113] The first participating device combines each first projection feature data corresponding to the first target sample with the second projection feature data corresponding to the first target sample to form a first data pair, and assigns a first label to each first data pair. In this way, the number of first data pairs labeled with the first label can be increased, thereby improving the training efficiency and training effect in the pre-training stage, shortening the pre-training time, and reducing the computational resource consumption in the pre-training stage.

[0114] Furthermore, based on the first embodiment described above, a second embodiment of the model construction optimization method of the present invention is proposed. In this embodiment, when the first original feature data is tabular data including the values ​​of corresponding samples under multiple data items, since there may be data items with discrete values ​​and data items with continuous values ​​among the data items, how to set the first feature projection model for the values ​​of the two types of data items is a problem to be solved. In this embodiment, a neural network and a vocabulary are combined as the first feature projection model to address this problem. Based on the first feature projection model, the first original feature data is projected to obtain the first projected feature data. It should be noted that the implementation method of the second feature projection model can also refer to the first feature projection model.

[0115] For ease of description, this embodiment will be described using any one of the first original feature data as an example. For distinction, this first original feature data will be referred to as the target original feature data.

[0116] Step S10, which involves inputting the original target feature data into the first feature projection model to obtain the first projected feature data corresponding to the original target feature data, includes:

[0117] Step S101: Select the embedded feature data corresponding to the value under the first data item in the target original feature data from the embedded feature data corresponding to the various values ​​under the first data item in the vocabulary, and use it as the first part of the projection feature data, wherein the first data item is a data item with discrete values;

[0118] Data items with discrete values ​​are called the first data item, and data items with continuous values ​​are called the second data item. Discrete values ​​refer to values ​​that are not continuous and are specific discrete values. For example, gender is a discrete data item, while savings amount is a continuous data item.

[0119] For each first data item, the vocabulary can be configured with embedded feature data corresponding to various values ​​for that first data item, with different embedded feature data for different values. Each embedded feature data in the vocabulary can be initialized based on human experience or randomly. During pre-training, updating the first feature projection model specifically includes updating each embedded feature data. The data format of the embedded feature data is not limited in this embodiment. For example, when the input data of the first encoding model is a vector sequence, the embedded feature data can be a vector.

[0120] The first participating device can select the embedded feature data corresponding to the values ​​of the first data items in the target original feature data from the embedded feature data corresponding to the various values ​​under the first data item in the vocabulary. The selected embedded feature data is then used as the first part of the projected feature data corresponding to the target original feature data. It can be understood that when the embedded feature data is a vector and there are multiple first data items, the first part of the projected feature data is a vector sequence composed of the embedded feature data corresponding to the values ​​of each first data item.

[0121] For step S102, the values ​​of each second data item in the original target feature data are input into the neural network for feature projection to obtain the second part of projected feature data, wherein the second data item is a data item with continuous values;

[0122] The number of neurons and their connection structure in the neural network are not limited in this embodiment and can be set according to specific needs. The network parameters in the neural network can be initialized based on human experience or randomly. When updating the first feature projection model during pre-training, this specifically includes updating each network parameter.

[0123] The first participating device can input the values ​​of each second data item in the original target feature data into the neural network for feature projection, and the result is used as the second part of the projected feature data corresponding to the original target feature data.

[0124] The data format of the second part of the projection feature data (i.e. the data format of the output data of the neural network) is not limited in this embodiment. For example, when the data format of the input data of the first encoding model is a vector sequence, the second part of the projection feature data can be a vector sequence, or it can be a vector. The first participating device cuts the vector to obtain a vector sequence.

[0125] Step S103: Obtain the first projection feature data corresponding to the original feature data of the target based on the first part of the projection feature data and the second part of the projection feature data.

[0126] After obtaining the first part of the projected feature data and the second part of the projected feature data corresponding to the original feature data of the target, the first participating device can obtain the first projected feature data corresponding to the original feature data of the target based on the first part of the projected feature data and the second part of the projected feature data. In a specific embodiment, the first projected feature data can be obtained by splicing, calculating or otherwise processing the first part of the projected feature data and the second part of the projected feature data. That is, in this embodiment, the implementation method of obtaining the first projected feature data corresponding to the original feature data of the target based on the first part of the projected feature data and the second part of the projected feature data is not limited.

[0127] In one embodiment, when the input data of the first encoding model is in the form of a vector sequence, the data forms of the first projection feature data and the second projection feature data can also be set as vector sequences. When inputting the data pair into the first encoding model, the two vector sequences of the first projection feature data and the second projection feature data in the data pair can be concatenated before being input into the first encoding model. The length L of the vector sequence input to the first encoding model can be preset as needed; the lengths of the two vector sequences of the first projection feature data and the second projection feature data are n1 and n2, respectively. n1 and n2 can be equal or unequal, and can be set as needed, with n1 + n2 ≤ L. The data forms of the first part of the projection feature data and the second part of the projection feature data can also be vector sequences. The first participating device concatenates the first part of the projection feature data and the second part of the projection feature data, and the resulting vector sequence is used as the first projection feature data; the number of the first data items and the number of the second data items are respectively represented by d. s and d c The length of the first part of the projected feature data vector sequence is d. s To ensure the dimensionality balance of the vector sequence, the length K of the second part, the projected feature data vector sequence, should be d. c However, considering that the length of the vector sequence of the first projected feature data is n1, the value of K can be determined as follows:

[0128]

[0129] If the length of the vector sequence obtained by concatenating the first part of the projection feature data and the second part of the projection feature data is less than n1, then a set vector can be used to fill in the gaps to obtain a vector sequence of length n1, and this vector sequence can be used as the first projection feature data.

[0130] Furthermore, based on the first and / or second embodiments described above, a third embodiment of the model construction optimization method of the present invention is proposed. To further improve the pre-training effect, that is, to enhance the ability of each pre-trained model to mine sample features, in this embodiment, a self-attention-based encoding model is used as the first encoding model. For example, the encoder in the transformer model can be used as the first encoding model. It should be noted that the self-attention-based encoding model can more effectively extract relevant information between two input sequences, thus making it more suitable for encoding data pairs in classification tasks during the pre-training stage; that is, it can achieve better pre-training results.

[0131] When the first encoding model is based on a self-attention mechanism, the input data of the first encoding model can be set as a vector sequence of a certain length. Inputting the vector sequence into the first encoding model for encoding yields the encoded vectors corresponding to each vector in the vector sequence. Correspondingly, the first and second projected feature data can each be a vector sequence comprising multiple vectors (hereinafter referred to as "projected feature vectors" for distinction).

[0132] For ease of explanation, this embodiment will be described using any one of the data pairs as an example. For distinction, this data pair will be referred to as the target data pair.

[0133] The step S30, which involves inputting the target data pair into the first encoding model to encode the target data pair and obtain the first encoded feature data corresponding to the target data pair, includes:

[0134] Step S301: Combine the projection feature vectors included in the first target projection feature data with the projection feature vectors included in the second target projection feature data to form a feature vector sequence, wherein the first target projection feature data is the first projection feature data included in the target data pair, and the second target projection feature data is the second projection feature data included in the target data pair.

[0135] The first projection feature data and the second projection feature data included in the target data pair are respectively referred to as the first target projection feature data and the second target projection feature data for distinction. The first participating device can combine the projection feature vectors included in the first target projection feature data with the projection feature vectors included in the second target projection feature data to form a feature vector sequence. The combination method is not limited in this embodiment. For example, suppose the first target projection feature data is represented as (v1, v2, ..., v...). n1 The second target projection feature data is represented as (v1', v2', ..., v...). n2 In one embodiment, the resulting feature vector sequence can be (v1, v2, ..., v...). n1 v1', v2', ..., v n2 In another embodiment, a start vector can be added before the first target projection feature data, a separator vector can be added between the first and second target projection feature data, and an end vector can be added after the second target projection feature data. For example, the resulting feature vector sequence can be ([CLS], v1, v2, ..., v n1 ,[PAD],v1',v2',...,v n2 '、[PAD]).

[0136] Step S302: Input the feature vector sequence into the first encoding model for encoding to obtain the encoded feature vectors corresponding to each vector in the feature vector sequence;

[0137] The first participating device can input the combined feature vector sequence into the first encoding model for encoding, obtaining the encoded feature vectors corresponding to each vector in the feature vector sequence. For example, when the feature vector sequence is represented as ([CLS], v1, v2, ..., v... n1 ,[PAD],v1',v2',...,v n2 When encoding '、[PAD]), the encoded feature vectors corresponding to each vector in the feature vector sequence are obtained, represented as (E([CLS]), E(v1), E(v2), ..., E(v... n1 ), E([PAD]), E(v1'), E(v2'),..., E(v n2 '), E([PAD])).

[0138] Step S303: Obtain the first encoded feature data corresponding to the target data pair based on the encoded feature vector.

[0139] The first participating device can obtain the first encoded feature data corresponding to the target data pair based on the encoded feature vectors. In specific embodiments, the first participating device can obtain the first encoded feature data corresponding to the target data pair based on all encoded feature vectors, or it can obtain the first encoded feature data corresponding to the target data pair using only one or a portion of the encoded feature vectors. For example, in one embodiment, the first participating device can use the encoded feature vector corresponding to the first vector in the feature vector sequence as the first encoded feature data, such as using E([CLS]) as the first encoded feature data; in another embodiment, all encoded feature vectors can be averaged to obtain the first encoded feature data.

[0140] For example, in one implementation, such as Figure 3 As shown, the first participating device inputs the original feature data (corresponding to the original data in the figure) X into the feature projection model to obtain the feature vector sequence (v1, v2, ..., v...). n The second participating device projects the original feature data X' onto the feature projection model to obtain the feature vector sequence (v1', v2', ..., v...). n2 The first participating device will (v1, v2, ..., v) n ) and (v1', v2', ..., v n2The vectors [CLS] and [SEP] are concatenated and added. Then, they are input into the encoding model (the encoder in the transformer model is used in the figure) for encoding to obtain the encoded vector corresponding to the first vector [CLS] (represented as [CLS] in the figure). This vector is then input into the classification model for classification to obtain the classification result of whether they belong to the same user (sample).

[0141] Furthermore, based on the first, second, and / or third embodiments described above, a fourth embodiment of the model construction method of the present invention is proposed. In this embodiment, the second participating device further deploys a second encoding model and a second classification model, and step S50 further includes:

[0142] Step S501: Before updating each model in the next round of pre-training, the first target coding layer in the updated first coding model is sent to the aggregation device, so that the aggregation device can aggregate the first target coding layer and the second target coding layer received from the second participating device to obtain a third target coding layer, wherein the first target coding layer is the nth coding layer in the updated first coding model, and the second target coding layer is the nth coding layer in the updated second coding model, where n is greater than 1;

[0143] To further improve the effectiveness of the pre-training phase, in this embodiment, the second participating device may also deploy a second encoding model and a second classification model. The second participating device may also perform pre-training updates on the second feature projection model, the second encoding model, and the second classification model in the same manner as the first participating device. On this basis, the first and second participating devices may also aggregate some encoding layers in the first and second encoding models, thereby improving the training effect and training speed of the pre-training phase.

[0144] Specifically, before updating each model in the next round of pre-training, the first participating device can send the first target encoding layer in the updated first encoding model to the aggregation device. The aggregation device can be one of the first and second participating devices, or it can be a third-party device other than the first and second participating devices; this embodiment does not impose any restrictions. The first target encoding layer is the nth encoding layer in the updated first encoding model. Sending the first target encoding layer to the aggregation device can specifically involve sending the model parameters in the first target encoding layer to the aggregation device. Similarly, before updating each model in the next round of pre-training, the second participating device can also send the second target encoding layer in the updated second encoding model to the aggregation device. The second target encoding layer is the nth encoding layer in the updated second encoding model.

[0145] n is greater than 1, that is, the first participating device and the second participating device can send any coding layer in their respective coding models other than the first coding layer to the aggregation device for aggregation. For example, all coding layers other than the first layer can be sent to the aggregation device for aggregation.

[0146] The aggregation device aggregates the received first and second target coding layers. Specifically, the aggregation can involve averaging or weighted averaging the model parameters in the first and second target coding layers. The resulting layer is called the third target coding layer for distinction. The aggregation device then sends the third target coding layer to the first and second participating devices.

[0147] Step S502: Receive the third target coding layer sent by the aggregation device, update the first target coding layer in the first coding model using the third target coding layer, and then perform the next round of pre-training update for each model.

[0148] After receiving the third target encoding layer from the aggregation device, the first participating device can update the first target encoding layer in the first encoding model using the third target encoding layer. That is, it can update the model parameters corresponding to the first target encoding layer in the first encoding model using the model parameters corresponding to the third target encoding layer. After updating the first target encoding layer, the first participating device can perform the next round of pre-training updates for each model. Similarly, the second participating device can update the second target encoding layer in the second encoding model using the third target encoding layer, and then perform the next round of pre-training updates for each model.

[0149] It should be noted that in the first coding model, the features extracted by the coding layer that is closer to the input data are more relevant to the data, while the features extracted by the coding layer that is farther away from the input data are less relevant to the data. Therefore, in this embodiment, this characteristic is utilized to propose that the coding layers (i.e., non-first coding layers) that are far away from the input data in the coding models of each participating device can be aggregated, thereby further improving the ability to utilize unlabeled samples in the pre-training stage and thus improving the effect of pre-training.

[0150] Furthermore, in one embodiment, after step S40, the method further includes:

[0151] Step A10: Calculate the first intermediate result for updating the second feature projection model based on the error between the corresponding label and the classification result for each data pair;

[0152] Step A20: Send the first intermediate result to the second participating device so that the second participating device can use the first intermediate result to perform a round of pre-training update on the second feature projection model.

[0153] In this embodiment, a method for the second participating device to pre-train and update the second feature projection model is proposed. This method is applicable to scenarios where the second participating device does not deploy the second encoding model and the second classification model. Of course, it is also applicable to scenarios where the second participating device deploys the second encoding model and the second classification model.

[0154] Specifically, after obtaining the classification results for each data pair, the first participating device can calculate the error between the label and the classification result of the data pair. Based on the error, it calculates an intermediate result (hereinafter referred to as the first intermediate result for distinction) for updating the second feature projection model. The first intermediate result is then sent to the second participating device, which performs a pre-training update of the second feature projection model based on the first intermediate result. Specifically, a loss function representing the error between the label and the classification data of the data pair can be calculated. The loss function can be a cross-entropy loss function or a binary cross-entropy loss function, which is not limited in this embodiment. The gradient value of the loss function relative to the second projected feature data is calculated, and the gradient value of the second projected feature data is sent to the second participating device as the first intermediate result. Based on the first intermediate result, the second participating device uses the backpropagation algorithm to calculate the gradient value of the loss function relative to each model parameter in the second feature projection model, and then updates the model parameters in the second feature projection model based on each gradient value, thereby completing a pre-training update of the second feature projection model.

[0155] Furthermore, in one embodiment, when the second participating device is equipped with a second encoding model and a second classification model, the pre-training process of the first participating device and the second participating device may include the following two implementation methods.

[0156] The first method involves alternating sequential training between the two parties. During the t-th round of pre-training update, after receiving the second projection feature data from the second participant device, the first participant device constructs a data pair based on the second projection feature data and the first projection feature data calculated in this round. This data pair then performs a pre-training update on the first feature projection model, the first encoding model, and the first classification model. Simultaneously, it calculates the first intermediate result and sends it to the second participant device. The second participant device updates the second feature projection model based on the first intermediate result. The first participant device calculates the first projection feature data based on the updated first feature projection model and sends it to the second participant device. The second participant device constructs a data pair based on the first projection feature data and the second projection feature data calculated based on the current second feature projection model. This data pair then performs a pre-training update on the second feature projection model, the second encoding model, and the second classification model. Simultaneously, it calculates the intermediate result used to update the first feature projection model and sends it to the first participant device. The first participant device updates the first feature projection model based on the intermediate result, thus completing one round of pre-training update. This method provides more stable training results.

[0157] The second approach involves both parties immediately beginning their own training after receiving each other's data. The first and second participating devices each calculate projected feature data based on their current feature projection models and send it to the other. The other party then constructs data pairs based on its received projected feature data and its own projected feature data, updates its own model, and calculates corresponding intermediate results, sending them back to the other party for updating its feature projection model. This method eliminates the need for each party to wait for the other's intermediate results, resulting in higher training efficiency.

[0158] In one implementation, such as Figure 4 As shown, taking participant A and participant B as examples, participant A inputs the original feature data (corresponding to the original data in the figure) X into the feature projection model to obtain the feature vector sequence (v1, v2, ..., v...). n Participant B projects the original feature data X' onto the feature projection model to obtain the feature vector sequence (v1', v2', ..., v...). n2 '), and (v1', v2', ..., v n2 ') is sent to participant A; participant A will send (v1, v2, ..., v n ) and (v1', v2', ..., v n2The vectors [CLS] and [SEP] are concatenated and then input into the encoding model (the encoder in the transformer model is used in the figure) for encoding, resulting in the encoded vector corresponding to the first vector [CLS] (represented as [CLS] in the figure). This vector is then input into the classification model for classification to obtain the classification result of whether they belong to the same user (sample); Participant A calculates the loss function relative to (v1', v2', ..., v...). n2 The gradient value of ') (in the figure) ), and return it to participant B, who then uses Update the feature projection model; participant B performs the same operation as participant A; participants A and B will encode the portion of the transformer model encoder that is far from the input data (represented as transformer in the figure). up It is sent to a third-party device for aggregation, and the local encoding layer is updated using the aggregated encoding layer.

[0159] Furthermore, based on the first, second, third, and / or fourth embodiments described above, a fifth embodiment of the model construction optimization method of the present invention is proposed. In this embodiment, after step S60, the method further includes:

[0160] Step B10: Input the first original feature data of each aligned sample in the first participating device into the updated first feature projection model for projection to obtain each third projection feature data.

[0161] This embodiment proposes a specific implementation method for vertical federated learning based on a pre-trained model using a first participating device and a second participating device. Specifically, the first and second participating devices can perform vertical federated learning using the first and second original feature data of aligned samples. In this embodiment, the first participating device is a device of the participating party that has the label data of each aligned sample. The first participating device also deploys a prediction model to complete the model task in the vertical federated learning. Depending on the model task, the implementation method of the prediction model varies. For example, when the model task is to predict whether a user will default on repayments, the prediction model can be implemented using a binary classifier.

[0162] The vertical federated learning process may include at least one or more rounds of vertical federated updates to the first feature projection model, the second feature projection model, the first encoding model, and the prediction model. The following explanation uses a single round of vertical federated updates as an example to illustrate the vertical federated learning process.

[0163] The first participating device inputs the first original feature data of each aligned sample into the first feature projection model updated by pre-training for projection. The data obtained by projection is referred to as the third projection feature data.

[0164] Step B20: Receive each fourth projection feature data sent by the second participating device, wherein each fourth projection feature data is obtained by the second participating device by inputting the second original feature data of each aligned sample into the updated second feature projection model for projection.

[0165] The second participating device also inputs the second original feature data of each aligned sample into the pre-trained and updated second feature projection model for projection. For clarity, the data obtained from the projection is referred to as the fourth projection feature data. The second participating device then sends the fourth projection feature data to the first participating device.

[0166] Step B30: Combine one third projection feature data and one fourth projection feature data corresponding to the same alignment sample and input them into the updated first encoding model for encoding to obtain the second encoding feature data corresponding to each alignment sample.

[0167] For each aligned sample, the first participating device combines the third and fourth projection feature data corresponding to the aligned sample and inputs them into the pre-trained and updated first encoding model for encoding, thereby obtaining the encoded feature data corresponding to the aligned sample (hereinafter referred to as the second encoded feature data for distinction). Each aligned sample can obtain corresponding second encoded feature data.

[0168] It should be noted that the specific implementation method of the first participating device combining the third projection feature data and the fourth projection feature data and inputting them into the first encoding model is the same as the specific implementation method of inputting the data pair obtained by combining the first projection feature data and the second projection feature data into the first encoding model, and will not be described in detail here.

[0169] Step B40: Input each of the second encoded feature data into the prediction model for prediction to obtain the prediction results corresponding to each of the aligned samples;

[0170] Step B50: Perform a round of vertical federated updates on the first feature projection model, the first encoding model, and the prediction model based on the error between the predicted label corresponding to each aligned sample and the prediction result, and calculate a second intermediate result for updating the second feature projection model;

[0171] Step B60: Send the second intermediate result to the second participating device so that the second participating device can perform a round of vertical federated update on the second feature projection model based on the second intermediate result;

[0172] The first participating device inputs each second encoded feature data into the prediction model for prediction, thereby obtaining the prediction results corresponding to each aligned sample.

[0173] The first participating device performs a round of vertical federated updates on the first feature projection model, the first encoding model, and the prediction model based on the error between the predicted labels (i.e., label data related to the model task) and the prediction results for each aligned sample. The first participating device also calculates an intermediate result (hereinafter referred to as the second intermediate result for distinction) for updating the second feature projection model based on the error between the predicted labels and the prediction results, and sends the second intermediate result to the second participating device, which then performs a round of vertical federated updates on the second feature projection model based on the second intermediate result.

[0174] Specifically, the first participating device can calculate a loss function representing the error between the predicted label and the prediction result of the aligned sample. The loss function can be a cross-entropy loss function or a binary classification cross-entropy loss function, and there is no specific limitation in this embodiment. It calculates the gradient value of the loss function relative to each parameter in the first feature projection model, the first encoding model, and the prediction model, and uses each gradient value to update each parameter in the first feature projection model, the first encoding model, and the first classification model. It also calculates the gradient value of the loss function relative to the fourth projected feature data and sends this gradient value as a second intermediate result to the second participating device. The second participating device calculates the gradient value of the loss function relative to each model parameter in the second feature projection model using the gradient backpropagation algorithm based on the second intermediate result, and then uses each gradient value to update each model parameter in the second feature projection model. Thus, a round of vertical federated update of the second feature projection model is completed.

[0175] Step B70: After performing at least one round of vertical federated updates on each model, the target task model is obtained based on the updated first feature projection model, second feature projection model, first encoding model, and prediction model.

[0176] After performing one round of vertical federated updates on each model (including the first feature projection model, the second feature projection model, the first encoding model, and the prediction model), it is checked whether the conditions for ending vertical federated learning are met. If the conditions for ending vertical federated learning are not met, another round of vertical federated updates can be performed based on the updated models. If the conditions for ending vertical federated learning are met, the updated first feature projection model, the second feature projection model, the first encoding model, and the prediction model can be used as the target task model.

[0177] The conditions for ending vertical federated learning can be that the number of rounds of vertical federated updates reaches a preset number of rounds, the duration of vertical federated learning reaches a preset duration, or the error between the predicted labels and the prediction results of the aligned samples converges. In this embodiment, there are no restrictions on the conditions for ending vertical federated learning. In specific implementations, these conditions can be set according to specific needs.

[0178] Further, in one embodiment, the first participant can be a bank, which possesses business data generated when users (samples) conduct business with the bank, specifically including deposit and withdrawal records, loan records, repayment records, etc., and may also possess some users' loan repayment risk labels. The second participant can be an e-commerce institution, which possesses purchase record data generated when users purchase goods on the e-commerce platform, specifically including purchase amount, payment method, number of returns and exchanges, etc. The first and second participants can set up a model task to predict users' loan repayment risk based on users' business data and purchase record data. A first feature projection model, a first encoding model, and a prediction model are deployed on the first participant's device, and a second feature projection model is deployed on the second participant's device. By performing vertical federated learning on the first feature projection model, the second feature projection model, the first encoding model, and the prediction model using labeled shared users' business data and purchase record data, a loan repayment risk prediction model is obtained, which is used to predict users' loan repayment risk. The first participating device also deploys a first classification model. Before performing longitudinal federated learning on the first feature projection model, second feature projection model, first encoding model, and prediction model, the first and second participating devices use unlabeled user business data (first raw feature data) and purchase record data (second raw feature data) to pre-train the first feature projection model, second feature projection model, and first encoding model. One round of pre-training update process may include:

[0179] The first participating device inputs the business data of each user into the first feature projection model for projection to obtain each first projection feature data;

[0180] The first participating device receives each second projection feature data sent by the second participating device and forms a data pair with each first projection feature data. Each second projection feature data is obtained by the second participating device by inputting the purchase record data of each user into the second feature projection model for projection.

[0181] The first participating device inputs the data pair into the first encoding model to encode it and obtain the first encoded feature data. The data pair includes a first data pair labeled with a first label or a second data pair labeled with a second label. The first data pair consists of a first projection feature data and a second projection feature data corresponding to the same sample. The second data pair consists of a first projection feature data and a second projection feature data corresponding to different samples.

[0182] The first encoded feature data is input into the first classification model for classification to obtain the classification result corresponding to the data pair, wherein the classification result is used to characterize whether the two projected feature data in the data pair correspond to the same sample;

[0183] Based on the error between the corresponding labels and classification results, the first feature projection model, the first encoding model, and the first classification model are pre-trained and updated in one round.

[0184] After at least one round of pre-training and updating each model, a loan repayment risk prediction model is obtained by longitudinal federated learning based on the updated first feature projection model, first encoding model, and second feature projection model and the second participating device. During the pre-training and updating of each model, the second feature projection model is updated by the second participating device.

[0185] In this embodiment, business data and purchase record data of unlabeled shared users of banks and e-commerce institutions are used to pre-train the first feature projection model, the second feature projection model, and the first encoding model. Compared to using labeled sample data from banks and e-commerce institutions to perform vertical federated learning on randomly initialized or manually initialized first feature projection models, second feature projection models, first encoding models, and prediction models, this embodiment uses pre-trained first feature projection models, second feature projection models, and first encoding models as the foundation for vertical federated learning. The pre-trained first feature projection models, second feature projection models, and first encoding models have learned how to mine user feature-related information. This can be figuratively understood as the randomly initialized or manually initialized feature projection models and encoding models being at the starting line, while the pre-trained feature projection models and encoding models have already run a certain distance in the right direction (the direction of improving prediction accuracy). Therefore, in the vertical federated learning stage, the pre-trained first feature projection models, second feature projection models, and first encoding models can encode more coded feature data that is more conducive to the prediction model obtaining accurate prediction results. This helps improve the prediction accuracy of the trained loan repayment risk prediction model and also helps shorten the duration of vertical federated learning, thereby reducing the computational resource consumption in the vertical federated learning stage. Furthermore, it also enabled the application of unlabeled sample raw feature data to model training in vertical federated learning scenarios, thereby improving the model prediction accuracy in vertical federated learning scenarios based on unlabeled sample raw feature data. In addition, during the pre-training process, the participating devices did not directly send the raw feature data of the samples, thus ensuring the privacy and security of user data in banks and e-commerce institutions.

[0186] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a model building optimization program, wherein when the model building optimization program is executed by a processor, it implements the steps of the model building optimization method described below.

[0187] The present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the steps of the model building optimization method as described above.

[0188] The various embodiments of the model building and optimization device, computer-readable storage medium, and computer program product of the present invention can all refer to the various embodiments of the model building and optimization method of the present invention, and will not be repeated here.

[0189] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0190] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0192] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A model construction optimization method, characterized in that, The method is applied to a first participating device in vertical federated learning, wherein the first participating device deploys a first feature projection model, a first encoding model, and a first classification model, and a second participating device in vertical federated learning deploys a second feature projection model. The method includes the following steps: The first original feature data of each sample in the first participating device are respectively input into the first feature projection model for projection to obtain each first projected feature data; The system receives each second projection feature data sent by the second participating device and forms a data pair with each first projection feature data. Each second projection feature data is obtained by the second participating device by inputting the second original feature data of each sample into the second feature projection model for projection. The data pairs are input into the first encoding model to obtain the first encoded feature data. The data pairs include a first data pair labeled with a first label or a second data pair labeled with a second label. The first data pair consists of a first projection feature data and a second projection feature data corresponding to the same sample. The second data pair consists of a first projection feature data and a second projection feature data corresponding to different samples. The first encoded feature data is input into the first classification model for classification to obtain the classification result corresponding to the data pair, wherein the classification result is used to characterize whether the two projected feature data in the data pair correspond to the same sample; Based on the error between the corresponding label and the classification result, the first feature projection model, the first encoding model and the first classification model are pre-trained and updated in one round. After at least one round of pre-training updates for each model, the target task model is obtained by longitudinal federated learning based on the updated first feature projection model, the first encoding model, and the second feature projection model and the second participating device. During the pre-training update process for each model, the second feature projection model is updated by the second participating device.

2. The model construction and optimization method as described in claim 1, characterized in that, The step of receiving each second projection feature data sent by the second participating device and forming a data pair with each first projection feature data includes: For a first target sample in each sample, calculate the sample distance between the first target sample and the other samples in each sample except the first target sample; The sample whose sample distance is closest to the first target sample is selected as the second target sample, or the sample whose sample distance is less than a preset threshold is selected as the second target sample. Add the first projection feature data corresponding to each second target sample to the first projection feature data corresponding to the first target sample, and obtain multiple first projection feature data corresponding to the first target sample; The first data pairs are formed by combining multiple first projection feature data corresponding to the first target sample with the second projection feature data corresponding to the first target sample, and each first data pair is labeled with the first label.

3. The model construction and optimization method as described in claim 1, characterized in that, When the first original feature data is a table data including the values ​​of the corresponding sample under multiple data items, the first feature projection model includes a vocabulary and a neural network, and the target original feature data is any one of the first original feature data; The step of inputting the original target feature data into the first feature projection model to obtain the first projected feature data corresponding to the original target feature data includes: From the embedded feature data corresponding to the various values ​​under the first data item in the vocabulary, select the embedded feature data corresponding to the values ​​under the first data item in the target original feature data as the first part of the projected feature data, wherein the first data item is a data item with discrete values; The values ​​of each second data item in the original target feature data are input into the neural network for feature projection to obtain the second part of projected feature data, wherein the second data item is a data item with continuous values; The first projection feature data corresponding to the original feature data of the target is obtained based on the first part of the projection feature data and the second part of the projection feature data.

4. The model construction and optimization method as described in claim 1, characterized in that, When the first encoding model is a self-attention mechanism-based encoding model, the first projection feature data and the second projection feature data each include multiple projection feature vectors, and the target data pair is any one of the data pairs. The step of inputting the target data pair into the first encoding model for encoding to obtain the first encoded feature data corresponding to the target data pair includes: The projection feature vectors included in the first target projection feature data and the projection feature vectors included in the second target projection feature data are combined to form a feature vector sequence, wherein the first target projection feature data is the first projection feature data included in the target data pair, and the second target projection feature data is the second projection feature data included in the target data pair. The feature vector sequence is input into the first encoding model for encoding to obtain the encoded feature vectors corresponding to each vector in the feature vector sequence. The first encoded feature data corresponding to the target data pair is obtained based on the encoded feature vector.

5. The model construction and optimization method as described in claim 1, characterized in that, The second participating device is also deployed with a second encoding model and a second classification model. The step of performing a pre-training update on the first feature projection model, the first encoding model, and the first classification model based on the error between the corresponding label and the classification result according to the data further includes: Before updating each model in the next round of pre-training, the first target coding layer in the updated first coding model is sent to the aggregation device, so that the aggregation device can aggregate the first target coding layer and the second target coding layer received from the second participating device to obtain a third target coding layer. The first target coding layer is the nth coding layer in the updated first coding model, and the second target coding layer is the nth coding layer in the updated second coding model, where n is greater than 1. After receiving the third target coding layer sent by the aggregation device, updating the first target coding layer in the first coding model using the third target coding layer, and then performing the next round of pre-training update for each model.

6. The model construction and optimization method as described in claim 1, characterized in that, After the step of inputting the first encoded feature data into the first classification model for classification to obtain the classification result corresponding to the data pair, the method further includes: Based on the error between the corresponding label and the classification result, a first intermediate result is calculated to update the second feature projection model. The first intermediate result is sent to the second participating device so that the second participating device can use the first intermediate result to perform a round of pre-training update on the second feature projection model.

7. The model construction and optimization method according to any one of claims 1 to 6, characterized in that, The first participating device is also deployed with a prediction model. The step of obtaining the target task model by performing longitudinal federated learning with the second participating device based on the updated first feature projection model, the first encoding model, and the second feature projection model includes: The first original feature data of each aligned sample in the first participating device are respectively input into the updated first feature projection model for projection to obtain each third projection feature data. The device receives each fourth projection feature data sent by the second participating device, wherein each fourth projection feature data is obtained by the second participating device by projecting the second original feature data of each aligned sample into the updated second feature projection model. The third projection feature data and the fourth projection feature data corresponding to the same alignment sample are combined and input into the updated first encoding model for encoding to obtain the second encoding feature data corresponding to each alignment sample. Each of the second encoded feature data is input into the prediction model for prediction, and the prediction results corresponding to each of the aligned samples are obtained. Based on the error between the predicted label corresponding to each aligned sample and the prediction result, a round of vertical federated update is performed on the first feature projection model, the first encoding model and the prediction model, and a second intermediate result for updating the second feature projection model is calculated; The second intermediate result is sent to the second participating device so that the second participating device can perform a round of vertical federated update on the second feature projection model based on the second intermediate result; After performing at least one round of vertical federated updates on each model, the target task model is obtained based on the updated first feature projection model, second feature projection model, first encoding model, and prediction model.

8. A model building optimization device, characterized in that, The model building optimization device includes: a memory, a processor, and a model building optimization program stored in the memory and executable on the processor. When the model building optimization program is executed by the processor, it implements the steps of the model building optimization method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a model building optimization program, which, when executed by a processor, implements the steps of the model building optimization method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the model building optimization method as described in any one of claims 1-7.

Images