Longitudinal federal prediction optimization method and device, equipment, storage medium and product

Abstract

The invention discloses a longitudinal federal prediction optimization method and device, equipment, a storage medium and a product, and the method comprises the following steps: carrying out the sample alignment with second equipment, and determining an aligned sample; obtaining first-party training sample business data and business label prediction residual errors of the aligned samples; receiving training residual prediction results sent by the second devices; obtaining a weight parameter, performing weighted aggregation on each training residual prediction result according to the weight parameter to obtain a federated training residual, and updating the weight parameter based on the federated training residual; and determining a residual error prediction model gradient corresponding to each second device based on the federated training residual error and the service label prediction residual error, and sending each residual error prediction model gradient to the second device, so that each second device updates a self-deployed residual error prediction model based on the received residual error prediction model gradient. The technical problem of low robustness of longitudinal federated learning in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology in financial technology (Fintech), and in particular to a vertical federated prediction optimization method, device, equipment, storage medium and product. Background Art

[0002] With the continuous development of financial technology, especially Internet technology finance, more and more technologies (such as distributed, artificial intelligence, etc.) are applied in the financial field, but the financial industry also puts forward higher requirements for technology.

[0003] Artificial intelligence is increasingly used in the financial industry. Model training often requires a large amount of user data. Federated learning can effectively expand the scale of user data, thereby improving model performance. Vertical federated learning is to extract the users and data of the same participants but different user data features for joint machine learning training when the data features of the participants overlap less and the users overlap more. Each participant in vertical federated learning cooperates with the label holder to train a vertical federated model, which is used to perform label prediction for the label holder.

[0004] However, during the vertical federated training process, participants may send malicious intermediate results to label holders for malicious purposes to destroy the model, thereby interfering with the training of the vertical federated learning model. In the face of such attacks, existing related technologies can only protect simple vertical federated learning based on logistic regression against Byzantine attacks, or can only be used for horizontal federated learning. The scope of application is too narrow, resulting in the current vertical federated learning model having low robustness when facing Byzantine attacks launched by participants. Summary of the invention

[0005] The main purpose of this application is to provide a vertical federated prediction optimization method, device, equipment, storage medium and product, aiming to solve the technical problem of low robustness of vertical federated learning in related technologies.

[0006] To achieve the above-mentioned object, the present application provides a vertical federated prediction optimization method, which is applied to a first device in a vertical federated learning system, where a service label prediction model is deployed on the first device; the vertical federated learning system also includes multiple second devices, where a residual prediction model is deployed on each of the second devices; the vertical federated prediction optimization method includes the following steps:

[0007] Performing sample alignment with the second device to determine an aligned sample;

[0008] Obtaining the first-party training sample business data of the aligned sample, and obtaining the business label prediction residual generated by the trained business label prediction model performing business label prediction based on the first-party training sample business data;

[0009] Receiving a training residual prediction result sent by each of the second devices, where the training residual prediction result is obtained by each of the second devices performing residual prediction based on the second-party training sample service data of the aligned sample through the residual prediction model;

[0010] Obtaining a weight parameter, performing weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and updating the weight parameter based on a difference between the federated training residual and each of the training residual prediction results;

[0011] Based on the federal training residual and the business label prediction residual, the residual prediction model gradient corresponding to each of the second devices is determined, and each of the residual prediction model gradients is sent to the corresponding second device, so that each of the second devices updates its deployed residual prediction model based on the received residual prediction model gradient.

[0012] The present application also provides a longitudinal federation prediction optimization method, which is applied to a third device and includes the following steps:

[0013] Obtaining first-party sample business data to be predicted of the sample to be predicted, and obtaining a local business label prediction result obtained by a business label prediction model based on the first-party sample business data to be predicted;

[0014] Performing sample alignment with multiple second devices, so that the target second device including the second party sample service data to be predicted corresponding to the sample to be predicted in each of the second devices performs prediction based on the second party sample service data to be predicted by using the residual prediction model deployed by the second party to obtain a residual prediction result;

[0015] Receiving the residual prediction results sent by each of the target second devices, obtaining trained weight parameters, and weightedly aggregating each of the residual prediction results according to the trained weight parameters to obtain a federated residual, wherein the weight parameters are trained using the longitudinal federated prediction optimization method as described above;

[0016] The local service label prediction result and the federation residual are aggregated to obtain a vertical federation prediction result.

[0017] The present application also provides a vertical federated prediction optimization device, which is applied to a first device in a vertical federated learning system, wherein a service label prediction model is deployed on the first device; the vertical federated learning system further includes a plurality of second devices, each of which is deployed with a residual prediction model; the vertical federated prediction optimization device includes:

[0018] an alignment module, configured to perform sample alignment with the second device and determine an alignment sample;

[0019] A first acquisition module is used to acquire the first-party training sample business data of the aligned sample, and acquire the business label prediction residual generated by the trained business label prediction model performing business label prediction based on the first-party training sample business data;

[0020] A receiving module, configured to receive a training residual prediction result sent by each of the second devices, wherein the training residual prediction result is obtained by each of the second devices performing residual prediction based on the second-party training sample service data of the aligned sample through the residual prediction model;

[0021] A first aggregation module is used to obtain a weight parameter, perform weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and update the weight parameter based on a difference between the federated training residual and each of the training residual prediction results;

[0022] The first update module is used to determine the residual prediction model gradient corresponding to each of the second devices based on the federal training residual and the business label prediction residual, and send each of the residual prediction model gradients to the corresponding second devices, so that each of the second devices updates its deployed residual prediction model based on the received residual prediction model gradients.

[0023] The present application also provides a longitudinal federation prediction optimization device, which is applied to a third device and includes:

[0024] A second acquisition module is used to acquire the first-party sample business data to be predicted of the sample to be predicted, and to acquire the local business label prediction result obtained by the business label prediction model based on the first-party sample business data to be predicted;

[0025] A residual prediction module, configured to perform sample alignment with a plurality of second devices, so that a target second device including the second party's sample service data to be predicted corresponding to the sample to be predicted in each of the second devices performs prediction based on the second party's sample service data to be predicted through a residual prediction model deployed by the second device, to obtain a residual prediction result;

[0026] A second aggregation module is used to receive the residual prediction results sent by each of the target second devices, obtain the trained weight parameters, and perform weighted aggregation on each of the residual prediction results according to the trained weight parameters to obtain a federated residual, wherein the weight parameters are trained using the longitudinal federated prediction optimization method as described above;

[0027] The third aggregation module is used to aggregate the local service label prediction result and the federal residual to obtain a vertical federal prediction result.

[0028] The present application also provides an electronic device, which is a physical device, and includes: a memory, a processor, and a program of the vertical federated prediction optimization method stored in the memory and executable on the processor. When the program of the vertical federated prediction optimization method is executed by the processor, the steps of the vertical federated prediction optimization method as described above can be implemented.

[0029] The present application also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a program for implementing the vertical federated prediction optimization method. When the program of the vertical federated prediction optimization method is executed by a processor, the steps of the vertical federated prediction optimization method as described above are implemented.

[0030] The present application also provides a computer program product, including a computer program, which implements the steps of the vertical federated prediction optimization method as described above when executed by a processor.

[0031] The present application provides a vertical federated prediction optimization method, apparatus, device, storage medium and product. The vertical federated prediction optimization method is applied to a first device in a vertical federated learning system, and a business label prediction model is deployed on the first device; the vertical federated learning system also includes multiple second devices, and a residual prediction model is deployed on each of the second devices. First, sample alignment is performed with the second device to determine the aligned sample, and the first-party training sample business data and the first sample business label of the aligned sample are obtained, and the business label training prediction result obtained by the trained business label prediction model based on the first-party training sample business data for business label prediction is obtained. According to the first sample business label and the business label training prediction result, the business label prediction residual is determined. The business label prediction residual determined in this way is the true value of the residual generated by the model prediction of the trained business label prediction model, thereby achieving the purpose of determining the true value of the residual; and then the training residual prediction result sent by each second device is received. The training residual prediction result is obtained by each second device through the residual prediction model based on the second-party training sample business data of the aligned sample for residual prediction. In this way, the second-party training sample business data owned by the second device can be used for residual prediction, thereby achieving The purpose of determining the predicted value of the residual is to obtain a weight parameter, and then weight the training residual prediction results according to the weight parameter to obtain a federal training residual, and update the weight parameter based on the difference between the federal training residual and each training residual prediction result. In this way, by setting the weight and adjusting the weight based on the training residual prediction result returned by each second device, the influence of the training residual prediction result provided by each second device on the vertical federation training can be adjusted. The training residual prediction result provided by a single malicious individual will obviously deviate from the federal training residual. On the one hand, the final use of the federal training residual can be used to limit malicious The influence of individuals on the training effect, on the other hand, can also be further reduced by updating the weight coefficient to further reduce the influence of malicious individuals on the whole, and then based on the federated training residual and the business label prediction residual, determine the residual prediction model gradient corresponding to each of the second devices, and send each of the residual prediction model gradients to the corresponding second devices, so that each of the second devices can update its own deployed residual prediction model based on the received residual prediction model gradient. In this way, the influence of malicious second devices on the residual prediction model gradient can be reduced through the setting of weights, thereby effectively reducing the influence of malicious second devices on the vertical federated learning model. Therefore, it overcomes the technical defect that in the vertical federated training process, if the participants send malicious intermediate results to the label holder, it will interfere with the training of the vertical federated learning model, and improves the robustness of the model.Moreover, compared with the business label prediction model obtained only based on local data training, vertical federated learning can utilize a variety of sample business data distributed on multiple second devices to more accurately predict the residual between the predicted results of the business label prediction model and the actual results, thereby correcting the prediction results of the business label prediction model and improving the prediction accuracy of business labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 This is a flow chart of the first embodiment of the vertical federated forecasting optimization method of the present application;

[0035] Figure 2 A schematic diagram of a scenario of a possible implementation method of the federated learning system in the embodiment of the present application;

[0036] Figure 3 This is a flow chart of the third embodiment of the vertical federated forecasting optimization method of the present application;

[0037] Figure 4 This is a schematic diagram of the structure of the longitudinal federation prediction optimization device in the embodiment of the present application;

[0038] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the vertical federated prediction optimization method in the embodiment of the present application.

[0039] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0041] Artificial intelligence is increasingly used in the financial industry. Model training often requires a large amount of user data. The user data that each company can obtain is limited and the features are relatively simple, so the user portraits portrayed may be inaccurate. If each company only trains the model based on its own local user data, the performance of the prediction model obtained will be limited and the prediction accuracy will be low. Therefore, federated learning can be used to expand the scale of user data and thus improve model performance.

[0042] Vertical federated learning is to extract the users and data of the same participants but different user data features for joint machine learning training when the data features of the participants overlap less but the users overlap more. In the vertical federated learning system, it usually includes participants with label information and participants without label information. For the sake of convenience, in the subsequent embodiments, the first device refers to the participant with label information, and the first device needs to perform the prediction task. The second device refers to the participant without label information, and the second device can assist the first device in completing the prediction task and improve the prediction accuracy of the prediction task.

[0043] In the related art, the first device and each second device work together to train a vertical federation model, and the vertical federation model is used to perform label prediction for the first device. However, during the vertical federation training process, if a participant sends a malicious intermediate result to the label holder, it will interfere with the training of the vertical federation learning model.

[0044] Exemplarily, the business tag prediction task may be the prediction of information click-through rate, the first device may be a device of an information recommendation platform, such as a video website, a short video platform, etc., the second device may be a device of an e-commerce company, a financial institution, a bank, etc., the information recommendation platform may cooperate with e-commerce companies, financial institutions, banks, etc. to predict the information click behavior of common users, so that the information recommendation platform may more accurately determine to recommend information to users. The information recommendation platform has the user's click record, and the click record may be used as a sample business tag for model training. The information recommendation platform may also have the user's information browsing behavior, page dwell time, and other first-party sample business data, which may be used to locally train the business tag prediction model deployed by the party, so as to better predict the information click-through rate, and the second device may have other sample business data of the same user as the information recommendation platform, such as second-party sample business data such as purchase behavior, purchase preferences, and financial information, which may be used to learn the residual, and realize the prediction of the business tag prediction residual of the business tag prediction model deployed on the first device, so as to correct the information click-through rate predicted by the business tag prediction model deployed on the first device, and obtain a more accurate information click-through rate. During the training process of such a vertical federated learning system, if any one or more cooperative institutions send malicious intermediate results to the information recommendation platform, it will affect the accuracy of the residual prediction, thereby affecting the accuracy of the information recommendation platform's final prediction of the information click-through rate, and the robustness is low.

[0045] The present application deploys a business label prediction model on the first device and a residual prediction model on the second device, thereby dividing the entire prediction task into two parts: prediction and result correction based on residuals. In this way, the first device can train the business label prediction model only based on the locally owned sample business data. However, due to the limited locally owned sample business data, the performance of the business label prediction model is limited, the prediction accuracy is low, and there is still a large prediction residual between the prediction result of the business label prediction model and the actual result. Therefore, by utilizing the sample business data of the samples aligned with the first device by each second device, the prediction residual of the business label prediction model on the first device is predicted through the residual prediction model on each second device, and the prediction result of the business label prediction model is corrected by combining the prediction residuals of each second device. A vertical federal prediction result that is closer to the actual result can be obtained, thereby achieving the purpose of improving the prediction effect. Moreover, since the residual prediction model is deployed on the second device, during the entire training process, only the training residual prediction results and the residual prediction model gradients are shared between the first device and the second device. For the second device, since the first device cannot know the sample features based on which the second device obtains the training residual prediction results, it is impossible to infer the second-party training sample business data owned by the second device, so the privacy of the second device can be achieved; for the first device, since the residual is the difference between the first sample business label and the local model training prediction result, it is a reflection of the model performance. When the second device knows neither the first-party training sample business data nor the first sample business label nor the business label prediction model, the second device cannot infer the label information or privacy data owned by the first device based solely on the residual, so the privacy of the first device can be achieved. Furthermore, by setting weights and adjusting weights based on the training residual prediction results returned by each second device, the influence of the training residual prediction results provided by each second device on the vertical federated training can be adjusted. The training residual prediction results provided by a single malicious individual will obviously deviate from the federated training residuals. On the one hand, the final use of the federated training residuals can limit the influence of malicious individuals on the training effect through the federation as a whole. On the other hand, the influence of malicious individuals on the whole can be further reduced by updating the weight coefficient. Therefore, the technical defect that if a participant sends a malicious intermediate result to the label holder during the vertical federated training process, it will interfere with the training of the vertical federated learning model, and the robustness of the model is improved. Moreover, compared with the business label prediction model obtained by training only based on local data, the vertical federated learning can use a variety of sample business data distributed on multiple second devices to more accurately predict the residual between the prediction result of the business label prediction model and the actual result, thereby correcting the prediction result of the business label prediction model and improving the prediction accuracy of the business label.

[0046] Embodiment 1

[0047] The present application provides a method for optimizing vertical federated prediction. In the first embodiment of the method for optimizing vertical federated prediction, the method is applied to a first device in a vertical federated learning system, and a service tag prediction model is deployed on the first device; the vertical federated learning system further includes a plurality of second devices, and a residual prediction model is deployed on each of the second devices; Figure 1 , the longitudinal federated forecasting optimization method comprises the following steps:

[0048] Step S10, performing sample alignment with the second device to determine an alignment sample;

[0049] The executor of the method of this embodiment can be a vertical federated prediction optimization device, or a vertical federated prediction optimization terminal device or server. This embodiment takes a vertical federated prediction optimization device as an example. The vertical federated prediction optimization device can be integrated in terminal devices such as smart phones and computers with data processing functions.

[0050] In this embodiment, it should be noted that the vertical federated learning system includes at least one first device and a second device corresponding to each first device. For ease of understanding, the following description is based on a first device and multiple second devices corresponding to the first device. The vertical federated prediction optimization method includes a training process of a vertical federated learning model. The vertical federated learning model includes a residual prediction model deployed on each of the second devices. In addition, a business label prediction model is also deployed on the first device. The business label prediction model can be determined according to the actual business label prediction task and the form of sample data. For example, a convolutional neural network model can be selected for image data, and a classification model can be selected for classification tasks. This embodiment does not limit this; the residual prediction model can be a lifting model based on residual learning, and the business label prediction residual of the business label prediction model on the first device is fitted by executing a forward propagation algorithm. Therefore, the training process of the vertical federated learning model needs to be completed by the first device and the second device. The vertical federated prediction optimization method provided in this embodiment is applied to the first device in the vertical federated learning system.

[0051] In the training process of the vertical federated learning model, the business label prediction model is first trained, and then the residual prediction model is trained using the trained business label prediction model. The training of the business label prediction model can be performed on the first device or on other devices, and then the trained business label prediction model is deployed on the first device, and the residual prediction model is trained through the first device. This embodiment does not limit this. The business label prediction model is used to predict business labels based on sample business data owned by the first device. After the business label prediction model is trained, the business label prediction residual between the business label training prediction result obtained by the business label prediction model when executing the business label prediction task and the first sample business label owned by the first device can be used as the true value of the residual prediction of the residual prediction model on the second device; the residual prediction model is used to predict the business label prediction residual of the business label prediction model on the first device to obtain the predicted value of the business label prediction residual; when the true value and predicted value of the residual prediction of the residual prediction model are known, the residual prediction model on the second device can be updated by calculating the loss and gradient, and then by the gradient descent method.

[0052] In the process of calculating the gradient of the true value and predicted value of the residual prediction based on the residual prediction model, since federated learning can receive multiple prediction values ​​from multiple second devices, the multiple prediction values ​​received can be combined to perform joint training on the residual prediction models on the multiple second devices, and the influence of the prediction value provided by each second device on the vertical federated training can be adjusted by setting weights, so that the influence of malicious second devices on the gradient of the residual prediction model is reduced, thereby effectively reducing the influence of malicious second devices on the vertical federated learning model.

[0053] As an example, step S10 includes: the first device and the second device perform sample alignment to determine the aligned samples. The sample alignment relationship between each of the first-party training sample business data and each of the second-party training sample business data can be determined by the aligned sample ID, or the sample alignment relationship between each of the first-party training sample business data and each of the second-party training sample business data can be determined by other identification information of the aligned samples. The specific method can be determined according to actual conditions, and this embodiment does not limit this. For the sake of convenience, the aligned sample ID will be used as an example for explanation. The first device has the sample data of all aligned samples, and each second device may have part or all of the sample data of the aligned samples.

[0054] Step S20, obtaining the first-party training sample business data of the aligned sample, and obtaining the business label prediction residual generated by the trained business label prediction model performing business label prediction based on the first-party training sample business data;

[0055] In this embodiment, the first device may have first-party training sample business data of multiple aligned samples and first sample business labels corresponding to each of the aligned samples. Each second device may have part or all of the second-party training sample business data of the aligned samples. Through sample alignment, each second device can extract the second-party training sample business data of the aligned samples aligned with the first device from its own sample business data, wherein the second-party training sample business data refers to the sample business data of the aligned samples determined by the first device on the second device. For example, the first device has the deposit data of aligned samples S1, S2, S3 and S4, the first-party training sample business data is the deposit data of aligned samples S1, S2, S3 and S4, the second device P1 has the consumption data of S1, S2, S3 and S5, and the second device P2 has the credit data of S1, S4 and S6. Through sample alignment, it can be determined that the second-party training sample business data owned by P1 is the consumption data of S1, S2 and S3, and the second-party training sample business data owned by P2 is the credit data of S1 and S4.

[0056] As an example, step S20 includes: obtaining a batch of first-party training sample business data, and obtaining a predicted value of the business label by inputting the first-party training sample business data into a trained business label prediction model for business label prediction. The first device is the owner of the label information, and therefore the true value of the business label can be obtained, thereby determining the true value of the residual generated by the trained business label prediction model based on the first-party training sample business data for business label prediction, that is, the business label prediction residual. Since the true values ​​of the first-party training sample business data and the business labels are both owned by the first device, the business label prediction residual corresponding to the first-party training sample business data can be predetermined, or can be determined when the residual prediction model is trained, and this embodiment does not limit this.

[0057] Furthermore, the step of obtaining a service label prediction residual generated by performing service label prediction on the first-party training sample service data by the trained service label prediction model includes:

[0058] Step S11, obtaining a first sample business label, and obtaining a business label training prediction result obtained by performing business label prediction on the first-party training sample business data by a trained business label prediction model;

[0059] Step S12: determining a service label prediction residual according to the first sample service label and the service label training prediction result.

[0060] As an example, the steps S11-S12 include: obtaining a first sample business label corresponding to the first-party training sample business data; then, inputting the first-party training sample business data into a trained business label prediction model to perform business label prediction to obtain a business label training prediction result; then, based on the difference between the aligned sample and the business label training prediction result, calculating the business label prediction residual, the business label prediction residual being the true value of the residual generated by the business label prediction model performing business label prediction.

[0061] In one implementable manner, the business label prediction model may include a feature extractor, a fully connected layer and an activation layer. The step of inputting the first-party training sample business data into the trained business label prediction model to perform business label prediction and obtain the business label training prediction result includes: after inputting the first-party training sample business data into the trained business label prediction model, feature extraction can be performed on the first-party training sample business data based on the feature extractor to obtain first-party training sample features, and then the first-party training sample features can be spliced ​​into a first-party training sample feature vector or a first-party training sample feature matrix, and the first-party training sample feature vector or the first-party training sample feature matrix is ​​fully connected through the fully connected layer, and then the output of the fully connected layer is activated through the activation function preset by the activation layer to obtain the business label training prediction result.

[0062] Furthermore, before the step of obtaining the first-party training sample business data and the first sample business label of the aligned sample, and obtaining the business label training prediction result obtained by performing business label prediction on the first-party training sample business data by the trained business label prediction model, it also includes:

[0063] Obtaining sample business data and a second sample business label for training a business label prediction model, iteratively optimizing the business label prediction model based on the sample business data and the second sample business label for training the business label prediction model, and obtaining a trained business label prediction model.

[0064] In this embodiment, it should be noted that since the true value of the residual prediction performed by the residual prediction model on the second device is determined by the business label prediction model on the first device, before iteratively optimizing the residual prediction model on the second device, it is necessary to complete the training of the business label prediction model to ensure the correctness of the true value used to train the residual prediction model.

[0065] Exemplarily, before iteratively optimizing the residual prediction model on the second device, the business label prediction model is first initialized; then, a batch of business label prediction model training sample business data and second sample business labels are obtained from the local sample data of the first device, and the business label prediction model training sample business data is input into the business label prediction model to perform business label prediction to obtain a business label training prediction result, and the business label prediction model loss is calculated according to the difference between the business label training prediction result and the second sample business label to determine whether the business label prediction model loss converges. If the business label prediction model loss converges, it is determined that the business label prediction model training is completed. If the business label prediction model loss does not converge, the business label prediction model is updated for a round based on the business label prediction model gradient calculated using the business label prediction model loss, and the step of obtaining the business label prediction model training sample business data and the second sample business label is returned to execute until the business label prediction model converges to obtain a trained business label prediction model.

[0066] Step S30, receiving the training residual prediction result sent by each of the second devices, wherein the training residual prediction result is obtained by each of the second devices performing residual prediction based on the second-party training sample service data of the aligned samples through the residual prediction model;

[0067] As an example, step S30 includes: after sample alignment, the second device can search for the second-party training sample business data corresponding to the alignment sample ID from the sample business data owned by the second device based on the alignment sample ID, wherein it should be noted that for each second device, part or all of the second-party training sample business data corresponding to the alignment sample ID can be found. After finding the second-party training sample business data, the second device inputs the second-party training sample business data into the residual prediction model deployed by the second party for residual learning, and calculates the training residual prediction result based on the forward propagation algorithm, which is the predicted value of the residual generated by the business label prediction model for business label prediction; and then sends the training residual prediction result to the first device. Therefore, the first device can receive the training residual prediction result sent by the second device, and it should be noted that the training residual prediction result is also one-to-one corresponding to the alignment sample ID, so the training residual prediction result can be one-to-one corresponding to the business label prediction residual determined on the first device based on the alignment sample ID.

[0068] Step S40, obtaining a weight parameter, performing weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and updating the weight parameter based on a difference between the federated training residual and each of the training residual prediction results;

[0069] In this embodiment, it should be noted that the weight parameter includes the device weight corresponding to each of the second devices. The higher the probability of the second device launching an attack, the higher the risk of model training, and the greater the impact on the accuracy of business label prediction, so the assigned weight should be smaller. The weight parameter can be determined in advance based on historical data, test results, risk assessment results of the second device, etc., and can also be adjusted during the model training process according to the training residual prediction results sent by each of the second devices. It can be determined specifically according to actual conditions, and this embodiment does not limit this.

[0070] In one implementable manner, the residual difference between the training residual prediction result sent by the second device and the service label prediction residual can be calculated, and the device weight corresponding to the second device is negatively correlated with the residual difference corresponding to the second device.

[0071] In an operative manner, the weight parameter is a weight vector, and the weight vector is formed by arranging the device weights corresponding to each of the second devices in sequence according to a preset order. Exemplarily, the vertical federated learning system includes N second devices,

[0072] Weight Parameters

[0073] An equal initial weight may be set for each second device, and the weight corresponding to each second device may be subsequently updated based on the difference between the federated training residual and each training residual prediction result.

[0074] As an example, step S40 includes: obtaining weight parameters, determining the device weights corresponding to each of the second devices based on the weight parameters, multiplying the training residual prediction results sent by each of the second devices by the corresponding device weights, and then performing aggregation such as averaging or summing to obtain a federal training residual; and updating the weight parameters based on the difference between the federal training residual and each of the training residual prediction results, so that the weights of each second device in the weight parameters are negatively correlated with the differences between the respective ones and the federal training residuals, that is, the more the training residual prediction results sent by the second device deviate from the federal training residual, the lower the updated weight, thereby reducing its impact on the federal training residual. In this way, during the iterative training process, the weight of the Byzantine attacker will gradually decrease and eventually tend to 0, thereby achieving the purpose of limiting malicious individuals as a whole.

[0075] Furthermore, the step of obtaining a weight parameter, performing weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and updating the weight parameter based on the difference between the federated training residual and each of the training residual prediction results includes:

[0076] Step A10, determining a residual prediction loss according to a difference between the federated training residual and the service label prediction residual;

[0077] Step A20, calculating the gradient of the weight parameter based on the residual prediction loss to obtain the weight gradient, and updating the weight parameter based on the weight gradient.

[0078] In this embodiment, it should be noted that the second device may launch an attack during the current model training process. In this case, the attack behavior of the second device cannot be predicted based on historical data, previous test results or experience, etc. By synchronously calculating the gradient of the weight parameters during the federated training process, the weight parameters are updated, and the attacks launched by the second device can be sensed at any time during the training process, and the weight parameters can be adjusted in time to effectively resist the attacks. For example, after receiving the training residual prediction results R1, R2, and R3 sent by the second devices P1, P2, and P3, it is detected that R1<R2<R3<business label prediction residual, then the device weight corresponding to P1 can be reduced by n1, the device weight corresponding to P2 can be reduced by n2, and the device weight corresponding to P3 can be reduced by n3, wherein n3<n2<n1. In this way, if P1 does launch a malicious attack, the device weight of P1 can be quickly reduced, thereby reducing the impact of the training residual prediction results sent by P1 on the federal training residual, thereby quickly and effectively reducing the impact of the malicious attack launched by P1 on the entire training process, ensuring the stability of the model when attacked, and improving the robustness of the model.

[0079] Exemplarily, steps A10-A20 include: calculating the residual prediction loss based on the difference between the federated training residual and the business label prediction residual, the residual prediction loss being the total loss of the residual prediction losses of all residual prediction models on the second device, and then calculating the gradient of the weight parameter based on the residual prediction loss to obtain the weight gradient, and updating the weight parameter based on the weight gradient to achieve synchronous training of the weight parameter.

[0080] In this embodiment, the attention mechanism is used to learn the weight parameters. Each second device is assigned a corresponding device weight. The device weights are iteratively updated together during the model training process. The more the training residual prediction result sent by the second device deviates from the federated training residual, the lower the updated weight is, thereby reducing its impact on the federated training residual. In this way, during the iterative training process, the weight of the Byzantine attacker will gradually decrease and eventually approach 0, so that malicious second devices can be effectively discovered and eliminated, and the attacks of Byzantine attackers can be resisted, thereby improving the robustness of vertical federated learning.

[0081] Step S50, based on the federal training residual and the business label prediction residual, determine the residual prediction model gradient corresponding to each of the second devices, and send each of the residual prediction model gradients to the corresponding second devices, so that each of the second devices can update its deployed residual prediction model based on the received residual prediction model gradients.

[0082] As an example, step S50 includes: calculating the total loss function of the residual prediction loss of the residual prediction models on all the second devices based on the difference between the federal training residual and the business label prediction residual. Then, the gradient of the federal training residual can be calculated based on the total loss function to determine the same residual prediction model gradient for each of the second devices; or the gradient of the training residual prediction result corresponding to each of the second devices can be calculated based on the total loss function to determine the different residual prediction model gradients corresponding to each of the second devices. Each of the residual prediction model gradients is sent to the corresponding second device, so that each of the second devices can use the gradient descent method to update its deployed residual prediction model based on the received residual prediction model gradient.

[0083] Furthermore, the step of determining the residual prediction model gradient corresponding to each of the second devices based on the federated training residual and the service label prediction residual includes:

[0084] Step B10, determining a residual prediction loss according to a difference between the federated training residual and the service label prediction residual;

[0085] Step B20, based on the residual prediction loss, respectively calculate the gradient of each of the training residual prediction results to obtain the residual prediction model gradient corresponding to each of the second devices.

[0086] As an example, steps B10-B20 include: calculating the residual prediction loss based on the difference between the federal training residual and the business label prediction residual, the residual prediction loss being the total loss of the residual prediction losses of the residual prediction models on all the second devices, and then calculating the gradient of the training residual prediction results corresponding to each of the second devices based on the residual prediction loss to obtain the residual prediction model gradient corresponding to each of the second devices.

[0087] In this way, one round of iterative update of the residual prediction model can be completed, and multiple rounds of iterative update of the residual prediction model can be performed until the preset federated training end condition is met. Among them, the preset federated training end condition can be that the residual prediction model in more than a preset number of second devices in all second devices participating in the federated learning converges, or reaches the preset maximum number of iterations of the federated learning, or reaches the preset maximum training time of the federated learning, etc., which can be determined according to actual conditions, and this embodiment does not limit this.

[0088] Furthermore, the step of obtaining a weight parameter, performing weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and updating the weight parameter based on the difference between the federated training residual and each of the training residual prediction results includes:

[0089] Step C10, judging whether the residual prediction model deployed on each of the second devices has converged according to the difference between each of the training residual prediction results and the service label prediction residual;

[0090] Step C20, when it is detected that the number of converged residual prediction models does not exceed the preset number threshold, it is determined that the federated learning training end condition is not met at present, and the step of obtaining the weight parameter is executed, and weighted aggregation is performed on each of the training residual prediction results according to the weight parameter to obtain the federated training residual;

[0091] After the step of sending each of the residual prediction model gradients to the corresponding second devices so that each of the second devices updates the residual prediction model deployed by itself based on the received residual prediction model gradients, the method further includes:

[0092] Step C30, returning to the step of obtaining the first-party training sample business data of the aligned sample, and obtaining the business label prediction residual generated by the trained business label prediction model performing business label prediction based on the first-party training sample business data.

[0093] In this embodiment, it should be noted that after each time the training residual prediction result sent by the second device is received, it can be determined whether the federal training end condition is currently met. If the federal training end condition is not currently met, it is necessary to further calculate the loss and gradient and iteratively optimize the residual prediction model. If the federal training end condition is currently met, it means that the residual prediction model has been trained.

[0094] The preset federated training end condition may be that the residual prediction models in more than a preset number of second devices among all the second devices participating in the federated learning converge, or that a preset maximum number of iterations of the federated learning is reached, or that a preset maximum training time of the federated learning is reached, etc., which may be determined based on actual conditions and is not limited in this embodiment.

[0095] As an example, the steps C10-C30 include: receiving the training residual prediction results sent by each second device, judging whether the residual prediction model deployed on each second device has converged according to the difference between each training residual prediction result corresponding to each second device and the business label prediction residual corresponding to each training residual prediction result, and counting the number of converged residual prediction models, and when it is detected that the number of converged residual prediction models exceeds a preset number threshold, judging that the federal training end condition is currently met, and when it is detected that the number of converged residual prediction models does not exceed the preset number threshold, judging that the federal training end condition is not currently met, executing the step of obtaining weight parameters, weightedly aggregating each training residual prediction result according to the weight parameters to obtain the federal training residual, and continuing to calculate the loss and gradient to continue to optimize the residual prediction model. Since the model training is not completed, after sending each of the residual prediction model gradients to the corresponding second devices so that each of the second devices can update its deployed residual prediction model based on the received residual prediction model gradients, it can return to execute the steps of obtaining the first-party training sample business data of the aligned samples, and obtaining the trained business label prediction model based on the first-party training sample business data to generate the business label prediction residual, and perform the next round of iterative updates until the federation training end conditions are met, thereby completing the training of the weight parameters and the residual prediction model.

[0096] In one practicable manner, referring to Figure 2 The vertical federated learning system includes a first device and N second devices, each of which is in communication connection with the first device. After sample alignment, each of the second devices can perform residual prediction based on the second training sample business data of the aligned aligned samples through the residual prediction model to obtain training residual prediction results R1, R2, R3, ..., R N , and the training residual prediction results R1, R2, R3, ..., R N The first device receives the training residual prediction results R1, R2, R3, ..., R N After that, we get the weight parameters [λ1,λ2,…,λ N ], calculate the federated training residual Based on the federated training residual and the service label prediction residual, determine the residual prediction model gradients G1, G2, G3, ..., G corresponding to each of the second devices. N , and the residual prediction model gradient G1, G2, G3, ..., G Nare respectively sent to the corresponding second devices, so that each of the second devices updates its own deployed residual prediction models M1, M2, M3, ..., M based on the received residual prediction model gradient N .

[0097] In this embodiment, the vertical federated prediction optimization method is applied to a first device in a vertical federated learning system, and a service label prediction model is deployed on the first device; the vertical federated learning system also includes multiple second devices, and a residual prediction model is deployed on each of the second devices. First, sample alignment is performed with the second device to determine the aligned sample, and the first-party training sample service data and the first sample service label of the aligned sample are obtained, and the service label training prediction result obtained by the trained service label prediction model based on the first-party training sample service data for service label prediction is obtained. According to the first sample service label and the service label training prediction result, the service label prediction residual is determined. The service label prediction residual determined in this way is the true value of the residual generated by the trained service label prediction model for model prediction, thereby achieving the purpose of determining the true value of the residual; and then receiving the training residual prediction result sent by each of the second devices, the training residual prediction result is obtained by each of the second devices through the residual prediction model based on the second-party training sample service data of the aligned sample for residual prediction. In this way, the second-party training sample service data owned by the second device can be used for residual prediction, thereby achieving the purpose of determining the predicted value of the residual; and then The weight parameter is obtained, and the weighted aggregation of each training residual prediction result is performed according to the weight parameter to obtain a federal training residual, and the weight parameter is updated based on the difference between the federal training residual and each training residual prediction result. In this way, by setting the weight and adjusting the weight based on the training residual prediction result returned by each second device, the influence of the training residual prediction result provided by each second device on the vertical federated training can be adjusted. The training residual prediction result provided by a single malicious individual will obviously deviate from the federal training residual. On the one hand, the final use of the federal training residual can limit the influence of malicious individuals on the training effect through the federation as a whole. On the other hand, the influence of malicious individuals on the whole can be further reduced by updating the weight coefficient. Then, based on the federal training residual and the business label prediction residual, the residual prediction model gradient corresponding to each second device is determined, and each residual prediction model gradient is sent to the corresponding second device, so that each second device can update its deployed residual prediction model based on the received residual prediction model gradient. In this way, the influence of the malicious second device on the residual prediction model gradient can be reduced by setting and updating the weight, thereby effectively reducing the influence of the malicious second device on the vertical federated learning model. Therefore, the technical defect that if a participant sends malicious intermediate results to the label holder during the vertical federated training process, it will interfere with the training of the vertical federated learning model, thereby improving the robustness of the model.Moreover, compared with the business label prediction model obtained only based on local data training, vertical federated learning can utilize a variety of sample business data distributed on multiple second devices to more accurately predict the residual between the predicted results of the business label prediction model and the actual results, thereby correcting the prediction results of the business label prediction model and improving the prediction accuracy of business labels.

[0098] Embodiment 2

[0099] Furthermore, the present application also provides a longitudinal federated prediction optimization method. In the second embodiment of the present application, the same or similar contents as those in the above embodiment can be referred to the above description and will not be described in detail later. On this basis, the longitudinal federated prediction optimization method is applied to a second device, and a residual prediction model is deployed on the second device; the longitudinal federated prediction optimization method includes the following steps:

[0100] Step D10, performing sample alignment with the first device to determine second-party training sample service data corresponding to the first-party training sample service data in the first device;

[0101] The executor of the method of this embodiment can be a vertical federated prediction optimization device, or a vertical federated prediction optimization terminal device or server. This embodiment takes a vertical federated prediction optimization device as an example. The vertical federated prediction optimization device can be integrated in terminal devices such as smart phones and computers with data processing functions.

[0102] In this embodiment, it should be noted that the vertical federated prediction optimization method provided in this embodiment is applied to any one of multiple second devices in the vertical federated learning system, and a residual prediction model is deployed on the second device.

[0103] As an example, step D10 includes: first initializing the residual prediction model, and then receiving the alignment sample ID sent by the first device, and based on the received alignment sample IDs, searching for the second-party training sample business data corresponding to the alignment sample ID from the sample business data owned by the first device, thereby achieving sample alignment with each of the second devices.

[0104] Step D20, inputting the second-party training sample business data into the residual prediction model to perform residual prediction to obtain a training residual prediction value;

[0105] As an example, the step D20 includes: inputting the second-party training sample business data into the residual prediction model deployed by the own party to perform residual learning, and calculating the training residual prediction value based on the forward propagation algorithm.

[0106] Step D30, sending the training residual prediction value to the first device, so that the first device determines the residual prediction model gradient based on the training residual prediction value;

[0107] As an example, step D30 includes: sending the training residual prediction value to the first device, so that the first device determines the residual prediction model gradient based on the training residual prediction value, and returning the residual prediction model gradient to the second device. Among them, the method for the first device to determine the residual prediction model gradient based on the training residual prediction value can refer to the specific content corresponding to the above steps S10-S40, which will not be repeated here, but it should be noted that the second device cannot determine the specific generation method of the residual prediction model gradient returned by the first device, so that the privacy protection of the first device can be achieved, and the attack of the malicious second device can be better resisted.

[0108] Step D40: receiving the residual prediction model gradient sent by the first device, and updating the residual prediction model based on the residual prediction model gradient.

[0109] As an example, the step D40 includes: receiving the residual prediction model gradient sent by the first device, and based on the received residual prediction model gradient, updating the residual prediction model deployed by the self-deployed party by using the gradient descent method.

[0110] In this embodiment, for the second device, the second-party training sample business data also belongs to the user's private information. By performing residual prediction through the residual prediction model, only the residual information can be sent to the first device to achieve decoupling of the residual information and the sample data. When the first device does not know the second-party training sample business data, it is difficult for the second device to infer the private data owned by the second device based solely on the residual information. Therefore, the privacy protection of the second device can be achieved.

[0111] Embodiment 3

[0112] Furthermore, the present application also provides a method for optimizing vertical federation prediction. In the third embodiment of the present application, the same or similar contents as those in the above embodiment can be referred to the above introduction, and will not be described in detail later. On this basis, the method for optimizing vertical federation prediction is applied to the third device, referring to Figure 3 , the longitudinal federated forecasting optimization method comprises the following steps:

[0113] Step E10, obtaining first-party sample service data to be predicted of the sample to be predicted, and obtaining a local service label prediction result obtained by performing model prediction on the service label prediction model based on the first-party sample service data to be predicted;

[0114] In this embodiment, it should be noted that the vertical federated prediction optimization method is applied to a third device, which can be the first device. After completing the model training of the business label prediction model and the residual prediction model, the business label prediction can be performed directly locally; the third device can also be other electronic devices other than the first device. After completing the model training of the business label prediction model and the residual prediction model using the first device, the trained business label prediction model is deployed on other electronic devices to perform business label prediction. The specific method can be determined according to the actual situation, and this embodiment does not limit this.

[0115] As an example, the step E10 includes: obtaining first-party sample business data to be predicted of the sample to be predicted, and then inputting the first-party sample business data to be predicted into a trained business label prediction model to perform business label prediction to obtain a local business label prediction result.

[0116] In one implementable manner, the business label prediction model may include a feature extractor, a fully connected layer and an activation layer. After obtaining the first-party sample business data to be predicted of the sample to be predicted, feature extraction can be performed on the first-party sample business data to be predicted based on the feature extractor to obtain first-party sample features, and then the first-party sample features can be spliced ​​into a first-party sample feature vector or a first-party sample feature matrix, and the first-party sample feature vector or the first-party sample feature matrix is ​​fully connected through the fully connected layer, and then the output of the fully connected layer is activated through the activation function preset by the activation layer to obtain a local business label prediction result.

[0117] Step E20, performing sample alignment with multiple second devices, so that the target second device containing the second party sample business data to be predicted corresponding to the sample to be predicted in each of the second devices performs prediction based on the second party sample business data to be predicted by the residual prediction model deployed by the party, and obtains a residual prediction result, wherein the residual prediction model is trained by the vertical federated prediction optimization method as described above;

[0118] As an example, the step E20 includes: the third device can send the sample ID to be predicted of the sample to be predicted to each of the second devices, so that each of the second devices can search for the second party sample business data to be predicted corresponding to the sample ID to be predicted from the sample business data owned by the second device based on the received sample ID to be predicted, thereby achieving sample alignment with each of the second devices. The second device that can perform sample alignment with the third device can be determined as the target second device. When the target second device finds the second party sample business data to be predicted, it inputs each of the second party sample business data to be predicted into the trained residual prediction model deployed by the second device for residual prediction, obtains the residual prediction result, and then sends each of the training residual prediction results to the third device. When the second device cannot find the second party sample business data to be predicted corresponding to the sample ID to be predicted, it may not need to perform the residual prediction task, or may send a null value to the third device, or send a prompt message of no aligned samples to the third device. Among them, the training method of the residual prediction model can refer to the specific content corresponding to the above steps S10-S40, which will not be repeated here.

[0119] Step E30, receiving the residual prediction results sent by each of the target second devices, obtaining trained weight parameters, and weightedly aggregating each of the residual prediction results according to the trained weight parameters to obtain a federated residual;

[0120] Step E40: Aggregate the local service label prediction result and the federation residual to obtain a vertical federation prediction result.

[0121] As an example, steps E30-E40 include: receiving the residual prediction results sent by each of the target second devices, obtaining the trained weight parameters, determining the device weights corresponding to each of the target second devices based on the weight parameters, multiplying the residual prediction results sent by each of the second devices with the corresponding device weights, and then performing average or summing aggregation to obtain a federal residual; and then aggregating the local business label prediction results with the federal residual to obtain a federal business label prediction result. Among them, the method of aggregating the local business label prediction results with the federal residual can be a summation operation, an average operation, a weighted summation operation, a weighted average operation, etc.; the weight parameters can be determined in advance based on historical data, test results, risk assessment results of the second device, etc., and can also be adjusted during the model training process based on the training residual prediction results sent by each of the second devices. It can be determined specifically according to actual conditions, and this embodiment is not limited to this.

[0122] In an practicable manner, the method for determining the weight parameter may refer to steps A10-A20, which will not be elaborated herein.

[0123] Since the model performance of the business label prediction model trained only on local sample business data is limited, the residual between the local business label prediction result and the true value will be large. In this embodiment, by using the residual prediction model distributed on different second devices, the prediction results of the business label prediction model can be corrected using the richer and more diverse sample business data owned by different second devices while ensuring the privacy of all parties. This can effectively reduce the residual between the local business label prediction result and the true value, thereby obtaining a federal business label prediction result that is closer to the true value and improving the accuracy of the business label prediction. Furthermore, by setting the weight of the training update, the influence of the residual prediction result provided by each second device on the business label prediction result can be adjusted. By reducing the weight corresponding to the malicious second device, the influence of the malicious second device on the vertical federated learning model can be effectively reduced, and the accuracy of the business label prediction result can be improved.

[0124] Embodiment 4

[0125] Furthermore, the present application embodiment also provides a vertical federation prediction optimization device, referring to Figure 4 The vertical federated prediction optimization device is applied to a first device in a vertical federated learning system, and a service label prediction model is deployed on the first device; the vertical federated learning system also includes a plurality of second devices, and a residual prediction model is deployed on each of the second devices; the vertical federated prediction optimization device includes:

[0126] An alignment module 10, configured to perform sample alignment with the second device and determine an alignment sample;

[0127] A first acquisition module 20 is used to acquire the first-party training sample business data of the alignment sample, and acquire the business label prediction residual generated by the trained business label prediction model based on the first-party training sample business data to perform business label prediction;

[0128] A receiving module 30 is used to receive the training residual prediction result sent by each second device, where the training residual prediction result is obtained by each second device performing residual prediction based on the second-party training sample service data of the aligned sample through the residual prediction model;

[0129] A first aggregation module 40 is used to obtain a weight parameter, perform weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and update the weight parameter based on a difference between the federated training residual and each of the training residual prediction results;

[0130] The first update module 50 is used to determine the residual prediction model gradient corresponding to each of the second devices based on the federal training residual and the business label prediction residual, and send each of the residual prediction model gradients to the corresponding second devices, so that each of the second devices can update its deployed residual prediction model based on the received residual prediction model gradients.

[0131] Furthermore, the first aggregation module 40 is used for:

[0132] Determining a residual prediction loss according to a difference between the federated training residual and the business label prediction residual;

[0133] The weight parameter is gradiented based on the residual prediction loss to obtain a weight gradient, and the weight parameter is updated based on the weight gradient.

[0134] Furthermore, the first acquisition module 20 is further configured to:

[0135] Obtaining a first sample business label, and obtaining a business label training prediction result obtained by performing business label prediction on the first-party training sample business data by a trained business label prediction model;

[0136] A service label prediction residual is determined according to the first sample service label and the service label training prediction result.

[0137] Furthermore, the first updating module 50 is further configured to:

[0138] Determining a residual prediction loss according to a difference between the federated training residual and the business label prediction residual;

[0139] Based on the residual prediction loss, the gradient of each of the training residual prediction results is calculated to obtain the residual prediction model gradient corresponding to each of the second devices.

[0140] Furthermore, the first aggregation module 40 is further configured to:

[0141] Determining whether the residual prediction model deployed on each of the second devices has converged according to the difference between each of the training residual prediction results and the service label prediction residual;

[0142] When it is detected that the number of converged residual prediction models does not exceed the preset number threshold, it is determined that the federated learning training end condition is not met at present, and the step of obtaining the weight parameter is executed, and weighted aggregation of each training residual prediction result is performed according to the weight parameter to obtain the federated training residual;

[0143] After the operation of sending each residual prediction model gradient to the corresponding second device so that each second device updates the residual prediction model deployed by itself based on the received residual prediction model gradient, the vertical federated prediction optimization apparatus further includes a return module, which is used to:

[0144] Return to the step of obtaining the first-party training sample business data of the aligned sample, and obtaining the business label prediction residual generated by the trained business label prediction model performing business label prediction based on the first-party training sample business data.

[0145] Further, before the operation of obtaining the first-party training sample business data and the first sample business label of the aligned sample, and obtaining the business label training prediction result obtained by performing business label prediction on the first-party training sample business data by the trained business label prediction model, the vertical federation prediction optimization device further includes a local training module, and the local training module is used to:

[0146] Obtaining sample business data and a second sample business label for training a business label prediction model, iteratively optimizing the business label prediction model based on the sample business data and the second sample business label for training the business label prediction model, and obtaining a trained business label prediction model.

[0147] The vertical federated prediction optimization device provided by the present invention adopts the vertical federated prediction optimization method in the above embodiment, and solves the technical problem of low robustness of vertical federated learning in the related art. Compared with the related art, the benefits of the vertical federated prediction optimization device provided by the embodiment of the present invention are the same as the benefits of the vertical federated prediction optimization method provided by the above embodiment, and the other technical features in the vertical federated prediction optimization device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0148] Embodiment 5

[0149] Furthermore, the embodiment of the present application also provides a longitudinal federation prediction optimization device, which is applied to a third device, and includes:

[0150] A second acquisition module is used to acquire the first-party sample business data to be predicted of the sample to be predicted, and to acquire the local business label prediction result obtained by the business label prediction model based on the first-party sample business data to be predicted;

[0151] A residual prediction module, configured to perform sample alignment with a plurality of second devices, so that a target second device including the second party's sample service data to be predicted corresponding to the sample to be predicted in each of the second devices performs prediction based on the second party's sample service data to be predicted through a residual prediction model deployed by the second device, to obtain a residual prediction result;

[0152] A second aggregation module is used to receive the residual prediction results sent by each of the target second devices, obtain the trained weight parameters, and perform weighted aggregation on each of the residual prediction results according to the trained weight parameters to obtain a federated residual, wherein the weight parameters are trained using the longitudinal federated prediction optimization method as described above;

[0153] The third aggregation module is used to aggregate the local service label prediction result and the federal residual to obtain a vertical federal prediction result.

[0154] Embodiment 6

[0155] Furthermore, an embodiment of the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the vertical federated prediction optimization method in the above embodiment.

[0156] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as Bluetooth headsets, mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0157] like Figure 5 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and arrays required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0158] Typically, the following systems can be connected to the I / O interface: input devices including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc.; and communication devices. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange arrays. Although the electronic device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0159] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0160] The electronic device provided by the present invention adopts the longitudinal federated prediction optimization method in the above embodiment to solve the technical problem of low robustness of longitudinal federated learning in the related art. Compared with the related art, the benefits of the electronic device provided by the embodiment of the present invention are the same as the benefits of the longitudinal federated prediction optimization method provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0161] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0162] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0163] Embodiment 7

[0164] Furthermore, this embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the vertical federated prediction optimization method in the above embodiment.

[0165] The computer-readable storage medium provided in the embodiment of the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0166] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0167] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: performs sample alignment with the second device to determine the aligned sample; obtains the first-party training sample business data of the aligned sample, and obtains the business label prediction residual generated by the trained business label prediction model based on the first-party training sample business data to predict the business label; receives the training residual prediction results sent by each second device, and the training residual prediction results are obtained by each second device performing residual prediction based on the second-party training sample business data of the aligned sample through the residual prediction model; obtains a weight parameter, performs weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federal training residual, and updates the weight parameter based on the difference between the federal training residual and each of the training residual prediction results; determines the residual prediction model gradient corresponding to each of the second devices based on the federal training residual and the business label prediction residual, and sends each of the residual prediction model gradients to the corresponding second device, so that each of the second devices can update its deployed residual prediction model based on the received residual prediction model gradient.

[0168] Alternatively, the computer-readable storage medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: obtains the first-party sample business data to be predicted of the sample to be predicted, and obtains the local business label prediction result obtained by the business label prediction model based on the first-party sample business data to be predicted; performs sample alignment with multiple second devices, so that the target second device containing the second-party sample business data corresponding to the sample to be predicted in each of the second devices performs prediction based on the second-party sample business data to be predicted through the residual prediction model deployed by the second device to obtain a residual prediction result; receives the residual prediction result sent by each of the target second devices, obtains the trained weight parameter, and performs weighted aggregation on each of the residual prediction results according to the trained weight parameter to obtain a federated residual, wherein the weight parameter is trained using the vertical federated prediction optimization method as described above;

[0169] The local service label prediction result and the federation residual are aggregated to obtain a vertical federation prediction result.

[0170] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0171] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0173] The computer-readable storage medium provided by the present invention stores computer-readable program instructions for executing the above-mentioned vertical federated prediction optimization method, which solves the technical problem of low robustness of vertical federated learning in the related art. Compared with the related art, the benefits of the computer-readable storage medium provided by the embodiment of the present invention are the same as the benefits of the vertical federated prediction optimization method provided by the above-mentioned embodiment, which will not be repeated here.

[0174] Embodiment 8

[0175] Furthermore, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned vertical federated prediction optimization method when executed by a processor.

[0176] The computer program product provided by this application solves the technical problem of low robustness of vertical federated learning in the related art. Compared with the related art, the benefits of the computer program product provided by the embodiment of the present invention are the same as the benefits of the vertical federated prediction optimization method provided by the above embodiment, which will not be repeated here.

[0177] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A vertical federated forecasting optimization method, characterized in that: The vertical federated prediction optimization method is applied to a first device in a vertical federated learning system, and a service label prediction model is deployed on the first device; the vertical federated learning system also includes multiple second devices, and a residual prediction model is deployed on each of the second devices; the vertical federated prediction optimization method includes the following steps: Performing sample alignment with the second device to determine an aligned sample; Obtaining the first-party training sample business data of the aligned sample, and obtaining the business label prediction residual generated by the trained business label prediction model performing business label prediction based on the first-party training sample business data; Receiving a training residual prediction result sent by each of the second devices, where the training residual prediction result is obtained by each of the second devices performing residual prediction based on the second-party training sample service data of the aligned sample through the residual prediction model; Obtaining a weight parameter, performing weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and updating the weight parameter based on a difference between the federated training residual and each of the training residual prediction results; Based on the federal training residual and the business label prediction residual, the residual prediction model gradient corresponding to each of the second devices is determined, and each of the residual prediction model gradients is sent to the corresponding second device, so that each of the second devices updates its deployed residual prediction model based on the received residual prediction model gradient.

2. The vertical federation prediction optimization method according to claim 1, characterized in that: The step of updating the weight parameter based on the difference between the federated training residual and each training residual prediction result comprises: Determining a residual prediction loss according to a difference between the federated training residual and the business label prediction residual; The weight parameter is gradiented based on the residual prediction loss to obtain a weight gradient, and the weight parameter is updated based on the weight gradient.

3. The vertical federation prediction optimization method according to claim 1, characterized in that: The step of obtaining a service label prediction residual generated by performing service label prediction based on the first-party training sample service data by the trained service label prediction model comprises: Obtaining a first sample business label, and obtaining a business label training prediction result obtained by performing business label prediction on the first-party training sample business data by a trained business label prediction model; A service label prediction residual is determined according to the first sample service label and the service label training prediction result.

4. The vertical federation prediction optimization method according to claim 1, characterized in that: The step of determining the residual prediction model gradient corresponding to each of the second devices based on the federated training residual and the service label prediction residual comprises: Determining a residual prediction loss according to a difference between the federated training residual and the business label prediction residual; Based on the residual prediction loss, the gradient of each of the training residual prediction results is calculated to obtain the residual prediction model gradient corresponding to each of the second devices.

5. The vertical federation forecasting optimization method according to claim 1, characterized in that: The steps of obtaining a weight parameter, performing weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and updating the weight parameter based on the difference between the federated training residual and each of the training residual prediction results include: Determining whether the residual prediction model deployed on each of the second devices has converged according to the difference between each of the training residual prediction results and the service label prediction residual; When it is detected that the number of converged residual prediction models does not exceed the preset number threshold, it is determined that the federated learning training end condition is not met at present, and the step of obtaining the weight parameter is executed, and weighted aggregation of each training residual prediction result is performed according to the weight parameter to obtain the federated training residual; After the step of sending each of the residual prediction model gradients to the corresponding second devices so that each of the second devices updates the residual prediction model deployed by itself based on the received residual prediction model gradients, the method further includes: Return to the step of obtaining the first-party training sample business data of the aligned sample, and obtaining the business label prediction residual generated by the trained business label prediction model performing business label prediction based on the first-party training sample business data.

6. The vertical federation forecasting optimization method according to claim 1, characterized in that: Before the step of obtaining the first-party training sample business data and the first sample business label of the aligned sample, and obtaining the business label training prediction result obtained by performing business label prediction based on the first-party training sample business data by the trained business label prediction model, the step further includes: Obtaining sample business data and a second sample business label for training a business label prediction model, iteratively optimizing the business label prediction model based on the sample business data and the second sample business label for training the business label prediction model, and obtaining a trained business label prediction model.

7. A vertical federated forecasting optimization method, characterized in that: The longitudinal federated prediction optimization method is applied to a third device, comprising the following steps: Obtaining first-party sample business data to be predicted of the sample to be predicted, and obtaining a local business label prediction result obtained by a business label prediction model based on the first-party sample business data to be predicted; Performing sample alignment with multiple second devices, so that the target second device including the second party sample service data to be predicted corresponding to the sample to be predicted in each of the second devices performs prediction based on the second party sample service data to be predicted by using the residual prediction model deployed by the second party to obtain a residual prediction result; Receiving the residual prediction results sent by each of the target second devices, obtaining trained weight parameters, and weightedly aggregating each of the residual prediction results according to the trained weight parameters to obtain a federated residual, wherein the weight parameters are trained using the longitudinal federated prediction optimization method according to any one of claims 1 to 6; The local service label prediction result and the federation residual are aggregated to obtain a vertical federation prediction result.

8. A vertical federation forecasting optimization device, characterized in that: The vertical federated prediction optimization device is applied to a first device in a vertical federated learning system, and a service label prediction model is deployed on the first device; the vertical federated learning system also includes a plurality of second devices, and a residual prediction model is deployed on each of the second devices; the vertical federated prediction optimization device includes: an alignment module, configured to perform sample alignment with the second device and determine an alignment sample; A first acquisition module is used to acquire the first-party training sample business data of the aligned sample, and acquire the business label prediction residual generated by the trained business label prediction model performing business label prediction based on the first-party training sample business data; A receiving module, configured to receive a training residual prediction result sent by each of the second devices, wherein the training residual prediction result is obtained by each of the second devices performing residual prediction based on the second-party training sample service data of the aligned sample through the residual prediction model; A first aggregation module is used to obtain a weight parameter, perform weighted aggregation on each of the training residual prediction results according to the weight parameter to obtain a federated training residual, and update the weight parameter based on a difference between the federated training residual and each of the training residual prediction results; The first update module is used to determine the residual prediction model gradient corresponding to each of the second devices based on the federal training residual and the business label prediction residual, and send each of the residual prediction model gradients to the corresponding second devices, so that each of the second devices updates its deployed residual prediction model based on the received residual prediction model gradients.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors to enable at least one of the processors to perform the steps of the longitudinal federated forecasting optimization method described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which is stored a program for implementing the vertical federated forecasting optimization method. The program for implementing the vertical federated forecasting optimization method is executed by a processor to implement the steps of the vertical federated forecasting optimization method as described in any one of claims 1 to 7.

11. A product, the product being a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the longitudinal federated forecasting optimization method as described in any one of claims 1 to 7 are implemented.

Images