Longitudinal federal learning method and system, electronic equipment and readable storage medium

Abstract

The invention discloses a longitudinal federal learning method and system, electronic equipment and a readable storage medium, and relates to the technical field of data privacy protection, and the method comprises the steps: calculating feature correlation according to sample data, and selecting an optimal subset; dividing the samples into different buckets by adopting a weighted quantile sketch; evaluating the features of each sample based on a dynamic privacy budget allocation strategy to obtain feature importance, allocating a privacy budget to each feature, and desensitizing the samples; the first participant receives the feature importance information of all the participants, finds the optimal segmentation point of the features and updates the prediction model to obtain a trained prediction model; and performing prediction analysis on the new data sample through the trained prediction model. According to the method, the high performance, the non-uniform bucket dividing strategy, the feature selection and the probability order-preserving algorithm of the boosting tree are utilized, so that more efficient data utilization and better model performance are realized, and meanwhile, the operation efficiency is improved and the privacy is protected on the premise that the precision change is not large.

Description

Technical Field

[0001] The present invention relates to the field of data privacy protection technology, and more specifically to a vertical federated learning method, system, electronic device and readable storage medium. Background Art

[0002] With the rapid development of information technology, the speed of data generation and aggregation is accelerating, and machine learning has become an important tool for processing massive data. However, data privacy and security issues have gradually attracted widespread attention, especially in application scenarios involving personal information and sensitive data. In order to meet these challenges, Federated Learning has emerged as a distributed machine learning method that allows multiple participants to jointly train models without sharing original data.

[0003] Although federated learning provides new ideas for data privacy protection, existing vertical federated learning (VFL) solutions still have some problems. First, existing methods usually require a large number of encryption operations, which leads to high computational and communication overheads. Second, although solutions based on local differential privacy (LDP) can protect user privacy, they often lead to a decrease in model accuracy. In addition, many existing privacy budget allocation strategies are uniform, which ignores the impact of dynamic changes in local gradients during model training, further reducing the utility of the model.

[0004] As an effective supervised learning method, the boosted tree algorithm is widely used in prediction tasks of heterogeneous features. The algorithm optimizes the prediction performance of the model by iteratively training a series of decision trees. These trees are trained on the basis of the previous tree to reduce the bias, making the boosted tree both efficient and interpretable. Studies have shown that the prediction performance of boosted trees when dealing with heterogeneous features is generally better than that of state-of-the-art deep learning algorithms. This is because boosted trees can directly handle multiple types of features without complex preprocessing. However, most existing boosted tree algorithms are designed and implemented in a centralized environment, requiring direct access to the entire training dataset. This limits its application in the context of increasing attention to privacy protection.

[0005] In order to solve the problem of data privacy protection, researchers have proposed a variety of solutions based on cryptography and LDP. For example, the SecureBoost system proposed by Cheng et al. first performs entity alignment under a privacy-preserving protocol, and then constructs a boosted tree through an encryption strategy. Although these solutions have achieved certain results in privacy protection, due to the complexity of encryption operations and homomorphic operations, the communication overhead and computational overhead are still high. In addition, many existing solutions fail to effectively balance the relationship between privacy protection and model performance. For example, the FederBoost framework proposed by Tian et al. can resist multiple attacks, but the introduction of randomness leads to a loss of training model accuracy.

[0006] Therefore, how to provide a vertical federated learning method, system, electronic device and readable storage medium to achieve more efficient privacy protection while improving the accuracy and robustness of the model is an urgent problem that technicians in this field need to solve. Summary of the invention

[0007] In view of this, the present invention provides a vertical federated learning method, system, electronic device and readable storage medium. By combining the boosting tree algorithm and adaptive differential privacy, the problems of high communication cost, insufficient privacy protection and low model accuracy existing in the existing vertical federated learning scheme are solved, and efficient model training is achieved while protecting data privacy, which provides important reference value for future privacy protection machine learning research.

[0008] In order to achieve the above object, the present invention adopts the following technical solution: a vertical federated learning method, comprising:

[0009] Get new data samples;

[0010] Constructing a prediction model and training the prediction model;

[0011] The training process of the prediction model includes:

[0012] The first participant preprocesses the feature values ​​of the sample data according to the sample data contained therein, calculates feature correlation, performs feature selection, and selects the optimal subset;

[0013] Based on the optimal subset, the weighted quantile sketch is used to divide the samples into different buckets;

[0014] Based on the dynamic privacy budget allocation strategy, the features of each sample are evaluated, the feature importance is obtained according to the evaluation results, a privacy budget is allocated to each feature, and the order-preserving desensitization algorithm is improved based on the dynamic privacy budget allocation strategy to desensitize the samples;

[0015] The first participant receives feature importance information from all participants, finds the best segmentation point of the feature and updates the prediction model to obtain a trained prediction model;

[0016] The trained prediction model is used to perform prediction analysis on new data samples.

[0017] Preferably, the all participants include the first participant and the second participant;

[0018] The first participant holds complete training sample labels;

[0019] All participants hold some specific features of the sample data, and the sample identifiers overlap across all participants.

[0020] Preferably, during feature selection, a boosted tree algorithm is used to calculate the importance of features and sort them;

[0021] Calculate the correlation between features and target variables and between features;

[0022] For each feature combination, its CFS score is calculated, and the optimal subset is selected based on the CFS score.

[0023] Preferably, a weighted quantile outline is used to divide the samples into different buckets, including:

[0024] For each feature in the optimal subset, sort the feature values ​​of the samples;

[0025] Convert continuous numerical features into ordered categorical features;

[0026] According to the objective function, the contribution of the sample to the objective function value is obtained by the second-order derivative h i Determine, dataset D k ={(x 1k ,h1),(x 2k ,h2),…,(x nk ,h n )} represents the set of the kth eigenvalue and second-order derivative of each sample;

[0027] Define the function r k (z) represents the percentage of the weight of samples with eigenvalues ​​less than z to the total weight of all samples;

[0028] For the tangent point {s k1 ,s k2 ,…,s kl}, set the sampling frequency a to standardize the points to be divided so that the r of two connected tangent points k The absolute value of the difference between the values ​​is less than a, which can be expressed as:

[0029]

[0030] Preferably, the features of each sample are evaluated based on a dynamic privacy budget allocation strategy, the feature importance is obtained according to the evaluation result, and a privacy budget is allocated to each feature, including:

[0031] Set feature k = 1, 2, ..., K, and the feature importance measurement formula is as follows:

[0032]

[0033] Where T represents the total number of trees, N(t) is the number of nodes in the t-th tree, and I(β(t,i)=k) is an indicator function. When node i is split by feature k on tree t, the function value is 1; otherwise, it is 0. is the square of the gradient of feature γ at node i, H γ(t,i) is the second-order derivative of feature γ at node i; is the square of the gradient of feature γ on the left subtree of node i, is the square of the gradient of feature γ on the right subtree of node i, H γ(t,i,L) is the second-order derivative of feature γ on the left subtree of node i, H γ(t,i,R) is the second-order derivative of feature γ on the right subtree of node i;

[0034] in, It represents the sum of the gains of all nodes i in all trees t when splitting on feature k, and the regularization term λ is added to control the model complexity;

[0035] It indicates the number of times feature k is used as a segmentation feature in all nodes on all trees, and counts the frequency of feature k being used as a segmentation condition.

[0036] Preferably, the first participant normalizes the feature importance to obtain v norm(k) ;

[0037]

[0038] The formula for the privacy budget allocation strategy based on the normalized results is as follows:

[0039]

[0040] Among them, 0<α<1 and 0<β<1, α represents the growth rate of the controlled privacy budget, β represents the highest allocated privacy budget when the normalized feature influence is 0%, ε represents the total privacy budget, and ε′ represents the privacy budget allocated to the sample.

[0041] Preferably, a standardized importance score is obtained by calculating the sum of information gains brought by the feature when it is used as a split node in all decision trees, and the higher the score, the greater the importance of feature k;

[0042] If the denominator of the feature importance metric is 0, then feature k has never been used for segmentation, and the importance of feature k v (k) Also 0.

[0043] A vertical federated learning system, comprising:

[0044] A data acquisition module, used to acquire new data samples;

[0045] The model building and training module is used to build a prediction model and train the prediction model, wherein the training process of the prediction model includes:

[0046] The first participant preprocesses the feature values ​​of the sample data according to the sample data contained therein, calculates feature correlation, performs feature selection, and selects the optimal subset;

[0047] Based on the optimal subset, the weighted quantile sketch is used to divide the samples into different buckets;

[0048] Based on the dynamic privacy budget allocation strategy, the features of each sample are evaluated, the feature importance is obtained according to the evaluation results, a privacy budget is allocated to each feature, and the order-preserving desensitization algorithm is improved based on the dynamic privacy budget allocation strategy to desensitize the samples;

[0049] The first participant collects feature information of all participants, finds the best segmentation point of the features and updates the prediction model to obtain a trained prediction model;

[0050] The prediction module is used to perform prediction analysis on new data samples through the trained prediction model.

[0051] An electronic device comprising: a memory and a processor;

[0052] The memory is configured to: store computer program instructions;

[0053] The processor is configured to: execute the computer program instructions so that the electronic device implements the above-mentioned vertical federated learning method.

[0054] A readable storage medium, characterized by comprising: computer program instructions;

[0055] The computer program instructions are executed by at least one processor of an electronic device, so that the electronic device implements the above-mentioned vertical federated learning method.

[0056] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a longitudinal federated learning method, system, electronic device and readable storage medium, including: obtaining new data samples; constructing a prediction model and training the prediction model; the training process of the prediction model includes: the first participant preprocesses the characteristic values ​​of the sample data according to the sample data it contains, calculates the feature correlation, performs feature selection, and selects the optimal subset; based on the optimal subset, the weighted quantile outline is used to divide the sample into different buckets; based on the dynamic privacy budget allocation strategy, the characteristics of each sample are evaluated, the feature importance is obtained according to the evaluation result, a privacy budget is allocated to each feature, and the order-preserving desensitization algorithm is improved based on the dynamic privacy budget allocation strategy to desensitize the sample; the first participant receives the feature importance information of all participants, finds the best segmentation point of the feature and updates the prediction model to obtain a trained prediction model; and the new data sample is predicted and analyzed by the trained prediction model. The present invention utilizes the high performance of the boosting tree, the uneven bucketing strategy, the feature selection and the probability order-preserving algorithm to achieve more efficient data utilization and better model performance, while improving the operating efficiency and protecting privacy while maintaining little change in accuracy. By leveraging the connection between data features and global model outputs, we adjust the allocation of the privacy budget to improve the accuracy and robustness of the model while achieving more efficient privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0058] Figure 1 A schematic diagram of a framework of a vertical federated learning mechanism provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In the existing field of federated learning, although some progress has been made, especially in VFL, existing technologies still face several urgent problems. First, existing VFL technologies generally face high computational and communication overheads. Many VFL schemes require a large number of encryption operations to ensure data privacy, which leads to significant computational and communication costs. For example, although protocols based on homomorphic encryption and multi-party computing can protect privacy in theory, they are often difficult to implement efficiently in practical applications due to their complexity and high computational burden. Secondly, although privacy protection mechanisms such as local differential privacy can protect privacy to a certain extent, they often lead to a decline in model performance. Especially when the privacy budget is evenly distributed, the information loss of certain features in model training may reduce the accuracy of the final model, which may not meet business needs in practical applications. Furthermore, most existing methods fail to effectively adjust the privacy budget allocation dynamically. The even distribution of the privacy budget ignores the dynamic changes of local gradients and cannot balance the relationship between model privacy and utility. This affects the performance of the model in some key training steps, and the privacy protection effect is difficult to achieve the best state. In addition, existing technologies are also vulnerable to reasoning attacks. Although some solutions are designed for privacy protection, in actual applications, attackers can still infer the original data by analyzing the output of the model and the order of features. This privacy leakage risk is particularly prominent in application scenarios involving sensitive data, posing a challenge to data security.

[0061] In summary, the existing VFL technology still has many problems in terms of computing and communication overhead, model performance, privacy budget allocation, inference attack risk, etc. These problems limit the wide application and practical effect of VFL technology, so new solutions are urgently needed to meet these challenges and achieve efficient and secure vertical federated learning.

[0062] The embodiments of the present invention disclose a method, system, electronic device and readable storage medium for vertical federated learning; federated learning can be divided into horizontal federated learning (Horizontal Federated Learning, HFL), vertical federated learning (Vertical Federated Learning, VFL) and migration federated learning. Horizontal federated learning is mainly used in scenarios with the same features but different samples, such as customer data between different banks. In this case, each participant can train the model locally and share the model updates with other participants to achieve shared learning of the model. Vertical federated learning is suitable for scenarios with the same samples but different features, such as data left by a user on different platforms. In this case, each participant can also train the model locally and share the model updates.

[0063] The embodiment of the present invention discloses a vertical federated learning method, including:

[0064] Get new data samples;

[0065] Constructing a prediction model and training the prediction model;

[0066] The training process of the prediction model includes:

[0067] The first participant preprocesses the feature values ​​of the sample data according to the sample data contained therein, calculates feature correlation, performs feature selection, and selects the optimal subset;

[0068] Based on the optimal subset, the weighted quantile sketch is used to divide the samples into different buckets;

[0069] Based on the dynamic privacy budget allocation strategy, the features of each sample are evaluated, the feature importance is obtained according to the evaluation results, a privacy budget is allocated to each feature, and the order-preserving desensitization algorithm is improved based on the dynamic privacy budget allocation strategy to desensitize the samples;

[0070] The first participant receives feature importance information from all participants, finds the best segmentation point of the feature and updates the prediction model to obtain a trained prediction model;

[0071] The trained prediction model is used to perform prediction analysis on new data samples.

[0072] The embodiment of the present invention combines theoretical analysis with experimental verification methods to conduct research, deeply analyzes the problems faced in the task, and divides the vertical federated learning mechanism based on boosted tree and adaptive differential privacy into two research topics.

[0073] (1) The embodiment of the present invention proposes a differential privacy vertical federated learning framework based on uneven bucketing and feature selection method of boosting tree. The core of the framework is to use the high performance of boosting tree, uneven bucketing strategy, feature selection and probabilistic order-preserving algorithm to achieve more efficient data utilization and better model performance, while improving operation efficiency and protecting privacy under the premise of keeping the accuracy unchanged. First, the framework uses boosting tree as the basic model and significantly improves the prediction performance of the model by combining multiple weak learners. Secondly, an uneven bucketing strategy is introduced. Different from traditional uniform bucketing, uneven bucketing can better capture the distribution characteristics of data, thereby improving the performance of the model. In addition, feature selection is performed based on the relevance and importance of features, and the optimal subset is selected to further improve the efficiency and effect of the model. This feature selection method can reduce data redundancy, enhance the generalization ability of the model, and reduce computational complexity. Finally, the training data is desensitized by the probabilistic order-preserving algorithm. Different from the existing solution based on local differential privacy, the probabilistic order-preserving algorithm does not retain the relative order of all value pairs, but retains the relative order of some value pairs. This approach works by adding carefully selected random noise to sensitive data. The introduced randomness provides provable privacy protection against various attacks based on auxiliary information.

[0074] (2) The embodiment of the present invention proposes a strategy for dynamically allocating a privacy budget based on feature importance. The core idea of ​​this strategy is to use the connection between data features and global model outputs to adjust the allocation of privacy budgets, thereby achieving more efficient privacy protection while improving the accuracy and robustness of the model. Specifically, the importance of each feature is measured by observing its contribution to the model output. This method can identify the impact of features on model performance, thereby making more effective use of limited privacy budgets. Subsequently, the privacy budget is dynamically allocated according to the importance of the features.

[0075] Specifically, the all participants include the first participant and the second participant;

[0076] The first participant holds complete training sample labels;

[0077] All participants hold some specific features of the sample data, and the sample identifiers overlap across all participants.

[0078] In a specific embodiment of the present invention, Figure 1 As shown, the embodiment of the present invention considers a scenario where there are k participants P1, ..., P k , holding m data sets X1,…,X m, they jointly train a model. The training data set is vertically split and distributed on the devices of different users. Each user only holds certain specific features of the sample, and the identifier (ID) of the sample is overlapping among all users. Only one user holds the label information necessary to perform the supervised learning process. Since labels are crucial to supervised learning, the user holding the label is usually the central node responsible for aggregating information and updating the model. In order to simplify the description, the embodiment of the present invention divides users into two main groups: Party1--Partyn-1 (Party A) and Party n (Party B). Party B includes users who hold complete training sample labels and may also hold part of the sample features at the same time. In the model training stage, Party B plays the role of a central server, responsible for exchanging information with Party A and guiding the training of the model. The trained model is only stored in Party B, and Party B is responsible for using the model to predict new data samples. On the other hand, Party A consists of users who only hold part of the features of the sample. In the model training process, Party A plays the role of a client, participating in the training by exchanging necessary information with Party B, so that the entire training process can proceed smoothly.

[0079] The present invention will describe in detail the process of training decision trees and explain how to use these trained decision trees for prediction. Figure 1 As shown in Table 1, the algorithm of the vertical federated learning mechanism based on boosted tree and adaptive differential privacy (DPTB-VFL) is shown in Table 1.

[0080] Table 1 DPTB-VFL algorithm

[0081]

[0082]

[0083] Specifically, the eigenvalue preprocessing process includes:

[0084] In the data preprocessing stage, the embodiments of the present invention focus on processing numerical and categorical features with natural ordering properties. For categorical features with natural order, they are first mapped into continuous numerical values ​​to form an ordered numerical sequence, and then the mapped continuous numerical values ​​are subjected to privacy protection processing. In order to further simplify the processing process, it is assumed that all features are numerical, regardless of whether they are continuous or discrete. The advantage of this assumption is that specific numerical values ​​do not directly determine the structure of the decision tree, so that features with different distance metrics can be uniformly mapped into a discrete data domain. This unified mapping not only simplifies the data processing process, but also lays the foundation for the subsequent application of distance-based privacy protection algorithms. Through the mapping function, discrete numerical values ​​are remapped to a unified discrete value domain middle, Represents a set of numerical values ​​for a feature. Partyk maps each value as follows

[0085] The lower and upper are The lower and upper bounds of L and R are The lower and upper bounds of . The larger the domain, the more the original relative order of the mapped values ​​is preserved.

[0086] Specifically, during feature selection, the boosted tree algorithm is used to calculate the importance of features and rank them;

[0087] Calculate the correlation between features and target variables and between features;

[0088] For each feature combination, its CFS score is calculated, and the optimal subset is selected based on the CFS score.

[0089] In an embodiment of the present invention, in the feature selection stage, a boosting tree algorithm is used to calculate the importance of features and sort them. Considering that in many cases, highly correlated features may lead to overfitting of the model, especially in small sample data sets, unnecessary model complexity is reduced by identifying and processing features with low importance but high correlation, thereby improving the generalization ability of the model. Feature correlation analysis (Correlation-based Feature Selection, CFS) is a method for selecting features, the goal of which is to find a subset of features that are highly correlated with the output variable but have low correlation with each other. This process is achieved by calculating the correlation between the feature and the target variable, as well as the correlation between the features.

[0090] The calculation formula of the CFS score is:

[0091]

[0092] Where k is the number of features in the feature subset, is the average correlation coefficient between features and class labels, r ff is the average correlation coefficient between features, s i Indicates the correlation between each feature and the target variable.

[0093] The correlation between all features and the target variable and the correlation between features are calculated by the following formula. The correlation between two feature vectors i and j is calculated as follows:

[0094]

[0095] Among them, x i,mRepresents the value of feature i in the mth sample; represents the mean of feature i, that is, the average value of all samples on feature i; x j,m Represents the value of feature j in the mth sample; It represents the mean of feature j, that is, the average value of all samples on feature j.

[0096] Then, CFS is applied to select feature subsets. For each possible feature combination, its CFS score is calculated, and the feature subset with the highest CFS score is selected as the optimal feature combination. Through feature correlation analysis, we can better understand the interaction between the features in the data set and their impact on the model prediction performance. This method not only improves the accuracy of the model, but also helps to improve the interpretability of the model.

[0097] In a specific embodiment of the present invention, a weighted quantile sketch is used for bucketing. For each feature, the feature values ​​of the samples are first sorted, and then the samples are divided and assigned to different buckets. This method ensures that Party B can only recognize the order of the buckets, but cannot understand the specific sorting of the samples in the buckets. The bucketing operation converts continuous numerical features into ordered categorical features, effectively reducing the sensitivity of the model to specific values, thereby reducing the possibility of overfitting. In addition, the bucketing operation makes the model more stable to small changes in the input data by segmenting the data. Even if the values ​​change, as long as they are still in the same bucket, the impact on the model remains unchanged.

[0098] Specifically, a weighted quantile outline is used to divide samples into different buckets, including:

[0099] For each feature in the optimal subset, sort the feature values ​​of the samples;

[0100] Convert continuous numerical features into ordered categorical features;

[0101] The objective function is expressed as:

[0102]

[0103] Among them, g i represents the gradient of sample i, which indicates the contribution of sample i to the loss function under the prediction of the current decision tree model; f t (x i ) indicates that in the decision tree t, for the sample x i The predicted value of h i is the second-order derivative of sample i; represents the square of the predicted value; Ω(f t ) is a regularization term to prevent overfitting.

[0104] According to the objective function, the contribution of the sample to the objective function value is obtained by the second-order derivative h i Decide;

[0105]

[0106] Where c is a constant.

[0107] Dataset D k ={(x 1k ,h1),(x 2k ,h2),…,(x nk ,h n )} represents the set of the kth eigenvalue and second-order derivative of each sample.

[0108] Define the function r k (z) represents the percentage of the weight of samples with eigenvalues ​​less than z to the total weight of all samples; it can also be used to indicate the importance of a feature, r k (z) is expressed as:

[0109]

[0110] For the tangent point {s k1 ,s k2 ,…,s kl}, set the sampling frequency a to standardize the points to be divided so that the r of two connected tangent points k The absolute value of the difference between the values ​​is less than a, which can be expressed as:

[0111]

[0112] In the vertical federated learning scenario, different organizations hold different features of the same batch of samples, and the contributions of these features to the model training process are not the same. As an efficient machine learning algorithm, the evaluation of feature importance of boosted tree can accurately indicate the degree of influence of each feature on the model performance. By analyzing the importance of features, more privacy budget is allocated to features that have a greater impact on the model, while reducing the privacy protection of features with less impact, maximizing the value of data while ensuring data privacy.

[0113] In a specific embodiment of the present invention, first, the feature set of each organization is evaluated separately to accurately quantify the impact of the entire feature set on the model. During model training, the nodes of each tree are analyzed in detail and Gain is used as the evaluation indicator of feature importance. After the evaluation is completed, the client sends the calculated feature importance information to the server, which intuitively shows the contribution of each feature to the model performance. The server then integrates this data and effectively identifies the features that have the greatest impact on the model by sorting and normalizing the feature importance. Therefore, the first step is to evaluate the feature importance of each organization. Then, a privacy budget allocation strategy is designed that allows the accuracy of the model to be maintained as much as possible while protecting privacy.

[0114] Specifically, the features of each sample are evaluated based on the dynamic privacy budget allocation strategy, and the feature importance is obtained according to the evaluation results. A privacy budget is allocated to each feature, including:

[0115] Set feature k = 1, 2, ..., K, and the feature importance measurement formula is as follows:

[0116]

[0117] Where T represents the total number of trees, N(t) is the number of nodes in the t-th tree, and I(β(t,i)=k) is an indicator function. When node i is split by feature k on tree t, the function value is 1; otherwise, it is 0. is the square of the gradient of feature γ at node i, H γ(t,i) is the second-order derivative of feature γ at node i; is the square of the gradient of feature γ on the left subtree of node i, is the square of the gradient of feature γ on the right subtree of node i, H γ(t,i,L) is the second-order derivative of feature γ on the left subtree of node i, H γ(t,i,R) is the second-order derivative of feature γ on the right subtree of node i;

[0118] By calculating the sum of the information gain brought by the feature when it is used as a split node in all decision trees, a standardized importance score is obtained. The higher the score, the greater the importance of feature k.

[0119] Specifically, It represents the sum of the gains of all nodes i in all trees t when splitting on feature k, and the regularization term λ is added to control the model complexity;

[0120] It indicates the number of times feature k is used as a segmentation feature in all nodes on all trees, and counts the frequency of feature k being used as a segmentation condition.

[0121] If the denominator of the feature importance metric is 0, then feature k has never been used for segmentation, and the importance of feature k v (k) Also 0.

[0122] Specifically, the first participant normalizes the importance of all features to obtain v norm(k) ;

[0123]

[0124] By normalization, the importance of each feature v norm(k) It becomes a value between 0 and 1, which makes the comparison of feature importance more intuitive and fair, especially when comparing different datasets or models.

[0125] After constructing a normalized measure of feature importance, it is necessary to consider its coordination with the allocation of privacy budget. An intuitive approach is: if a feature has low importance, it indicates that its contribution to the model is limited, so the level of privacy protection it requires can also be relatively low. Relatively speaking, for those features with high importance, their impact on the model is more significant, thus requiring stronger privacy protection measures. Inspired by feature inference attacks, as the proportion of features owned by an organization increases, the likelihood of an attacker revealing the feature increases linearly. The feature importance is calculated using the feature importance measurement formula, and the feature importance is normalized to obtain v norm(k) ; The formula for the privacy budget allocation strategy based on the normalization results is as follows:

[0126]

[0127] Among them, 0<α<1 and 0<β<1, α represents the growth rate of the privacy budget, β represents the highest privacy budget allocated when the normalized feature influence is 0%, ε represents the total privacy budget, and ε′ represents the privacy budget allocated to the sample. The privacy budget allocation strategy algorithm is shown in Table 2.

[0128] Table 2 Privacy budget allocation strategy algorithm

[0129]

[0130] Specifically, the probabilistic order-preserving desensitization process includes:

[0131] When dealing with data privacy issues, retaining the order information of all values ​​may lead to inference attacks, and it is necessary to introduce randomness into the order of desensitized values. Although the use of local differential privacy can resist privacy attacks, a large amount of order information will be lost, resulting in poor performance of the training model. In contrast, distance-based LDP (dLDP) provides different indistinguishability for value pairs with different distances, making value pairs with smaller distances more difficult to distinguish. The embodiment of the present invention is based on the Global-Map order-preserving desensitization algorithm, and according to the dynamic allocation of privacy budget strategy, the algorithm is improved to obtain Global-Maps, and the output probability is assigned to each value according to the scoring function. The scoring function can be defined as the distance between the value and the sensitive value, so that the value will be desensitized to a closer value with a higher probability. The details of Global-maps are shown in Table 3.

[0132] Table 3 Global-maps algorithm

[0133]

[0134]

[0135] Then, Party B collects all feature information from Party A, finds the best segmentation point of the feature and updates the tree model.

[0136] After the above training steps, Party B finally builds a complete decision tree model, which will be used for predictive analysis of new data samples.

[0137] The embodiments of the present invention have the following beneficial effects:

[0138] (1) The embodiments of the present invention innovatively integrate boosting tree, differential privacy and vertical federated learning, and propose a differential privacy vertical federated learning framework based on boosting tree, aiming to solve the problems of high communication cost, insufficient privacy protection and low model accuracy in existing vertical federated learning schemes. Existing methods usually require a large number of encryption operations and frequent data exchange, which leads to a significant increase in communication overhead, and often fail to maintain high accuracy of the model while protecting user privacy. In addition, many schemes based on local differential privacy evenly distribute the privacy budget and fail to effectively cope with the dynamic changes of local gradients, further affecting the performance of the model. The embodiments of the present invention utilize the high efficiency of the boosting tree algorithm, significantly improve the prediction ability of the model by integrating multiple weak learners, introduce uneven bucketing and feature selection strategies, optimize data utilization and model performance, and ensure that communication costs are reduced while improving privacy protection.

[0139] (2) The embodiment of the present invention innovatively combines feature importance with privacy budget allocation, and proposes a privacy budget allocation strategy based on feature importance, aiming to solve the imbalance between privacy protection and model performance in existing vertical federated learning. Traditional methods usually evenly allocate the privacy budget, ignoring the different contributions of features to the model output, resulting in poor privacy protection and reduced model accuracy. By dynamically adjusting the privacy budget allocation, the method in this paper can flexibly adjust according to the importance of each feature, thereby improving model performance while ensuring privacy protection. In addition, the order-preserving desensitization algorithm is used to process the training data, which retains more relative order information while providing strong privacy protection, thereby further improving the performance of the model.

[0140] The embodiment of the present invention discloses a vertical federated learning system, including:

[0141] A data acquisition module, used to acquire new data samples;

[0142] The model building and training module is used to build a prediction model and train the prediction model, wherein the training process of the prediction model includes:

[0143] The first participant preprocesses the feature values ​​of the sample data according to the sample data contained therein, calculates feature correlation, performs feature selection, and selects the optimal subset;

[0144] Based on the optimal subset, the weighted quantile sketch is used to divide the samples into different buckets;

[0145] Based on the dynamic privacy budget allocation strategy, the features of each sample are evaluated, the feature importance is obtained according to the evaluation results, a privacy budget is allocated to each feature, and the order-preserving desensitization algorithm is improved based on the dynamic privacy budget allocation strategy to desensitize the samples;

[0146] The first participant collects feature information of all participants, finds the best segmentation point of the features and updates the prediction model to obtain a trained prediction model;

[0147] The prediction module is used to perform prediction analysis on new data samples through the trained prediction model.

[0148] An embodiment of the present invention discloses an electronic device, including: a memory and a processor;

[0149] The memory is configured to: store computer program instructions;

[0150] The processor is configured to: execute the computer program instructions so that the electronic device implements the above-mentioned vertical federated learning method.

[0151] The embodiment of the present invention discloses a readable storage medium, characterized by comprising: computer program instructions;

[0152] The computer program instructions are executed by at least one processor of an electronic device, so that the electronic device implements the above-mentioned vertical federated learning method.

[0153] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a vertical federated learning method, system, electronic device and readable storage medium.

[0154] The vertical federated learning mechanism based on boosted tree and adaptive differential privacy (DPTB-VFL) proposed in the embodiment of the present invention shows wide application potential in multiple fields, especially in scenarios where data privacy needs to be protected.

[0155] Specifically, in the field of financial services, banks and credit institutions can use DPTB-VFL for joint credit scoring and risk assessment without directly sharing customers' private data; secondly, in the healthcare industry, hospitals and research institutions can jointly train disease prediction models through this framework to improve the accuracy of diagnosis while complying with strict privacy regulations; furthermore, DPTB-VFL also has important applications in the field of e-commerce, and can help different e-commerce platforms optimize recommendation systems while protecting user privacy; finally, social network analysis is also an important application scenario of DPTB-VFL, and different social media platforms can jointly analyze user behavior through this framework to improve marketing effectiveness without having to worry about the leakage of user data.

[0156] In a specific embodiment of the present invention, taking the financial services industry as an example, especially banks and lending institutions, the credit scoring system is a key tool for assessing the credit risk of borrowers. In order to improve the efficiency of loan approval, financial institutions usually need to use a large number of users' personal information, transaction records and historical credit data to build prediction models. But at the same time, it is crucial to protect users' privacy data and avoid data leakage and abuse. In order to balance data privacy and efficient data analysis, DPTB-VFL can be used as an effective solution. First, each financial institution (such as a bank, credit rating company, etc.) pre-processes local data (users' personal basic information, credit history, loan records, repayment status, etc.), calculates feature importance and relevance, and selects features related to credit scoring. For example, the user's "credit card debt" feature may have a greater impact on credit score prediction, so it can be given a higher weight. Secondly, a weighted quantile thumbnail is used to divide the sorted data. Subsequently, based on the dynamic privacy budget allocation strategy, each financial institution evaluates the features and desensitizes the sensitive data through a probabilistic order-preserving desensitization algorithm to ensure that user privacy is not leaked. During the entire training process, user data is always kept locally, and the institutions share the model parameters and feature information after privacy protection. With the collaboration of all participants, after multiple rounds of model updates and optimization, a trained credit score prediction model will eventually be obtained. This model can integrate data from multiple financial institutions to predict whether the borrower has the ability to repay and future credit risk. Ultimately, this method helps financial institutions improve risk control efficiency while protecting users' personal privacy and avoiding data leakage risks.

[0157] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0158] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vertical federated learning method, characterized in that: include: Get new data samples; Constructing a prediction model and training the prediction model; The training process of the prediction model includes: The first participant preprocesses the feature values ​​of the sample data according to the sample data contained therein, calculates feature correlation, performs feature selection, and selects the optimal subset; Based on the optimal subset, the weighted quantile sketch is used to divide the samples into different buckets; Based on the dynamic privacy budget allocation strategy, the features of each sample are evaluated, the feature importance is obtained according to the evaluation results, a privacy budget is allocated to each feature, and the order-preserving desensitization algorithm is improved based on the dynamic privacy budget allocation strategy to desensitize the samples; The first participant receives feature importance information from all participants, finds the best segmentation point of the feature and updates the prediction model to obtain a trained prediction model; The trained prediction model is used to perform prediction analysis on new data samples.

2. A vertical federated learning method according to claim 1, characterized in that: The all participants include the first participant and the second participant; The first participant holds complete training sample labels; All participants hold some specific features of the sample data, and the sample identifiers overlap across all participants.

3. A vertical federated learning method according to claim 1, characterized in that: When selecting features, the boosted tree algorithm is used to calculate the importance of features and sort them; Calculate the correlation between features and target variables and between features; For each feature combination, its CFS score is calculated, and the optimal subset is selected based on the CFS score.

4. A vertical federated learning method according to claim 1, characterized in that: A weighted quantile outline is used to divide the samples into different buckets, including: For each feature in the optimal subset, sort the feature values ​​of the samples; Convert continuous numerical features into ordered categorical features; According to the objective function, the contribution of the sample to the objective function value is obtained by the second-order derivative h i Decision, Dataset D k ={(x 1k ,h1),(x 2k ,h2),…,(x nk ,h n )} represents the set of the kth eigenvalue and second-order derivative of each sample; Function r k (z) represents the percentage of the weight of samples with eigenvalues ​​less than z to the total weight of all samples; For the tangent point {s k1 ,s k2 ,…,s kl }, set the sampling frequency a to standardize the points to be divided so that the r of two connected tangent points k The absolute value of the difference between the values ​​is less than a.

5. A vertical federated learning method according to claim 1, characterized in that: Based on the dynamic privacy budget allocation strategy, the features of each sample are evaluated, the feature importance is obtained according to the evaluation results, and a privacy budget is allocated to each feature, including: Set feature k = 1, 2, ..., K, and the feature importance measurement formula is as follows: Where T represents the total number of trees, N(t) is the number of nodes in the t-th tree, and I(β(t,i)=k) is an indicator function. When node i is split by feature k on tree t, the function value is 1; otherwise, it is 0. is the square of the gradient of feature γ at node i, H γ(t,i) is the second-order derivative of feature γ at node i, is the square of the gradient of feature γ on the left subtree of node i, is the square of the gradient of feature γ on the right subtree of node i, H γ(t,i,L) is the second-order derivative of feature γ on the left subtree of node i, H γ(t,i,R) is the second-order derivative of feature γ on the right subtree of node i; in, It represents the sum of gains of all nodes i in all trees t when splitting on feature k, and the regularization term λ is added to control the model complexity; Indicates the number of times feature k is used as a split feature in all nodes on all trees.

6. A vertical federated learning method according to claim 5, characterized in that: The first participant normalizes the feature importance to obtain v norm(k) ; The formula for the privacy budget allocation strategy based on the normalized results is as follows: Among them, 0<α<1 and 0<β<1, α represents the growth rate of the controlled privacy budget, β represents the highest allocated privacy budget when the normalized feature influence is 0%, ε represents the total privacy budget, and ε′ represents the privacy budget allocated to the sample.

7. A vertical federated learning method according to claim 5, characterized in that: By calculating the sum of information gains brought by the feature when it is used as a split node in all decision trees, a standardized importance score is obtained. The higher the score, the greater the importance of feature k; If the denominator of the feature importance metric is 0, then feature k has never been used for segmentation, and the importance of feature k v (k) Also 0.

8. A vertical federated learning system, applying a vertical federated learning method according to any one of claims 1 to 7, characterized in that: include: A data acquisition module, used to acquire new data samples; The model building and training module is used to build a prediction model and train the prediction model, wherein the training process of the prediction model includes: The first participant preprocesses the feature values ​​of the sample data according to the sample data contained therein, calculates feature correlation, performs feature selection, and selects the optimal subset; Based on the optimal subset, the weighted quantile sketch is used to divide the samples into different buckets; Based on the dynamic privacy budget allocation strategy, the features of each sample are evaluated, the feature importance is obtained according to the evaluation results, a privacy budget is allocated to each feature, and the order-preserving desensitization algorithm is improved based on the dynamic privacy budget allocation strategy to desensitize the samples; The first participant collects feature information of all participants, finds the best segmentation point of the features and updates the prediction model to obtain a trained prediction model; The prediction module is used to perform prediction analysis on new data samples through the trained prediction model.

9. An electronic device, characterized in that: include: Memory and processor; The memory is configured to: store computer program instructions; The processor is configured to: execute the computer program instructions so that the electronic device implements a vertical federated learning method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: include: Computer program instructions; The computer program instructions are executed by at least one processor of an electronic device, so that the electronic device implements a vertical federated learning method as described in any one of claims 1 to 7.

Images