Privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning

Abstract

The present invention provides a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning. The method adds a privacy protection mechanism and a sampling method on the basis of the independence test method, protects the data of the first data party, and samples the control sample to reduce the transmission cost, the calculation cost of the independence test, and the data encryption cost; the second data party uses the independence test method to screen out a large number of redundant features, obtain low-dimensional sub-features, and use the traditional vertical federated learning method to build a machine learning model. The present invention greatly reduces the transmission cost, encryption cost, and calculation cost caused by the sample size, and at the same time greatly reduces the transmission cost and decryption cost of federated learning caused by the high feature dimension.

Description

Technical Field

[0001] The present invention relates to the technical field of vertical federated learning, and in particular to a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning. Background Art

[0002] Vertical Federated Learning (VFL) technology enables different data parties to jointly build more accurate machine learning models without sharing the original data. These data parties have different characteristics on the same batch of samples, but for privacy reasons, they cannot share the original data they hold with each other. Vertical Federated Learning technology has broad application potential in areas such as precision marketing and financial risk control.

[0003] Developing machine learning models for massive user data from different data parties is an important research topic. In this context, cryptographic protocols that allow calculations on encrypted data have become a key mechanism that can enable model training while protecting user privacy. Cryptographic protocols based on secure multi-party computation (MPC) have been developed one after another to train various types of machine learning models on aggregated data, including decision tree models, linear regression models, and models based on neural network architectures. These algorithms use encrypted communication between feature parties (data parties containing features) and response parties (data parties containing response variables) to put the massive data features contained in each feature party into a given model for joint training in order to obtain a higher quality model.

[0004] Existing research on machine learning methods based on secure multi-party computing focuses on the model training stage, and pays less attention to the important link of redundant feature screening. In the presence of a large number of redundant features, machine learning methods will increase the estimation error while bringing higher computational complexity. However, the advancement of data collection technology has brought about a large number of features. In real financial risk control scenarios, the ultra-high-dimensional features of each borrower are recorded in various data platforms, such as payment flow, travel records, etc. Among these features, a large number of features will not contribute to the construction of the real model, but will bring a lot of computing costs, communication costs and data encryption costs. Therefore, before applying the machine learning model, it is very necessary to screen these massive features. However, the existing few feature screening schemes either focus on the linear correlation between features and responses, or require binning of data to identify nonlinear correlations. The former has the risk of incorrect model assumptions, and the latter has information loss in the binning process.

[0005] Therefore, this field needs a model-independent method for selecting ultra-high-dimensional data features in federated learning. It will comprehensively utilize ultra-high-dimensional hypothesis testing technology and feature selection statistical theory to select a set containing real features while protecting user privacy, thereby providing support for machine learning methods that use vertical federated learning technology. Under the premise of ensuring that the computing cost is basically stable, it greatly reduces the communication cost and encryption cost in the model training process, and improves the stability and interpretability of the model. The communication cost and encryption cost are generated between the feature party and the responder, which is much greater than the computing cost of a single data party. Summary of the invention

[0006] The purpose of the present invention is to provide a feature screening method based on ultra-high dimensional hypothesis testing, which mainly provides support for machine learning joint modeling of vertical federated learning, aims to reduce the number of features entering the machine learning joint modeling and reduce the computational complexity in the training process.

[0007] The technical solution of the present invention is as follows.

[0008] The first aspect of the present invention provides a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning, the method being performed by a first data party and a second data party, wherein the first data party has a response variable of a data set. ,in For events, For control; the second data party has a large number of features x , whose dimensions are d , The feature screening method is used to select d Dimensional Features x Extract Dimension vector ,in ;

[0009] The method comprises:

[0010] Step S1: the first data source responds to the variable of controls and retain all events, reducing the size of the dataset to ;in Indicates the number of events, Represents the sampling probability parameter; the distance matrix of the sampling data set Encrypted as and then transmit it to the second data party;

[0011] Step S2: the second data entity stores its own features Encrypted value of the distance from the response variable Align in the sample and store itselfd Dimensional Features Divide into multiple disjoint groups, each group constitutes a subvector of the feature , m is the number of features in each group during the first grouping; select the independence test statistic T , calculate the subvector With the response variable The encrypted value of the statistic , and obtain the distribution and rejection region of the statistic under the condition that the null hypothesis is established;

[0012] Step S3: Encrypt the statistic The first data is sent back for decryption to obtain the plaintext of the statistics. ;

[0013] Step S4: the second data entity uses statistics and its rejection region to determine whether to reject the null hypothesis. If the null hypothesis is not rejected, k The feature groups are considered to be independent of the response variable, thus k Feature groups are considered redundant features and are screened out;

[0014] Step S5, repeating steps S2 to S4, continuously splitting the small group containing important features into smaller groups and executing the hypothesis testing scheme until the number of features in each group is less than the first threshold ;

[0015] Step S6: Calculate the independence test statistic for each feature and filter out all features whose absolute value is less than the second threshold. The independent features are finally filtered out and put into the machine learning model.

[0016] Preferably, in step S1, the sampling process of the first data cube further includes:

[0017] For i indivual The control sample constructs a random variable ,when At that time, i indivual of controls are extracted into the new sample set; Indicates the number of controls, is a pre-specified constant.

[0018] Preferably, after sampling the data, the first data party constructs a distance matrix using a secure multi-party computing encryption method. ,in Each element of , and the ciphertext of the distance matrix Transmitted to the second data party.

[0019] Preferably, in step S2, after receiving the ciphertext from the first data party, the second data party selects a model-independent statistic Perform independence test, the model is irrelevant to the statistic The selection meets the following conditions:

[0020] (1) Statistics The ciphertext Can be calculated through secure multi-party computation;

[0021] (2) Statistics The null hypothesis is close to 0;

[0022] (3) Statistics It will deviate significantly from 0 when the null hypothesis is violated;

[0023] (4) Statistics It is model-free and can measure any form of correlation.

[0024] Preferably, the statistic T Selected as a statistic constructed based on distance covariance , which has the following form:

[0025] The kernel function is constructed as:

[0026]

[0027] but It has the form of the generalized U statistic,

[0028] .

[0029] Preferably, in step S4, after decryption, After the plain text, if Falling into the rejection region, it means that the feature group With the response variable Significantly not independent, the characteristics within the group are meaningful for model construction; otherwise, it is considered The features and response variables in There was no significant correlation between the two groups, so this group of features was eliminated.

[0030] Preferably, in step S5, each time the small group containing important features is split into two smaller groups and a hypothesis testing scheme is performed, the number of features in each small group is reduced to one half.

[0031] Preferably, in step S6, the machine learning model is constructed using a longitudinal federated learning method.

[0032] The second aspect of the present invention provides a computer device, comprising a processor and a memory; the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning as described in any one of the first aspects of the present invention.

[0033] The third aspect of the present invention provides a computer-readable storage medium, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning as described in any one of the first aspects of the present invention.

[0034] The ultra-high-dimensional feature screening scheme of the present invention is applicable to the feature screening process before training of any machine learning model, and is model-independent, and can identify the correlation between the response variable and the feature in any model form. On the one hand, the present invention considers the sampling method for rare event data, which greatly reduces the transmission cost, encryption cost and computing cost caused by the sample size. On the other hand, the present invention abandons the one-by-one screening scheme of the traditional feature screening scheme, and considers putting related features in a group for joint hypothesis testing. The feature screening method of joint hypothesis testing reduces the number of statistical calculations, thereby greatly reducing the transmission cost and decryption cost of federated learning caused by the excessively high feature dimension. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning according to the present invention;

[0036] Figure 2 yes Figure 1 A diagram showing the privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning.

[0037] The meaning of each reference numeral in the figure: A, first data cube; B, second data cube. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0039] The goal of the present invention is to design a feature screening method based on ultra-high dimensional hypothesis testing, which mainly provides support for machine learning joint modeling of longitudinal federated learning, aiming to reduce the number of features entering machine learning joint modeling and reduce the computational complexity in the training process. In particular, the present invention focuses on binary classification models in rare events. Rare events refer to a situation in which the amount of data in one category is large, while the data in the other category is relatively scarce in a classification task. Such scenarios are extremely common in financial risk control and precision marketing scenarios. For example, in financial risk control cases, user defaults are rare events.

[0040] like Figure 1-2 As shown, the feature screening method according to the present invention is performed by a first data cube A and a second data cube B. The first data cube A has a response variable of a data set ,in is a rare event, called an "event", is called "control". The second data entity B has a large number of features x , whose dimensions are d ,Right now , the feature screening task aims to d Dimensional Features x Extract Dimension vector ,in The subvector The response variable must be included The model construction has the characteristics of contribution. Under the framework of federated learning and privacy protection, the feature screening and joint modeling of the first data party A and the second data party B require computing costs, transmission costs and encryption costs. The sample size of the control is , the sample size of rare events is In classic practice scenarios such as financial risk control, there are often even .

[0041] The feature screening scheme designed by the present invention is based on ultra-high dimensional independence test. The vector of dimensional features Does not contribute to the construction of the machine learning model, i.e. the response variable With the Dimensional vector Mutually independent. The independence of features and response variables can be tested based on the theoretical method of ultra-high dimensional independence test, and a large number of features related to response variables Independent feature elimination, and using the remaining features to build a machine learning model.

[0042] The feature screening method designed by the present invention adds a privacy protection mechanism and a sampling method on the basis of the independence test method, protects the data of the first data party A, and samples the control sample to reduce the transmission cost, the computational cost of the independence test, and the data encryption cost; the second data party B uses the independence test method to screen out a large number of redundant features, obtain low-dimensional sub-features, and uses the traditional longitudinal federated learning method to build a machine learning model.

[0043] Specifically, Figure 1 As shown, the feature screening method of the present invention has the following six steps:

[0044] Step S1: The first data entity A calculates the response variable of controls and retain all events, reducing the size of the dataset to ,in Represents the sampling probability parameter. The distance matrix of the sampled data set Encrypted as Then transmit it to the second data party B;

[0045] Step S2: The second data entity B stores its own features Encrypted value of the distance from the response variable Align in the sample and store itself d Dimensional Features Divide into multiple disjoint groups, each group constitutes a subvector of the feature ,in m is the number of features in each group during the first grouping. Select an appropriate independence test statistic T , calculate the subvector With the response variable The encrypted value of the statistic , and the original hypothesis ( The distribution and rejection region of the statistic under the condition that independence holds;

[0046] Step S3: Encrypt the statistic The data is sent back to the first data party A for decryption, and the plain text of the statistics is obtained. ;

[0047] Step S4: The second data cube B uses statistics and its rejection region to determine whether to reject the null hypothesis. If the null hypothesis is not rejected, k The feature groups are considered to be independent of the response variable, thus k Feature groups are considered redundant features and are screened out;

[0048] Step S5, repeating steps S2 to S4, continuously splitting the small group containing important features into smaller groups and executing the hypothesis testing scheme until the number of features in each group is less than the first threshold .

[0049] Step S6: Calculate the independence test statistic for each feature and filter out all features whose absolute value is less than the second threshold. The independent features are finally filtered out and put into the machine learning model.

[0050] In the solution of the present invention, the response variable of the data set is is a rare event, and the important information in the data set is mainly concentrated in In the events, a large number of The control samples are redundant. Therefore, the scheme of the present invention samples the control samples before feature screening.

[0051] In a preferred embodiment, in step S1, the specific method of sampling the first data cube A is: i indivual The control sample constructs a random variable ,when At that time, i indivual The controls are extracted into the new sample set, where Indicates the number of events, Indicates the number of controls, is a pre-specified constant. Thus, the size of the data set is reduced to , which greatly reduces the transmission cost, encryption cost and computing cost caused by the sample size.

[0052] After sampling the data, the first data party A will use the encrypted method of secure multi-party computing to construct the distance matrix ,in Each element of , and the ciphertext of the distance matrix Transmitted to the second data party B. Considering the response is a univariate random variable, the encryption cost and computational cost of this step are . Transmission distance matrix The technical point is that directly transferring labels will make it impossible to calculate very high-dimensional test statistics based on sorting.

[0053] After receiving the ciphertext from the first data party A, the second data party B selects the model-independent statistic Conduct independence test.

[0054] In a preferred embodiment, before performing the independence test in step S2, the second data cube B first contains a large number of features. Divide into multiple non-overlapping feature groups , and calculate the response variable And each feature group The encrypted value of the statistic .

[0055] Among them, the model-independent statistics The selection should satisfy the following four properties:

[0056] (1) Statistics The ciphertext Can be calculated through secure multi-party computation;

[0057] (2) Statistics The null hypothesis is close to 0;

[0058] (3) Statistics It will deviate significantly from 0 when the null hypothesis is violated;

[0059] (4) Statistics It is model-free and can measure any form of correlation.

[0060] In general, statistics The common construction method is to choose a suitable kernel function , the expected value of the kernel function satisfies the above conditions (2)-(4). Based on this kernel function, the generalized U statistic is:

[0061] ,

[0062] in and Corresponding to the response variables and The sample characteristics of j 0 and j 1 It is needed when constructing the generalized U statistic Samples and The sample size of the sample.

[0063] In a preferred embodiment, the statistic T Select as a statistic constructed based on distance covariance , has the following form: The kernel function is constructed as,

[0064]

[0065] but It has the form of the generalized U statistic,

[0066] .

[0067] It can be verified that the statistic meets the above four requirements.

[0068] In step S4, decryption is performed to obtain After the plain text, if Falling into the rejection region, it means that the feature group With the response variable Significantly not independent, the characteristics within the group are meaningful for model construction; otherwise, it is considered The features and response variables in There was no significant correlation between the two groups, so this group of features was eliminated.

[0069] Repeat steps S2 to S4 above, split the group containing important features into smaller groups and perform hypothesis testing until the number of features in each group is less than the first threshold. .

[0070] In a preferred embodiment, in step S5, each time the group containing important features is split into two smaller groups and a hypothesis testing scheme is performed, the number of features in each group is reduced to one half.

[0071] Those skilled in the art will appreciate that the first threshold It can be determined according to the actual number of features, and in a preferred embodiment, the first threshold The reference value is 5.

[0072] Finally, the independence test statistic is calculated for each feature, and all the statistics with an absolute value less than the second threshold are screened out. The independent features are finally screened and the feature set is put into the machine learning model. Those skilled in the art will understand that the second threshold is a constant, and its value can be determined according to the total number of sample features required by actual business.

[0073] It can be proved that for Statistics, sampling and full samples have similar statistical properties in ultra-high-dimensional independence tests, which is guaranteed by the following theorem.

[0074] When the original hypothesis is true and the data distribution meets certain conditions under the high-dimensional hypothesis:

[0075] Statistics obtained using the full sample satisfy: The limiting distribution of is a normal distribution with mean 0.

[0076] Statistics obtained using sample satisfy: The limiting distribution of is a normal distribution with mean 0. When , the variance is the same as the variance of the full sample statistic.

[0077] Statistical theory can prove that the ultra-high-dimensional feature screening technology involved in the above steps can select a set of all important features.

[0078] For all responses The expected value of the correlation statistic The true characteristics of , in the event sample size Under the condition of , the probability that all true features are retained is close to 1.

[0079] Therefore, in the event sample size Under sufficiently large conditions, the ultra-high dimensional feature screening method proposed in the present invention can well meet the requirements of vertical federated machine learning technology.

[0080] Table 1 below illustrates the advantages of the screening method of the present invention over the traditional feature screening method in terms of computational cost, transmission cost, and encryption and decryption cost:

[0081] Table 1 Comparison between the present invention and traditional feature screening methods

[0082]

[0083] : The number of features to be screened in the data cube;

[0084] : The number of real features, satisfying ;

[0085] : The number of features in each group during the first grouping;

[0086] : After each hypothesis test, the significantly correlated feature groups are split into New groups;

[0087] : event( ) of the samples;

[0088] : Total sample size;

[0089] : Sampling probability parameter, which describes the probability of the control sample being sampled .

[0090] In real application scenarios, there are often .

[0091] When the number of real features is much smaller than the total number of features, the complexity of packet filtering in transmission and encryption and decryption can grow logarithmically. , while the cost of screening one by one increases linearly. Therefore, the federated learning feature screening scheme of the present invention reduces the transmission cost and encryption and decryption cost by an order of magnitude while ensuring that the calculation cost is not much different through the method of group inspection and sampling calculation.

[0092] The ultra-high-dimensional feature screening scheme designed by the present invention is applicable to the feature screening process before training any machine learning model, and is model-independent, and can identify the correlation between the response variable and the feature in any model form. On the one hand, the present invention considers the sampling method for rare event data, which greatly reduces the transmission cost, encryption cost and computing cost caused by the sample size. On the other hand, the present invention abandons the one-by-one screening scheme of the traditional feature screening scheme, and considers putting related features in a group for joint hypothesis testing. The feature screening method of joint hypothesis testing reduces the number of statistical calculations, thereby greatly reducing the transmission cost and decryption cost of federated learning caused by the excessively high feature dimension.

[0093] The present invention also provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the feature screening method in federated learning described in the above embodiment.

[0094] Optionally, the computer readable storage medium may include: ROM, RAM, solid state drives (SSD) or optical disks, etc. Among them, RAM may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0095] The present invention also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning described in the above embodiment.

[0096] Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0097] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning, characterized in that: The method is performed by a first data entity and a second data entity, wherein the first data entity has a response variable of a data set ,in For events, For control; the second data party has a large number of features x , whose dimensions are d , The feature screening method is used to select d Dimensional Features x Extract Dimension vector ,in ; The feature screening method comprises: Step S1: the first data source responds to the variable of controls and retain all events, reducing the size of the dataset to ;in Indicates the number of events, Represents the sampling probability parameter; the distance matrix of the sampling data set Encrypted as and then transmitting it to the second data party; Step S2: the second data entity stores its own features Encrypted distance value from the response variable Align in the sample and store itself d Dimensional Features Divide into multiple disjoint groups, each group constitutes a subvector of the feature , m is the number of features in each group during the first grouping; select the independence test statistic T , calculate the subvector With the response variable The encrypted value of the statistic , and obtain the distribution and rejection region of the statistic under the condition that the null hypothesis is established; Step S3: Encrypt the statistic The first data is sent back for decryption to obtain the plaintext of the statistics. ; Step S4: the second data entity uses statistics and its rejection region to determine whether to reject the null hypothesis; if the null hypothesis is not rejected, then k The feature groups are considered to be independent of the response variable, thus k Feature groups are considered as redundant features and are screened out; Step S5, repeating steps S2 to S4, continuously splitting the small group containing important features into smaller groups and executing the hypothesis testing scheme until the number of features in each group is less than the first threshold ; Step S6: Calculate the independence test statistic for each feature and filter out all the statistics whose absolute value is less than the second threshold. The independent features are finally filtered out and put into the machine learning model.

2. According to claim 1, a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning is characterized in that: In step S1, the sampling process of the first data cube further includes: For i indivual Construct random variables from the control sample ,when At that time, indivual of controls are extracted into the new sample set; Indicates the number of controls, is a pre-specified constant.

3. According to claim 2, a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning is characterized in that: After sampling the data, the first data party constructs a distance matrix using a secure multi-party computing encryption method. ,in Each element of , and the ciphertext of the distance matrix Transmitted to the second data party.

4. According to claim 1, a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning is characterized in that: In step S2, after receiving the ciphertext of the first data party, the second data party selects a model-independent statistic Perform independence test, the model is irrelevant to the statistic The selection meets the following conditions: (1) Statistics The ciphertext Can be calculated through secure multi-party computation; (2) Statistics The null hypothesis is close to 0; (3) Statistics It will deviate significantly from 0 when the null hypothesis is violated; (4) Statistics It is model-free and can measure any form of correlation.

5. According to claim 4, a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning is characterized in that: The statistic T Selected as a statistic constructed based on distance covariance , which has the following form: The kernel function is constructed as: but It has the form of the generalized U statistic, 。 6. According to claim 5, a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning is characterized in that: In step S4, after decryption, After the plain text, if falls into the rejection region, it means that k The sub-vectors composed of feature groups With the response variable Significantly not independent, the characteristics within the group are meaningful for model construction; otherwise, it is considered The features and response variables in There was no significant correlation between the two groups, so this group of features was eliminated.

7. According to claim 1, a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning is characterized in that: In step S5, each time the group containing important features is split into two smaller groups and the hypothesis testing scheme is performed, the number of features in each group is reduced to one half.

8. The privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning according to claim 1, characterized in that: In step S6, the machine learning model is constructed using a longitudinal federated learning method.

9. A computer device, characterized in that: The computer device includes a processor and a memory; the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement a privacy-preserving ultra-high-dimensional feature screening method in vertical federated learning as described in any one of claims 1 to 8.

Images