A method and system for generating multi-party, multi-class data for vertical federated learning

Abstract

This invention relates to a method and system for generating multi-party, multi-class data for vertical federated learning, belonging to the field of data perception and reconstruction in computer science. The invention includes: S1: establishing a training dataset for vertical federated learning aimed at generating multi-party, multi-class data models; S2: constructing a dual-conditional generative adversarial network (GAN) for vertical federated learning; S3: designing a multi-party loss function for the dual-conditional GAN ​​for vertical federated learning; S4: establishing a training process for the dual-conditional GAN ​​for vertical federated learning; S5: designing and implementing a multi-party, multi-class data generation scheme based on the dual-conditional GAN. This invention provides a method for jointly expanding the feature dimensions and sample quantity of tabular datasets by multiple parties, jointly constructing a high-quality tabular training dataset to meet the needs of machine learning model training in big data application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of data perception and reconstruction in computers, and relates to a method and system for generating multi-party, multi-class data for vertical federated learning. Background Technology

[0002] In today's era of big data, data is often isolated and owned by different organizations or institutions. Furthermore, data from institutions like hospitals and banks involves user privacy, making it extremely difficult for different stakeholders to aggregate data and build a high-performance machine learning model. Federated learning can effectively solve these problems by training a shared high-performance model for multiple stakeholders while protecting the data privacy of each stakeholder. Based on the distribution of training data across the data feature space and sample ID space of different stakeholders, federated learning can be categorized into horizontal federated learning, vertical federated learning, and federated transfer learning. Federated learning involves stakeholders and collaborators. Stakeholders collaborate to build a shared machine learning model, while collaborators are responsible for information encryption / decryption and determining whether global training should terminate. This invention primarily applies to vertical federated learning.

[0003] While vertical federated learning can effectively solve the data silo problem, it still has many problems in practical application scenarios. Data is the "raw material" for training machine learning models. When participants A and B have missing values ​​in their data, data imbalance, insufficient data from one party leading to very little data after data alignment and waste of non-aligned data, it will be impossible to train a high-performance machine learning model.

[0004] In recent years, generative adversarial networks (GANs) have become increasingly important and popular due to their applicability across various fields. For tabular datasets, multiple parties need to collaborate on data generation to expand the dimensionality and number of samples in the training dataset. Constructing high-quality tabular training datasets through collaborative multi-party data generation methods meets the training needs of machine learning models in big data application scenarios. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method and system for generating multi-party, multi-class data for vertical federated learning, constructing a dual conditional generative adversarial network (DCTGAN) for vertical federated learning, expanding the sample dimension and sample quantity in the training dataset, and jointly constructing a high-quality tabular training dataset with multiple parties to meet the needs of machine learning model training in big data application scenarios.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for generating multi-party, multi-class data for vertical federated learning, comprising the following steps:

[0008] S1: Establish a longitudinal federated learning training dataset for the purpose of generating multi-party, multi-class data model training;

[0009] S2: Constructing a dual-condition generative adversarial network for vertical federated learning;

[0010] S3: Design a multi-party loss function for a dual-condition generative adversarial network oriented towards vertical federation;

[0011] S4: Establishing the training process for a dual-condition generative adversarial network oriented towards vertical federated learning;

[0012] S5: Design and implement a multi-party, multi-class data generation scheme based on dual-condition generative adversarial networks.

[0013] Optionally, S1 specifically includes:

[0014] Client A and Client B have user groups belonging to different systems. They use Privacy Preservation Set Intersection (PSI) technology to align encrypted entities and represent the sample IDs of the party with completely missing data.

[0015] Optionally, S2 specifically includes:

[0016] Client A and Client B construct a dual-condition generative adversarial network, with each party having its own generative adversarial subnetwork, which includes a generative model and a discriminative model.

[0017] S2-1: The two-party generative model consists of two ResNet residual networks and a fully connected layer. Its input includes input data and a conditional vector, and both parties process the input data as follows:

[0018] (1) Client A's mask vector Client B Client A and Client B d When M = 0, it means that data at that position is missing; when M = 0, it means that data at that position is missing. d =1 indicates that the data is complete; the dimension of the element with value 1 in each party M is ||M||1, while the dimension of the element with value 0 is ||1-M||1. For example, in party A M, the dimension of the element with value 1 is ||M||1. A ||1, and so on; when all elements in both sides M are 0, it means that only the sample category is used as a condition.

[0019] The encoding vector X of the feature data of the real sample on client A's side A Client A simulates missing sample data vectors under different scenarios.

[0020]

[0021] Right now

[0022]

[0023] The encoding vector X of the feature data of client B B Client B simulates missing sample data vectors under different scenarios.

[0024]

[0025] Right now

[0026]

[0027] Where "⊙" represents element-wise multiplication between vectors;

[0028] (2) Client A and Client B respectively tested the noise sample Z A and noise sample Z B Encode the data and use the encoded results to fill in the missing sample data to obtain Z. A′ and Z B′ The process of client A is represented as follows: The process of client B is represented as follows

[0029] (3) Client A will use the condition vector cond A With Z A′ By splicing Client B will use the condition vector cond B With Z B′ By splicing As input to the generative models of both parties, This represents a vector concatenation operation;

[0030] S2-2: The discriminant model for Client A and Client B consists of three fully connected networks that process the generated data of the generative model to meet the input requirements of the discriminant model.

[0031] S2-3: Client A will generate data. Missing data With mask vector M A Element-wise multiplication is used to obtain the input data for the discriminant model. To represent, that is and Client B will generate data Missing data With mask vector M B Element-wise multiplication is used to obtain the input data for the discriminant model. To represent, that is and

[0032] Optionally, S3 specifically includes:

[0033] The steps for designing the objective function of the discriminant model are as follows:

[0034] S3-1: m represents the number of samples, and u represents the dimension of the encoded samples. Indicates sample X imp The given probability is satisfied; when training the discriminant model for Client A and Client B, the discriminant model must not only determine whether the generated data conforms to the data distribution of the real samples of both parties, that is, the degree of similarity between the generated elements and the real elements under the action of (1-M), but also determine whether the jointly generated samples of both parties satisfy the category label; for each sample of Client A, by calculating and real sample X A Given condition cond A The cross-entropy loss is used to determine whether a sample matches the category. For each sample from customer B, the cross-entropy loss is calculated... and real sample X B Given condition cond B The cross-entropy loss is used to determine whether a sample matches a category. In this case, the loss for client A is calculated as follows:

[0035]

[0036] The losses of client B are calculated as follows:

[0037]

[0038] S3-2: Client A and Client B use the Wasserstein distance from Wasserstein GANs to calculate X. imp Similarity of the missing parts between ⊙(1-M) and X⊙(1-M), Client A and They represent Set and X A Set Obedience Probability distribution; Client B and They represent Set and X B Set Obedience Probability distribution, where "γ" and "∈" are The hyperparameters; its loss is expressed as L. w This indicates that the Wasserstein distance for A at this point is calculated as follows:

[0039]

[0040] The Wasserstein distance for B is calculated as follows:

[0041]

[0042] in This represents the "gradient penalty" mechanism in the Wasserstein distance;

[0043] S3-3: At this point, the total discriminant model loss for client A is expressed as:

[0044]

[0045] The total loss of the discriminant model for Client B is:

[0046]

[0047] Where “δ” is a hyperparameter; D(X|cond) means inputting sample X and condition vector cond into the discriminant model; where m represents the number of samples and u represents the dimension of the encoded sample, which can be specific to A and B, such as uB representing the data dimension of the encoded sample in B.

[0048] S3-4: Aggregate the losses of the discriminant models for both A and B; the total loss for both sides is calculated as follows: As the loss of the two-party discriminant model;

[0049] The steps for designing the objective function of the generative model are as follows:

[0050] S3-5: Client A and Client B generate models to determine whether the generated data can fool the discriminant model, and the data distribution fit between the generated elements and the elements of the real samples under the influence of each party M. Client A is represented as... The larger the value, the more it indicates The more it meets the given conditions; Client B is represented as The larger the value, the more it indicates The more it meets the given conditions, the better, where i represents the i-th sample in the sample set, i = 1...m;

[0051] S3-6: Client A and Client B also need to calculate M⊙X miss and The mean squared error of each element in each sample, whose loss function is... The mean squared error loss for customer A is calculated as follows:

[0052]

[0053] The mean square error loss for customer B is calculated as follows:

[0054]

[0055] Where α is a hyperparameter

[0056] S3-7: The loss of the model generated by client A is expressed as:

[0057]

[0058] The loss of the model generated by client B is:

[0059]

[0060] S3-8: Aggregate the losses of the two generative models, i.e.

[0061] Optionally, S4 specifically includes:

[0062] S4: Considering that homomorphic encryption is used to encrypt intermediate information to protect the data privacy of clients A and B in federated learning, the training process of DCTGAN, which is designed for vertical federated learning, is as follows:

[0063] S4-1: Both parties use the method in step S1 to align encrypted entities;

[0064] S4-2: Party A initializes the network parameters of clients A and B and sends them to clients A and B; Party P generates a homomorphic encryption public key p. k Private key s k and the public key p k Send to both customer A and customer B;

[0065] S4-3: Client A and Client B respectively analyze the noise sample Z. A and noise sample Z B Encode the data and use the encoded results to fill in the missing sample data to obtain Z. A′ and Z B′ The process of client A is represented as follows: The process of client B is represented as follows Client A and Client B will each give Z A′ and Z B′ Each input data in the dataset is associated with its respective condition vector cond. A ,cond B Concatenate the two inputs to obtain the generative models from both sides, represented as follows:

[0066] S4-4: Client A and Client B first fix their own generation models and train their own discrimination models; both parties sample X from m real samples. A X B and The loss is obtained by training the discriminative model using the input as input.

[0067]

[0068] Similarly, Party B calculated the loss.

[0069]

[0070] Both parties will exchange and aggregate the encrypted losses to obtain... Party B will send the total loss of the discrimination model to Party P, the collaborating party.

[0071] S4-5: A and B are optimized using the Adam optimizer, and the gradient of parameter A is calculated as follows:

[0072]

[0073] The B-square gradient is calculated as follows:

[0074]

[0075] Party A adds a random mask to the gradient information of the discriminant model to obtain... Party B adds a random mask to the discriminant information to obtain... Both parties will send the gradient information to collaborating party P for decryption;

[0076] S4-6: Collaborator P decrypts the gradients of A and B. Collaborator P decrypts the gradient information and sends the result back to A and B. A and B remove the mask on the gradient information and obtain the true gradient information to update their respective discrimination model networks.

[0077] S4-7: Client A and Client B obtain the decryption gradient information and perform backpropagation. The backpropagation process is as follows:

[0078] The discriminant model network for Client A and Client B is L. D If the layer is l, then the gradient update process of the l-th layer is as follows:

[0079]

[0080] Where Φ D Let K represent the weight matrix. D Represents the bias matrix. This represents the differentiation of the weights of the l-th layer. Indicates the Lth D Input for layer +1, Indicates the Lth D Differentiate the layer output. This represents the differentiation with respect to the LeaykReLu activation function. This represents the input of the Lth layer. Indicates the Lth D Differentiate the output of layer -1. This represents the input of the (l+2)th layer. This represents the differentiation of the output of the (l+1)th layer. This represents the input of the (l+1)th layer;

[0081] The parameter update process for layer l is as follows:

[0082]

[0083] in and These represent the weights and biases of the l-th layer of the discriminant model network, respectively.

[0084] Both Party A and Party B shall perform the calculations according to the above process.

[0085] S4-8: Client A and Client B fix their own discrimination models and train their own generative models; A and B sample m Z′, where Z′ represents the result of filling missing sample data with encoded noise samples, and input them into their respective generative models to obtain m generated data. Indicates that data will be generated. Missing data X miss The input data for the discriminative model is obtained by performing element-wise multiplication with the mask vector M, and the result is represented by X. imp To represent, that is Get X impA X impB The input is used to determine X in the discriminant model. imp Whether the given conditions are met also needs to be calculated in relation to X. miss Mean square error loss between The losses suffered by Client A are as follows:

[0086]

[0087] Party B's losses are:

[0088]

[0089] Client A and Client B encrypt the calculated generative model loss. Both parties exchange mean squared error losses and converge to obtain Party B will send the total loss of the generated model to collaborating party P.

[0090] S4-9: Client A and Client B use the Adam optimizer to optimize and update parameters, lr G The learning rate for the generative model; the formula for calculating the gradient of the A-squared is:

[0091]

[0092] The formula for calculating the B-square gradient is:

[0093]

[0094] Both parties added a random mask to the encryption loss to obtain... and Both parties will send the gradient information to collaborating party P for decryption;

[0095] S4-10: Collaborator P decrypts the gradient information and sends the result back to clients A and B. Clients A and B remove the mask on the gradient information and obtain the real gradient information to update their respective generated model networks.

[0096] S4-11: Client A and Client B obtain the decrypted generator model gradient information and perform backpropagation. The backpropagation process is as follows:

[0097] During the reverse propagation process, the generator networks of client A and client B are L. G If the layer is l, then the gradient update process of the l-th layer is as follows:

[0098]

[0099] Where Φ G Let K represent the weight matrix. G Represents the bias matrix. This represents the differentiation of the weights of the l-th layer. L represents G Input for layer +1, Indicates the Lth G Differentiate the layer output. This represents the input of the Lth layer. Indicates the Lth G Differentiate the output of layer -1. This represents the input of the (l+2)th layer. This represents the differentiation of the output of the (l+1)th layer. This represents the input of the (l+1)th layer;

[0100] The parameter update process for layer l is as follows:

[0101]

[0102] in and These represent the weights and biases of the l-th layer of the generative network, respectively.

[0103] Both Party A and Party B shall perform the calculations according to the above process.

[0104] S4-12: Collaborating party P will calculate the total loss of the models generated by both parties. and discriminant model The total loss is aggregated. Global training terminates when the loss is less than a certain value. If the loss still does not converge, the above steps are repeated until the losses of both sides converge.

[0105] Optionally, S5 specifically includes:

[0106] The data loss situations for Client A and Client B differ and can be categorized into three scenarios:

[0107] (1) The feature data of the sample in both customer A and customer B have some missing values. Refer to step S2 to fill the missing data with random noise. Refer to step S4 to train the longitudinal federated learning DCTGAN. After both parties have finished training, input random noise, known data and corresponding condition vectors into the generator model to generate samples. Fill the missing data of the generated samples into the real data.

[0108] (2) The data features of the sample in either customer A or customer B are missing. Create a mask vector completely filled with "0". Fill the missing data with random noise according to the steps in S2. Train the longitudinal federated learning DCTGAN according to the steps in S4. After both parties are trained, input the condition vector corresponding to the random noise into the generator model to generate samples. Fill the missing data of the generated samples into the real data.

[0109] (3) There is a situation of imbalance in the sample categories when the samples are aligned by multiple parties: randomly fill the mask vector of the real data with 0, fill the missing data with random noise according to the steps in S2, and train the longitudinal federated learning DCTGAN according to the steps in S4. After both parties have finished training, concatenate the random noise with the minority class label and input it into the generative model to generate minority class sample filling data; or use the existing data to generate the model locally on each party, generate some existing data as one of the two conditions, and then use the above-mentioned scheme to generate the missing data, so as to generate minority class samples that meet a certain category condition.

[0110] A computer system includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the method.

[0111] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0112] The beneficial effects of this invention are as follows:

[0113] (1) This invention proposes a multi-party, multi-class data generation method for vertical federated learning, which allows multiple parties to jointly model and train the generative model while protecting their respective data security and privacy. Simultaneously, this invention designs a model structure and loss function for a dual-condition generative adversarial network for vertical federated learning, ultimately achieving multi-party, multi-class data generation for tabular datasets through the trained model.

[0114] (2) To address the various missing data scenarios in multi-party data after vertical federated learning alignment, this invention proposes a scheme for generating multi-party, multi-class data based on a dual-conditional generative adversarial network model of vertical federated learning. Furthermore, the technology and scheme of this invention are easily extensible and can be extended to three or more client parties holding features. It effectively solves problems such as missing data imputation and data imbalance between sample categories in multi-party data, expands the sample dimension and number in the training dataset, and jointly constructs a high-quality tabular training dataset with multiple parties, meeting the needs of machine learning model training in big data application scenarios.

[0115] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0116] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0117] Figure 1 This is a flowchart illustrating a multi-party, multi-class data generation method for vertical federated learning.

[0118] Figure 2 This is a schematic diagram illustrating multi-party data alignment in vertical federated learning.

[0119] Figure 3 This is a schematic diagram of training a dual generative adversarial network for vertical federated learning.

[0120] Figure 4 This is a schematic diagram illustrating the generation of multi-party, multi-class data for vertical federated learning. Detailed Implementation

[0121] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0122] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0123] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0124] Taking loan risk assessment as an example, information related to customer loan risk assessment is held by multiple parties, each with its own data security and privacy protection requirements. When multi-party data undergoes vertical federated learning and sample alignment, it is often difficult to guarantee obtaining massive amounts of high-quality sample data for learning. For example... Figure 2 As shown, taking vertical federated learning between two parties as an example, the sample labels corresponding to the combined data features of parties A and B are multiple categories such as 'a1', 'a2', ..., 'am', where 'a1' represents extremely high risk, 'a2' represents relatively high risk, 'a3' represents moderate risk, and 'a4' represents no risk, etc. After aligning the data from parties A and B, the following situation will occur:

[0125] ① In one or more categories, the data features of both Party A and Party B are complete (i.e., without missing data);

[0126] ② In one or more categories, data from either party A or party B is completely missing, while only the data features from the other party are complete, such as... Figure 1 The area marked by the number "2" in the middle;

[0127] ③ In one or more categories, data from Party A or Party B is partially missing, while data from the other party is complete, such as... Figure 1 The area marked by the number "3" in the middle;

[0128] ④ In one or more categories, data from both Party A and Party B are partially missing, and neither party has complete data features, such as... Figure 1 The area marked by the number "4" in the middle;

[0129] ⑤ The combined longitudinal sample of Party A and Party B has fewer samples in one or more categories, leading to an imbalance in sample data among the categories, such as... Figure 2 The categories 'a3', 'a4' and 'a1', 'a2' exhibit an imbalance in sample size.

[0130] In this embodiment: Party A in the loan risk assessment is an institution possessing information such as personal loans, repayments, defaults, and risk assessment levels; Party B is an institution possessing information such as personal age, income, family income, family debt, and risk assessment levels. When data gaps or imbalances in sample categories occur after the two parties are aligned, the multi-party, multi-class data generation method for vertical federated learning provided by this invention can generate data for Parties A and B while protecting data privacy, and jointly construct a high-quality training dataset that meets the requirements for training a high-performance loan risk assessment machine learning model.

[0131] Please see Figures 1-4 This is a method for generating multi-party, multi-class data for vertical federated learning, which includes the following steps:

[0132] S1: The specific steps for establishing a longitudinal federated learning sample set aimed at generating multi-party, multi-class data are as follows:

[0133] S1-1: Both parties use privacy-preserving set intersection techniques to align encrypted entities. Client A and Client B belong to different systems. A multi-party privacy set intersection method is used to align the IDs of encrypted users from both parties, and the sample IDs of the party with completely missing data are represented. Optionally, RSA-based algorithms and hash functions can be used for sample alignment.

[0134] S2: Client A and Client B construct a dual-condition generative adversarial network, with each side having its own generative adversarial sub-network, which includes a generative model and a discriminative model. The network structures of the generative and discriminative models for Client A and Client B are as follows:

[0135] S2-1: The two-party generative model consists of two ResNet residual networks and a fully connected layer. Its input includes input data and a conditional vector, and both parties process the input data as follows:

[0136] (1) Client A's mask vector Client B Client A and Client B d When M = 0, it means that data at that position is missing; when M = 0, it means that data at that position is missing. d =1 indicates that the data is complete; the dimension of the element with value 1 in each party M is ||M||1, while the dimension of the element with value 0 is ||1-M||1. For example, in party A M, the dimension of the element with value 1 is ||M||1. A ||1, and so on; when all elements in both sides M are 0, it means that only the sample category is used as a condition.

[0137] The encoding vector X of the feature data of the real sample on client A's side A Client A simulates missing sample data vectors under different scenarios.

[0138]

[0139] Right now

[0140]

[0141] The encoding vector X of the feature data of client B B Client B simulates missing sample data vectors under different scenarios.

[0142]

[0143] Right now

[0144]

[0145] Where "⊙" represents element-wise multiplication between vectors;

[0146] (2) Client A and Client B respectively tested the noise sample Z A and noise sample Z B Encode the data and use the encoded results to fill in the missing sample data to obtain Z. A′ and Z B′ The process of client A is represented as follows: The process of client B is represented as follows

[0147] (3) Client A will use the condition vector cond A With Z A′By splicing Client B will use the condition vector cond B With Z B′ By splicing As input to the generative models of both parties, This represents a vector concatenation operation;

[0148] S2-2: The discriminant model for Client A and Client B consists of three fully connected networks that process the generated data of the generative model to meet the input requirements of the discriminant model.

[0149] S2-3: Client A will generate data. Missing data With mask vector M A Element-wise multiplication is used to obtain the input data for the discriminant model. To represent, that is and Client B will generate data Missing data With mask vector M B Element-wise multiplication is used to obtain the input data for the discriminant model. To represent, that is and

[0150] S3: The following loss function was designed for the vertical federated learning scenario.

[0151] The steps for designing the objective function of the discriminant model are as follows:

[0152] S3-1: m represents the number of samples, and u represents the dimension of the encoded samples. Indicates sample X imp The given probability is satisfied; when training the discriminant model for Client A and Client B, the discriminant model must not only determine whether the generated data conforms to the data distribution of the real samples of both parties, that is, the degree of similarity between the generated elements and the real elements under the action of (1-M), but also determine whether the jointly generated samples of both parties satisfy the category label; for each sample of Client A, by calculating and real sample X A Given the condition cond A The cross-entropy loss is used to determine whether a sample matches the category. For each sample from customer B, the cross-entropy loss is calculated... and real sample X B Given the condition cond B The cross-entropy loss is used to determine whether a sample matches the category. In this case, the loss for client A is calculated as follows:

[0153]

[0154] The losses of client B are calculated as follows:

[0155]

[0156] S3-2: Client A and Client B use the Wasserstein distance from Wasserstein GANs to calculate X. imp Similarity of the missing parts between ⊙(1-M) and X⊙(1-M), Client A and They represent Set and X A Set Obedience Probability distribution; Client B and They represent Set and X B Set Obedience Probability distribution, where "γ" and "∈" are The hyperparameters; its loss is expressed as L. w This indicates that the Wasserstein distance for A at this point is calculated as follows:

[0157]

[0158] The Wasserstein distance for B is calculated as follows:

[0159]

[0160] in This represents the "gradient penalty" mechanism in the Wasserstein distance;

[0161] S3-3: At this point, the total discriminant model loss for client A is expressed as:

[0162]

[0163] The total loss of the discriminant model for Client B is:

[0164]

[0165] Where “δ” is a hyperparameter; D(X|cond) means inputting sample X and condition vector cond into the discriminant model; where m represents the number of samples and u represents the dimension of the encoded sample, which can be specific to A and B, such as uB representing the data dimension of the encoded sample in B.

[0166] An optional δ setting is 0.5;

[0167] S3-4: Considering that Client A and Client B are performing vertical federated learning, and although their data features are different, their generated data are correlated, the discriminant model losses of Client A and Client B should be clustered. The total loss for both parties is calculated as follows: As the loss of the two-party discriminant model;

[0168] The steps for designing the objective function of the generative model are as follows:

[0169] S3-5: Client A and Client B generate models to determine whether the generated data can fool the discriminant model, and the data distribution fit between the generated elements and the elements of the real samples under the influence of each party M. Client A is represented as... The larger the value, the more it indicates The more it meets the given conditions; Client B is represented as The larger the value, the more it indicates The more it meets the given conditions, the better, where i represents the i-th sample in the sample set, i = 1...m;

[0170] S3-6: Client A and Client B also need to calculate M⊙X miss and The mean squared error of each element in each sample, whose loss function is... The mean squared error loss for customer A is calculated as follows:

[0171]

[0172] The mean square error loss for customer B is calculated as follows:

[0173]

[0174] Optionally, α can be set to 0.5 for both customer A and customer B;

[0175] S3-7: At this point, the loss of the client A's generator model is expressed as:

[0176]

[0177] The loss of the model generated by client B is:

[0178]

[0179] S3-8: In this scenario, it is necessary to consider the correlation between the missing and non-missing parts of the data from both sides, as well as to determine whether the entire data set conforms to the category. Therefore, the losses of the two generative models need to be clustered.

[0180] S4: Considering that homomorphic encryption is used to encrypt intermediate information to protect the data privacy of client A and client B in federated learning, the training process of DCTGAN, which is oriented towards vertical federated learning, is designed as follows:

[0181] S4-1: Both parties use the method in step S1 to align encrypted entities.

[0182] S4-2: Collaborator P initializes the network parameters for clients A and B and sends them to clients A and B; Collaborator P generates a homomorphic encryption public key p. k Private key s k and the public key p k Send to both Client A and Client B; optionally, Paillier homomorphic encryption is used. Use [[a]] to indicate that data a is homomorphically encrypted.

[0183] S4-3: Client A and Client B respectively analyze the noise sample Z. A and noise sample Z B Encode the data and use the encoded results to fill in the missing sample data to obtain Z. A′ and Z B′ The process of client A is represented as follows: The process of client B is represented as follows Client A and Client B will each give Z A′ and Z B′ Each input data in the dataset is associated with its respective condition vector cond. A ,cond B Concatenate the two inputs to obtain the inputs to the generative models from both sides, represented as follows:

[0184] S4-4: Client A and Client B first fix their own generation models and train their own discrimination models; both parties sample X from m real samples. A X B and The loss is obtained by training the discriminative model using the input as input.

[0185]

[0186] Similarly, Party B calculated the loss.

[0187]

[0188] Both parties will exchange and aggregate the encrypted data to obtain... Party B will send the total loss of the discrimination model to Party P, the collaborating party.

[0189] Optionally, δ = 0.5, the total number of samples in the real sample set and the noise sample set can be set to 3000, and the total sample dimension after two-way encoding is 157.

[0190] Optionally, the batch size m can be set to 10.

[0191] S4-5: A and B are optimized using the Adam optimizer, and the gradient of parameter A is calculated as follows:

[0192]

[0193] The B-square gradient is calculated as follows:

[0194]

[0195] Party A adds a random mask to the gradient information of the discriminant model to obtain... Party B adds a random mask to the discriminant information to obtain... Both parties will send the gradient information to collaborating party P for decryption. D To determine the learning rate of the model, optionally, the learning rates for client A and client B can be set to lr. D =0.0002

[0196] S4-6: Collaborator P decrypts the gradients of clients A and B. Collaborator P decrypts the gradient information and sends the result back to clients A and B. Clients A and B remove the mask on the gradient information and obtain the true gradient information to update their respective discrimination model networks.

[0197] S4-7: Client A and Client B obtain the decryption gradient information and perform backpropagation. The backpropagation process is as follows:

[0198] The discriminant model network for Client A and Client B is L. D If the layer is l, then the gradient update process of the l-th layer is as follows:

[0199]

[0200] Where Φ D Let K represent the weight matrix. D Represents the bias matrix. This represents the differentiation of the weights of the l-th layer. Indicates the Lth D Input for layer +1, Indicates the Lth D Differentiate the layer output. This represents the differentiation of the LeakyReLu activation function. This represents the input of the Lth layer. Indicates the Lth D Differentiate the output of layer -1. This represents the input of the (l+2)th layer. This represents the differentiation of the output of the (l+1)th layer. This represents the input of the (l+1)th layer;

[0201] The parameter update process for layer l is as follows:

[0202]

[0203] in and These represent the weights and biases of the l-th layer of the discriminant model network, respectively.

[0204] Both Party A and Party B shall perform the calculations according to the above process.

[0205] S4-8: Client A and Client B fix their own discrimination models and train their own generation models; both A and B sample m Z samples. ′ Z ′ This represents the result of filling missing sample data with encoded noise samples, which is then input into a two-way generative model to obtain m generated data. Indicates that data will be generated. Missing data X miss The input data for the discriminative model is obtained by performing element-wise multiplication with the mask vector M, and the result is represented by X. imp To represent, that is get Input into the discriminant model to determine X imp Whether the given conditions are met also needs to be calculated in relation to X. miss Mean square error loss between The losses suffered by Client A are as follows:

[0206]

[0207] Party B's losses are:

[0208]

[0209] Client A and Client B encrypt the calculated generative model loss. Both parties exchange mean squared error losses and converge to obtain Party B will send the total loss of the generated model to collaborating party P.

[0210] S4-9: Client A and Client B use the Adam optimizer to optimize and update parameter lr. G The learning rate for the generative model; the formula for calculating the gradient of the A-squared is:

[0211]

[0212] The formula for calculating the B-square gradient is:

[0213]

[0214] Both parties added a random mask to the encryption loss to obtain... and Both parties will send the gradient information to collaborating party P for decryption; lr G Optionally, for the learning rate of the generative model, both Client A and Client B can set the learning rate to lr. G =0.0002.

[0215] S4-10: Collaborator P decrypts the gradient information and sends the result back to clients A and B. Clients A and B remove the mask on the gradient information and obtain the true gradient information to update their respective generated model networks.

[0216] S4-11: Client A and Client B obtain the decrypted generator model gradient information and perform backpropagation. The backpropagation process is as follows:

[0217] During the reverse propagation process, the generator networks of client A and client B are L. G If the layer is l, then the gradient update process of the l-th layer is as follows:

[0218]

[0219] Where Φ G Let B represent the weight matrix. G Represents the bias matrix. This represents the differentiation of the weights of the l-th layer. L represents G Input for layer +1, Indicates the Lth G Differentiate the layer output. This represents the input of the Lth layer. Indicates the Lth G Differentiate the output of layer -1. This represents the input of the (l+2)th layer. This represents the differentiation of the output of the (l+1)th layer. This represents the input of the (l+1)th layer;

[0220] The parameter update process for layer 1 is as follows:

[0221]

[0222] in and These represent the weights and biases of the l-th layer of the generative network, respectively.

[0223] Both Party A and Party B shall perform the calculations according to the above process.

[0224] S4-12: Collaborating party P will calculate the total loss of the models generated by both parties. and the total loss of the discriminant model The process involves convergence and iterative training until the model converges.

[0225] S5: The generation of multi-party, multi-type data in multi-party loan risk assessment applications includes the following three methods:

[0226] Option 1: Use a pre-trained longitudinal federated learning dual-conditional generative adversarial network model to generate and fill in missing data from multiple parties in loan risk assessment.

[0227] For example, in a multi-party loan risk assessment, a sample has 128 dimensions of data before encoding by both parties and 157 dimensions after encoding. Party A has 20 missing dimensions, and Party B has 30 missing dimensions. Based on the sample's category conditions, such as high risk [1,0,0], and the total 107 existing dimensions (67 for Party A and 40 for Party B), corresponding M-vectors for each party can be created. The 20 missing dimensions in Party A's M-vector and the 30 missing dimensions in Party B's M-vector are set to 0. A pre-trained longitudinal federated learning dual-conditional generative adversarial network is then used to generate and fill in the 50 missing dimensions of the high-risk sample from both parties.

[0228] Option 2: Generate local data through the local data generation model of each party in the loan risk assessment as part of the existing data in the "longitudinal federated learning of dual-condition generative adversarial network model", and then combine it with category samples to generate sample data under dual conditions;

[0229] For example, in the above multi-party loan risk assessment, a local data generation model is used to generate 20-dimensional data for Party A and 30-dimensional data for Party B, which serves as existing partial data and as one of the conditions for the dual conditions of each party. Optionally, the local data generation model can be a Conditional Generative Adversarial Network (CTGAN) or a Tabular Data Generative Adversarial Network (TGAN) model. Then, according to the sample generation requirements of different condition categories, following the process of "Scheme One," the two parties jointly generate a sample that satisfies a certain category condition.

[0230] Option 3: By clearing the mask vector in the dual-condition generative adversarial network sub-model of each party in the longitudinal federated learning, only sample data with the loan risk assessment sample category as the condition is generated.

[0231] For example, in the above multi-party loan risk assessment, all 157 dimensions of data from both parties are replaced with noisy samples, and the combined 157 dimensions of M from parties A and B are all set to 0. To generate a high-risk [1,0,0] sample, a newly generated high-risk sample based on both parties is generated by using a pre-trained longitudinal federated learning dual-conditional generative adversarial network. Noisy samples, M vectors, and sample condition vectors from each party are input into each party's sub-network.

[0232] Solution 1 aims to address the issue of generating and imputing missing data in vertical federal multi-party loan risk assessment, with the expectation of obtaining a high-quality aligned sample dataset, such as... Figure 2 Examples ②, ③, and ④ are shown; Schemes two and three are designed to address the imbalance in sample size between categories after vertical federal alignment in loan risk assessment, such as... Figure 2 The example shows case ⑤.

[0233] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for generating multi-party, multi-class data oriented to vertical federated learning, applied to a loan risk assessment system, characterized by: The method includes the following steps: S1: Establish a training dataset for a longitudinal federated learning table for the purpose of generating multi-party, multi-class data models; wherein, the customer A dataset contains personal loans, repayments, defaults, and risk assessment levels, and the customer B dataset contains age, annual income, household debt, and risk assessment levels. S2: Constructing a dual-conditional generative adversarial network (DCTGAN) for longitudinal federated learning; The specific structure of DCTGAN: The generative model consists of two ResNet residual networks and one fully connected layer, and the discriminative model consists of three fully connected networks. S3: Design a multi-party loss function for a dual-condition generative adversarial network oriented towards vertical federation; S4: Establishing the training process for a dual-condition generative adversarial network oriented towards vertical federated learning; S5: For tabular datasets, design and implement a multi-party, multi-class data generation scheme based on dual-condition generative adversarial networks, realize the method of jointly expanding the feature dimension and sample quantity of samples in tabular datasets by multiple parties, and jointly construct a high-quality tabular training dataset to meet the needs of machine learning model training in big data application scenarios. Specifically, S1 is: Customer A and Customer B have user groups belonging to different systems. Both parties use RSA encryption and hash functions to achieve sample alignment and represent the sample ID of the party with completely missing data. Specifically, S2 is: Client A and Client B construct a dual-condition generative adversarial network, with each party having a generative adversarial subnetwork, which includes a generative model and a discriminative model respectively. S2-1: The two-party generative model consists of two ResNet residual networks and a fully connected layer. Its input includes input data and a conditional vector, and both parties process the input data as follows: (1) Client A's mask vector Client B Client A and Client B d When M = 0, it means that data at that position is missing; when M = 0, it means that data at that position is missing. d =1 indicates that the data is complete; the dimension of the element value 1 in each party M is ||M||1, while the dimension of the element value 0 is ||1-M||1; when all elements in both parties M are 0, it means that only the sample category is used as a condition. The encoding vector X of the feature data of the real sample on client A's side A Client A simulates missing sample data vectors under different scenarios. Right now The encoding vector X of the feature data of client B B Client B simulates missing sample data vectors under different scenarios. Right now Here, "⊙" represents element-wise multiplication between vectors; (2) Client A and Client B respectively tested the noise sample Z A and noise sample Z B Encode the data and use the encoded results to fill in the missing sample data to obtain Z. A' and Z B' The process of client A is represented as follows: The process of client B is represented as follows (3) Client A will use the condition vector cond A With Z A' Z is obtained by splicing. A' ⊕cond A Client B will use the condition vector cond B With Z B' Z is obtained by splicing. B' ⊕cond B As input to the two generative models, "⊕" represents a vector concatenation operation; S2-2: The discriminant model for Client A and Client B consists of three fully connected networks that process the generated data of the generative model to meet the input requirements of the discriminant model. S2-3: Client A will generate data. Missing data With mask vector M A Element-wise multiplication is used to obtain the input data for the discriminant model. To represent, that is and Client B will generate data Missing data With mask vector M B Element-wise multiplication is used to obtain the input data for the discriminant model. To represent, that is and Specifically, S5 is: The data loss situations for Client A and Client B differ and can be categorized into three scenarios: (1) The feature data of the sample in both customer A and customer B have some missing values. Refer to step S2 to fill the missing data with random noise. Refer to step S4 to train the longitudinal federated learning DCTGAN. After both parties have finished training, input random noise, known data and corresponding condition vectors into the generator model to generate samples. Fill the missing data of the generated samples into the real data. (2) The data features of the sample in either customer A or customer B are missing. Create a mask vector completely filled with "0". Fill the missing data with random noise according to the steps in S2. Train the longitudinal federated learning DCTGAN according to the steps in S4. After both parties have finished training, input the condition vector corresponding to the random noise into the generator model to generate samples. Fill the missing data of the generated samples into the real data. (3) There is a situation of imbalance in the sample categories when the samples are aligned by multiple parties: randomly fill the mask vector of the real data with 0, fill the missing data with random noise according to the steps in S2, and train the longitudinal federated learning DCTGAN according to the steps in S4. After both parties have finished training, concatenate the random noise with the minority class label and input it into the generative model to generate minority class sample filling data; or use the existing data to generate the model locally on each party, generate some existing data as one of the two conditions, and then use the above-mentioned scheme to generate the missing data, so as to generate minority class samples that meet a certain category condition.

2. The method for generating multi-party, multi-class data for vertical federated learning according to claim 1, characterized in that: Specifically, S3 is: The steps for designing the objective function of the discriminant model are as follows: S3-1: m represents the number of samples, and u represents the dimension of the encoded samples. Indicates sample X imp The given probability is satisfied; when training the discriminant model for Client A and Client B, the discriminant model must not only determine whether the generated data conforms to the data distribution of the real samples of both parties, that is, the degree of similarity between the generated elements and the real elements under the action of (1-M), but also determine whether the jointly generated samples of both parties satisfy the category label; for each sample of Client A, by calculating and real sample X A Given condition cond A The cross-entropy loss is used to determine whether a sample matches the category. For each sample from customer B, the cross-entropy loss is calculated... and real sample X B Given condition cond B The cross-entropy loss is used to determine whether a sample matches a category. In this case, the loss for client A is calculated as follows: The losses of client B are calculated as follows: S3-2: Client A and Client B use the Wasserstein distance from Wasserstein GANs to calculate X. imp Similarity of the missing parts between ⊙(1-M) and X⊙(1-M), Client A and They represent Set and X A Set Obedience Probability distribution; Client B and They represent Set and X B Set Obedience Probability distribution, "γ" and "∈" are The hyperparameters; its loss is expressed as L. w This indicates that the Wasserstein distance for A at this point is calculated as follows: The Wasserstein distance for B is calculated as follows: in This represents the "gradient penalty" mechanism in the Wasserstein distance; S3-3: At this point, the total discriminant model loss for client A is expressed as: The total loss of the discriminant model for Client B is: Where "δ" is a hyperparameter; D(X|cond) means inputting sample X and condition vector cond into the discriminant model; where m represents the number of samples and u represents the dimension of the encoded sample, specifically for A and B sides, such as uB representing the data dimension of the encoded sample in B side; S3-4: Aggregate the losses of the discriminant models for both A and B; the total loss for both sides is calculated as follows: As the loss of the two-party discriminant model; The steps for designing the objective function of the generative model are as follows: S3-5: Client A and Client B generate models to determine whether the generated data can fool the discriminant model, and the data distribution fit between the generated elements and the elements of the real samples under the influence of each party M. Client A is represented as... The larger the value, the more it indicates The more it meets the given conditions; Client B is represented as The larger the value, the more it indicates The more it meets the given conditions, the better, where i represents the i-th sample in the sample set, i = 1...m; S3-6: Client A and Client B also need to calculate M⊙X miss and The mean squared error of each element in each sample, whose loss function is... The mean squared error loss for customer A is calculated as follows: The mean square error loss for customer B is calculated as follows: Where α is a hyperparameter; S3-7: The loss of the model generated by client A is expressed as: The loss of the model generated by client B is: S3-8: Aggregate the losses of the two generative models, i.e. Specifically, S4 is: S4: Considering that homomorphic encryption is used to encrypt intermediate information to protect the data privacy of clients A and B in federated learning, the training process of DCTGAN, which is designed for vertical federated learning, is as follows: S4-1: Both parties use the method in step S1 to align encrypted entities; S4-2: Collaborator P initializes the network parameters for clients A and B and sends them to clients A and B; Collaborator P generates a homomorphic encryption public key p. k Private key s k and the public key p k Send to both Client A and Client B; the homomorphic encryption scheme is Paillier homomorphic encryption; S4-3: Client A and Client B respectively analyze the noise sample Z. A and noise sample Z B Encode the data and use the encoded results to fill in the missing sample data to obtain Z. A' and Z B' The process of client A is represented as follows: The process of client B is represented as follows Client A and Client B will each give Z A' and Z B' Each input data in the dataset is associated with its respective condition vector cond. A ,cond B The inputs to both generative models are concatenated and denoted as Z. A' ⊕cond A Z B' ⊕cond B ; S4-4: Client A and Client B first fix their own generation models and train their own discrimination models; both parties sample X from m real samples. A X B and The loss is obtained by training the discriminative model using the input as input. Similarly, Party B calculated the loss. Both parties will exchange and aggregate the encrypted data to obtain... Party B will send the total loss of the discrimination model to Party P, the collaborating party. S4-5: A and B are optimized using the Adam optimizer, and the gradient of parameter A is calculated as follows: The B-square gradient is calculated as follows: Party A adds a random mask to the gradient information of the discriminant model to obtain... Party B adds a random mask to the discriminant gradient information to obtain... Both parties will send the gradient information to collaborating party P for decryption; S4-6: Collaborator P decrypts the gradients of A and B. Collaborator P decrypts the gradient information and sends the result back to A and B. A and B remove the mask on the gradient information and obtain the true gradient information to update their respective discrimination model networks. S4-7: Client A and Client B obtain the decryption gradient information and perform backpropagation. The backpropagation process is as follows: The discriminant model network for Client A and Client B is L. D If the layer is l, then the gradient update process of the l-th layer is as follows: Where Φ D Let K represent the weight matrix. D Represents the bias matrix. Indicates the first Differentiate the weights of the layers. Indicates the Lth D Input for layer +1, Indicates the Lth D Differentiate the layer output. This represents the differentiation of the LeakyReLu activation function. This represents the input of the Lth layer. Indicates the Lth D Differentiate the output of layer -1. Indicates the first Layer input, Indicates the first Differentiate the layer output. Indicates the first Layer input; No. The parameter update process of the layer is as follows: in and These represent the discriminant model network's first and second generations, respectively. Layer weights and biases; Both Party A and Party B shall perform the calculations according to the above process; S4-8: Client A and Client B fix their own discrimination models and train their own generative models; A and B sample m Z', where Z' represents the result of filling missing sample data with encoded noise samples, and input them into their respective generative models to obtain m generated data. Indicates that data will be generated. Missing data X miss The input data for the discriminative model is obtained by performing element-wise multiplication with the mask vector M, and the result is represented by X. imp To represent, that is get Input into the discriminant model to determine X imp Whether the given conditions are met also needs to be calculated in relation to X. miss Mean square error loss between The losses suffered by Client A are as follows: Party B's losses are: Client A and Client B encrypt the calculated generative model loss. Both parties exchange mean squared error losses and converge to obtain Party B will send the total loss of the generated model to collaborating party P. S4-9: Client A and Client B use the Adam optimizer to optimize and update parameters, lr G The learning rate for the generative model; the formula for calculating the gradient of the A-squared is: The formula for calculating the B-square gradient is: Both parties added a random mask to the encryption loss to obtain... and Both parties will send the gradient information to collaborating party P for decryption; S4-10: Collaborator P decrypts the gradient information and sends the result back to clients A and B. Clients A and B remove the mask on the gradient information and obtain the real gradient information to update their respective generated model networks. S4-11: Client A and Client B obtain the decrypted generator model gradient information and perform backpropagation. The backpropagation process is as follows: During the reverse propagation process, the generator networks of client A and client B are L. G If the layer is l, then the gradient update process of the l-th layer is as follows: Where Φ G Let K represent the weight matrix. G Represents the bias matrix. Indicates the first Differentiate the weights of the layers. L represents G Input for layer +1, Indicates the Lth G Differentiate the layer output. This represents the input of the Lth layer. Indicates the Lth G Differentiate the output of layer -1. Indicates the first Layer input, Indicates the first Differentiate the layer output. Indicates the first Layer input; The parameter update process for layer l is as follows: in and These represent the first generation of the birth chart network. Layer weights and biases; Both Party A and Party B shall perform the calculations according to the above process; S4-12: Collaborating party P will aggregate the total loss of the generated models and the total loss of the discriminant models of both parties. When the loss is less than a certain value, global training will terminate. If the loss still does not converge, the above steps will be repeated until the losses of both parties converge.

3. A computer system comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, it implements the method as described in any one of claims 1 to 2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 2.

Images