Longitudinal federated learning model training method based on privacy protection and split learning

Abstract

Embodiments of the present specification provide a longitudinal federated learning model training method based on privacy protection and split learning. A device of a first user stores feature data of n first features and first label data of a plurality of samples corresponding to a target model respectively, and is responsible for training a first model and a top model, the first model comprising n first feature selection parameters corresponding to the n first features. A device of a second user stores feature data of m second features of the plurality of samples respectively, and is responsible for training a second model, the second model comprising m second feature selection parameters corresponding to the m second features. The device of the second user provides an intermediate result output by the second model to the device of the first user for training the top model. The device of the first user calculates gradient values corresponding to the first model and the second model respectively, updates the n first feature selection parameters based on the gradient value corresponding to the first model, and sends the gradient value corresponding to the second model to the device of the second user.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of computer technology, and in particular relate to a training method for a vertical federated learning model based on privacy protection and split learning. Background Technology

[0002] Artificial intelligence has been applied to various industries, demonstrating its advantages in various fields. However, the development of artificial intelligence still faces some challenges. For example, most fields have limited data or poor data quality, which is very detrimental to artificial intelligence, which is extremely dependent on data. Due to concerns about data privacy, many data owners are unwilling to share data, creating "data silos" that prevent data from being used effectively. Summary of the Invention

[0003] The purpose of this invention is to provide a training scheme for a vertical federated learning model based on privacy protection and split learning, which enables multiple participants with the same sample space but different feature spaces to collaboratively train the model, thereby ensuring data security and preventing privacy leakage as much as possible, and reducing computational overhead, thus effectively improving model training efficiency.

[0004] This specification provides a method for training a longitudinal federated learning model based on privacy protection and split learning, executed by a first user's device. This device stores first label data and n first feature data for each of multiple samples corresponding to a target model. The model training also includes a second user whose device stores m second feature data for each of the multiple samples. The method comprises: inputting the feature data of n first features for each of k samples from the multiple samples into a local first model; the first model selecting the n first features based on the values ​​of n first feature selection parameters corresponding to the n first features, obtaining s first feature data for each of the k samples; and then, based on the values ​​of the m first features... The feature data of each of the s first features of each sample are calculated to obtain k first intermediate results; k second intermediate results are received from the second user's device, which are obtained by the second user's device by inputting the feature data of each of the m second features of the k samples into a local second model; the aggregated result of the k first intermediate results and the k second intermediate results is input into the local top-level model to obtain k model outputs; the gradient values ​​corresponding to the first model and the second model are calculated based on the k model outputs and the first label data of each of the k samples; the n first feature selection parameters are updated based on the gradient values ​​corresponding to the first model, and the gradient values ​​corresponding to the second model are sent to the second user's device.

[0005] The second aspect of this specification provides a method for training a longitudinal federated learning model based on privacy protection and split learning, executed by a second user's device. This device stores feature data of m second features for each of multiple samples corresponding to a target model. The model training also includes a first user whose device stores first label data and n first feature data for each of the multiple samples. The method includes: inputting the feature data of m second features for each of k samples from the multiple samples into a local second model; the second model selecting the m second features based on the values ​​of m second feature selection parameters corresponding to the m second features, obtaining feature data of t second features for each of the k samples; and then, based on the... The feature data of each of the k samples' t second features are calculated to obtain k second intermediate results; the k second intermediate results are sent to the first user's device; the gradient value corresponding to the second model is received from the first user's device, which is calculated by the first user's device based on the k model outputs and the first label data of each of the k samples after inputting the aggregated result of the k first intermediate results and the k second intermediate results into the local top-level model to obtain k model outputs; the k first intermediate results are obtained by the first user's device by inputting the feature data of each of the k samples' n first features into the local first model; the m second feature selection parameters are updated based on the gradient value corresponding to the second model.

[0006] A third aspect of this specification provides a training apparatus for a longitudinal federated learning model based on privacy protection and split learning, applied to a first user's device. This device stores first label data and n first feature data for each of multiple samples corresponding to a target model. The model training also includes a second user whose device stores m second feature data for each of the multiple samples. The apparatus includes a model training unit configured to input the feature data of n first features for each of k samples from the multiple samples into a local first model. The first model selects the n first features based on the values ​​of n first feature selection parameters corresponding to the n first features, obtaining s first feature data for each of the k samples, and then selects the feature data based on the s first features for each of the k samples. The feature data of the features are calculated to obtain k first intermediate results; the receiving unit is configured to receive k second intermediate results from the second user's device, wherein the k second intermediate results are obtained by the second user's device by inputting the feature data of m second features of each of the k samples into a local second model; the model training unit is further configured to input the aggregated result of the k first intermediate results and the k second intermediate results into a local top-level model to obtain k model outputs; the gradient values ​​corresponding to the first model and the second model are calculated based on the k model outputs and the first label data of each of the k samples; the n first feature selection parameters are updated based on the gradient values ​​corresponding to the first model; and the sending unit is configured to send the gradient values ​​corresponding to the second model to the second user's device.

[0007] This specification provides a training apparatus for a longitudinal federated learning model based on privacy protection and split learning, applied to a second user's device. The device stores feature data of m second features for each of multiple samples corresponding to a target model. The model training also includes a first user whose device stores first label data and n first feature data for each of the multiple samples. The apparatus includes a model training unit configured to input the feature data of m second features for each of k samples from the multiple samples into a local second model. The second model selects the m second features based on the values ​​of m second feature selection parameters corresponding to the m second features, obtaining feature data of t second features for each of the k samples, and then selects the feature data of t second features for each of the k samples based on the values ​​of the t first features. The feature data of the two features are calculated to obtain k second intermediate results; the sending unit is configured to send the k second intermediate results to the device of the first user; the receiving unit is configured to receive the gradient value corresponding to the second model from the device of the first user. The gradient value is calculated by the device of the first user based on the k model outputs and the first label data of the k samples after inputting the aggregated result of the k first intermediate results and the k second intermediate results into the local top-level model to obtain k model outputs. The k first intermediate results are obtained by the device of the first user inputting the feature data of the n first features of the k samples into the local first model; the model training unit is further configured to update the m second feature selection parameters based on the gradient value corresponding to the second model.

[0008] The fifth aspect of this specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in either the first or second aspect.

[0009] A sixth aspect of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in either the first or second aspect.

[0010] The seventh aspect of this specification provides a computer program product that, when executed in a computer, causes the computer to perform the method described in either the first or second aspect.

[0011] The solutions provided by the embodiments described above enable multiple participants with the same sample space but different feature spaces to collaboratively train the model, ensuring that feature data does not leave the domain, thereby maximizing data security and preventing privacy leaks. Furthermore, both the first and second models have feature selection parameters, which can be adjusted during model training to optimize feature selection. This allows feature data from several important features of k samples to participate in the calculation of intermediate results, avoiding the use of feature data from all features of those k samples. This reduces computational overhead and effectively improves model training efficiency. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating one application scenario in which the embodiments of this specification can be applied;

[0014] Figure 2 This is a schematic diagram of the training process of a vertical federated learning model based on privacy protection and split learning in the embodiments of this specification;

[0015] Figure 3 This is a timing diagram of the feature selection parameter initialization process in the embodiments of this specification;

[0016] Figure 4 This is a timeline diagram of a training method for a vertical federated learning model based on privacy protection and split learning in the embodiments of this specification;

[0017] Figure 5 This is a flowchart of the noise-adding process for k second intermediate results in the embodiments of this specification;

[0018] Figure 6 This is a timeline diagram of a training method for a vertical federated learning model based on privacy protection and split learning in the embodiments of this specification;

[0019] Figure 7 This is a schematic diagram of the structure of the training device for the vertical federated learning model based on privacy protection and split learning in the embodiments of this specification;

[0020] Figure 8 This is a schematic diagram of the structure of the training device for the vertical federated learning model based on privacy protection and split learning in the embodiments of this specification. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0022] As mentioned earlier, artificial intelligence has been applied to various industries and has demonstrated its advantages in various fields. However, the development of artificial intelligence still faces some challenges. For example, most fields have limited data or poor data quality, which is very detrimental to artificial intelligence, which is extremely dependent on data. Due to concerns about data privacy, many data owners are unwilling to share data, creating "data silos" that prevent data from being used effectively.

[0023] Federated learning breaks down the "data silos," allowing participants to collaboratively train models without sharing raw data. Depending on the data partitioning, federated learning can be categorized into horizontal federated learning, vertical federated learning, and federated transfer learning. Vertical federated learning generally refers to participants with the same sample space but different feature spaces collaboratively training a model, ensuring that data does not leave its domain. Specifically, participants in vertical federated learning can possess different dimensions of feature data from the same user group; for example, one party might hold credit records for the same user group, while another holds their consumption records. Vertical federated learning primarily expands feature data through sample alignment, thereby improving model performance. In practical applications, because different companies (e.g., banks and supermarkets) hold different feature data for the same user set, vertical federated learning is often more valuable than horizontal federated learning.

[0024] This specification provides a method for training a vertical federated learning model based on privacy protection and split learning. This method enables multiple participants with the same sample space but different feature spaces to collaboratively train the model, thereby maximizing data security, preventing privacy leaks, and reducing computational overhead, thus effectively improving model training efficiency.

[0025] See Figure 1 This is a schematic diagram illustrating one application scenario to which the embodiments of this specification can be applied. Figure 1 The application scenario shown can include device 101 of user A and device 102 of user B. Devices 101 and 102 can collaboratively train the target model. Either device 101 or device 102 can be a terminal device or a server.

[0026] like Figure 1 As shown, device 101 stores first label data and n first feature data for each of multiple samples corresponding to the target model, and device 102 stores feature data for m second features for each of the multiple samples. Here, n and m can be natural numbers greater than 1. In one example, both the first and second features can be referred to as dimensions or attributes. The first label data for each of the multiple samples can be obtained by one-hot encoding the category labels to which each of the multiple samples belongs among several category labels. Taking user A and user B as examples... Figure 2 The examples shown, Bank A and Internet Company B, represent multiple samples of multiple customers. The n first features could include, for example, deposit and loan status, investment activities, etc., and the m second features could include, for example, online behavior, search preferences, etc., along with several category labels (e.g., ...). Figure 2 The numbers 2, 1, 9, and 0 (illustrated in the diagram) can, for example, be used to reflect a customer's loan repayment status. Among them, Figure 2 This is a schematic diagram of the training process of a vertical federated learning model based on privacy protection and split learning in the embodiments of this specification.

[0027] It should be noted that the aforementioned multiple samples can be determined by devices 101 and 102 through sample alignment. In one embodiment, devices 101 and 102 each store a set of samples. During sample alignment, devices 101 and 102 can perform Private Set Intersection (PSI) on each set of samples, identifying at least a portion of the samples in the resulting intersection as the aforementioned multiple samples. Private Set Intersection is a cryptographic technique in secure multi-party computation, allowing the two parties involved in the computation to calculate the intersection of their data without obtaining additional information from each other (information other than the intersection). It should be noted that sample alignment technology is a well-known technique in the art. Besides using Private Set Intersection to achieve sample alignment, other well-known techniques can also be used, which will not be elaborated upon here.

[0028] The target model can be divided into a bottom-level model and a top-level model. For example... Figure 1As shown, the top-level model can reside in device 101, and the bottom-level model can reside in both device 101 and device 102. For example, the bottom-level model may be specifically contained in a first model local to device 101 and a second model local to device 102. The first model may include n first feature selection parameters corresponding to n first features, and the second model may include m second feature selection parameters corresponding to m second features. The values ​​of the feature selection parameters can be used by the model to perform embedded feature selection on the input feature data. Embedded feature selection means that the feature selection process is integrated with the learner training process, and both are completed in the same optimization process; that is, feature selection is automatically performed during learner training.

[0029] Before starting model training, devices 101 and 102 can be as follows: Figure 2 As indicated by reference numeral ①, the feature selection parameters are initialized. For example, after obtaining the first label data for each of the multiple samples by performing one-hot encoding on the category labels to which each sample belongs in several category labels, device 101 can perturb the first label data to obtain the second label data for each of the multiple samples, and then send the second label data to device 102. Afterwards, device 101 can use the Gini impurity algorithm to initialize the n first feature selection parameters based on the first label data and the feature data of n first features. Device 102 can use the Gini impurity algorithm to initialize the m second feature selection parameters based on the second label data and the feature data of m second features.

[0030] After initializing the feature selection parameters, devices 101 and 102 can begin model training. During model training, device 101 can input the feature data of n first features of each of the k samples from the plurality of samples into the first model, and device 102 can input the feature data of m second features of each of the k samples into the second model. The first model and the second model can be trained as follows: Figure 2 As shown by label ②, the input feature data is based on the values ​​of their respective feature selection parameters. Figure 2 Embedded feature selection is performed on the original data (illustrated in the diagram), and the feature data of the selected features are calculated, for example, by the included underlying model, to obtain intermediate results.

[0031] Furthermore, such as Figure 1As shown, the first model may further include a first feature selection model, which includes n first feature selection parameters; the second model may further include a second feature selection model, which includes m second feature selection parameters. Both the first and second feature selection models can perform feature selection on the input feature data based on the values ​​of the included feature selection parameters, obtaining feature data of the selected features. This selected feature data serves as the input to the corresponding underlying model, enabling the underlying model to perform calculations based on the selected feature data to obtain intermediate results.

[0032] Next, devices 101 and 102 can be as follows: Figure 2 As indicated by label ③, information exchange occurs. For example, device 102 can provide the intermediate results output by the second model to device 101 for training the top-level model. Device 101 can aggregate the intermediate results output by the first model and the intermediate results provided by device 102, input the aggregated results into its local top-level model to obtain the model output, and then calculate the gradient value corresponding to the first model and the gradient value corresponding to the second model (e.g., the gradient value corresponding to the intermediate results provided by device 102) based on the model output and the first label data of the k samples mentioned above. Then, based on the gradient value corresponding to the first model, n first feature selection parameters are updated, and the gradient value corresponding to the second model is sent to device 102 to update m second feature selection parameters.

[0033] It should be noted that, to ensure the security of transmitted data, avoid privacy leaks, and reduce computational overhead, thereby effectively improving model training efficiency, devices 101 and 102 can, for example, perform information exchange based on differential privacy. For instance, device 102 can use a localized differential privacy mechanism to add noise to the intermediate results output by the second model, and then provide the noisy intermediate results to device 101. Similarly, device 102 can use a localized differential privacy mechanism to add noise to the gradient values ​​corresponding to the second model, and then provide the noisy gradient values ​​to device 102.

[0034] Below, let's first combine Figure 3 This section describes the feature selection parameter initialization process. Figure 3 This is a timing diagram of the feature selection parameter initialization process in the embodiments of this specification.

[0035] like Figure 3 As shown, in step S301, user A's device 101 perturbs the first label data of each of the multiple samples to obtain the second label data of each of the multiple samples. The second label data meets the requirements of one-hot encoding.

[0036] Specifically, device 101 can perturb each element in the first label data of each of the plurality of samples according to a preset perturbation rule to obtain the perturbed label data of each of the plurality of samples. The preset perturbation rule may, for example, include making 1 in the first label data change to 0 with probability (1-p), and 0 change to 1 with probability q; wherein, ∈ can represent the privacy budget. Then, for each of the above samples, if the perturbation label data of that sample meets the requirements of one-hot encoding, then device 101 can determine the perturbation label data as the second label data of that sample. If the perturbation label data does not meet the requirements of one-hot encoding, then device 101 can... Figure 2 As shown, the perturbation label data is modified to obtain the second label data of the sample. In one implementation method, when modifying the perturbation label data, if the perturbation label data includes multiple 1s, the first 1 can be retained, and the others set to 0; if all elements in the perturbation label data are 0, the elements in the first column can be set to 1.

[0037] In step S303, device 101 sends the second tag data of each of the above-mentioned samples to user B's device 102.

[0038] In step S305, device 101 determines the Gini impurity corresponding to each of the n first features based on the first label data of each of the above multiple samples and the feature data of the n first features.

[0039] Specifically, for each of the n first features, the feature data of each of the n first features of the aforementioned samples can be divided into several subsets based on the different values ​​of the first feature. Then, for each subset, the Gini impurity corresponding to that subset can be determined based on the first label data of the samples corresponding to that subset and the sample size of those samples. The ratio of the sample size of those samples to the sample size of the aforementioned samples is determined as the weight value of the Gini impurity. Finally, the weighted sum of the Gini impurities corresponding to each subset can be obtained to obtain the Gini impurity corresponding to that first feature.

[0040] f i Taking the i-th first feature out of n first features as an example, f can be calculated using the following formula (1). i The corresponding Gini impurity G(f) i ):

[0041]

[0042] Where i∈[n], [n] can represent the set of positive integers {1, 2, ..., n}, and V is fi The number of possible values ​​for , [V] can represent the set of positive integers {1, 2, ..., V}, D v It can represent the v-th subset among the above subsets, |D v | can represent a subset D v The number of samples (i.e., the number corresponding to subset D) v The number of samples in a set D), where D in |D| can represent the set of feature data of each of the n first features of the above samples, and |D| can represent the number of samples in set D (that is, the number of samples in the above samples), Gini(D v ) can represent a subset D v The corresponding impurity of the gin. This should be understood. It can be represented as Gini(D) v The weight value of ).

[0043] Gini(D v It can be calculated using the formula (2) shown below:

[0044]

[0045] Where Y represents the number of category labels as described above, and [Y] can represent the set of positive integers {1, 2, ..., Y}, p y It can be represented from the subset D v The probability of randomly selecting a sample from a set of samples that belongs to the y-th category label.

[0046] p y It can be calculated using the formula (3) shown below:

[0047]

[0048] Among them, I(D) v ) can represent the subset D v Several samples, M v This can represent the first label data of each of the several samples, |M v | a,y This can represent the binary value of sample a under the y-th category label. It should be understood that... This can represent the probability that sample a belongs to the y-th category label.

[0049] It should be noted that for each of the n first features, the smaller the Gini impurity corresponding to the first feature, the more important the feature data of the first feature is; the larger the Gini impurity corresponding to the first feature, the less important the feature data of the first feature is.

[0050] In step S307, device 101 determines the initial values ​​of the n first feature selection parameters based on the Gini impurity corresponding to each of the n first features.

[0051] Continue with f i Taking the i-th first feature out of n first features as an example, it can be represented by the following formula (4), based on f i The corresponding Gini impurity G(f) i ), determine f i The corresponding feature selection parameter μ i Initial value:

[0052]

[0053] in, b can represent μ i and The scaling factor. It should be noted that b can be preset, and b can be greater than 0.

[0054] In step S309, device 102 determines the Gini impurity corresponding to each of the m second features based on the second label data of each of the above multiple samples and the feature data of the m second features.

[0055] Specifically, for each of the m second features, the feature data of each of the m second features of the aforementioned samples can first be divided into several subsets based on the different values ​​of the second feature. Then, for each subset, the Gini impurity corresponding to that subset can be determined based on the second label data of the samples corresponding to that subset and the sample size of those samples. The ratio of the sample size of those samples to the sample size of the aforementioned samples is determined as the weight value of the Gini impurity. Then, the weighted sum of the Gini impurities corresponding to each subset can be obtained to obtain the Gini impurity corresponding to the second feature. It should be noted that the calculation method for the Gini impurity corresponding to the second feature is similar to that for the first feature; refer to the calculation method for the Gini impurity corresponding to the first feature described above, and will not be elaborated further here.

[0056] In step S311, device 102 determines the initial values ​​of the m second feature selection parameters based on the Gini impurity corresponding to each of the m second features.

[0057] Specifically, for each of the m second features, the initial value of the second feature selection parameter corresponding to the second feature can be determined based on the Gini impurity corresponding to the second feature by using a calculation formula similar to the above formula (4).

[0058] exist Figure 3In the corresponding embodiment, device 101 can initialize n first feature selection parameters based on the original label data (first label data) of each of the multiple samples and the feature data of n first features. It also provides perturbed label data (second label data) of each of the multiple samples to device 102, enabling device 102 to initialize m second feature selection parameters based on the perturbed label data of each of the multiple samples and the feature data of m second features. By sending the perturbed label data of each of the multiple samples to device 102, the security of the original label data of each of the multiple samples can be guaranteed, preventing privacy leaks.

[0059] After completing the initialization of the feature selection parameters, devices 101 and 102 can begin model training.

[0060] Below, in conjunction with Figure 4 This section introduces the model training process. Figure 4 This is a timeline diagram of a training method for a vertical federated learning model based on privacy protection and split learning in the embodiments of this specification.

[0061] like Figure 4 As shown, in step S401, user A's device 101 inputs the feature data of n first features of each of k samples from multiple samples into the local first model. The first model selects the n first features based on the values ​​of the n first feature selection parameters corresponding to the n first features, and obtains the feature data of s first features of each of the k samples. Based on the feature data of the s first features of each of the k samples, it performs calculations to obtain k first intermediate results.

[0062] It should be noted that, when the first model includes the first feature selection model and the underlying model as described above, the first feature selection model can specifically select the n first features based on the values ​​of the n first feature selection parameters corresponding to the n first features, obtaining the feature data of s first features for each of the k samples. Then, the underlying model calculates based on the feature data of the s first features for each of the k samples to obtain k first intermediate results. It should be understood that these s first features are selected from the n first features.

[0063] In one implementation, the first model can select n first features based on the values ​​of n first feature selection parameters using a random gate feature selection algorithm, thereby obtaining feature data of s first features for each of the k samples. The random gate feature selection algorithm is a type of embedded feature selection algorithm that multiplies the neural network input layer with an indicator vector S, where the elements S in the indicator vector S... i =max(0,min(1,μ) i +∈i Among them, S i This indicates the i-th element in the indicator vector S. μ i These are feature selection parameters that need to be learned. When S i When S = 0, it can represent that the i-th first feature out of n first features was not selected; when S i When the value is greater than 0, it can represent that the i-th first feature out of n first features is selected.

[0064] When the first model selects n first features based on the values ​​of n first feature selection parameters according to the random gate feature selection algorithm, it can specifically calculate the values ​​of each of the n elements in the local indicator vector S based on the values ​​of the n first feature selection parameters, and then multiply the indicator vector S by the feature data of the n first features of each of the above k samples at corresponding positions. Continuing with f i Taking the i-th first feature out of n first features as an example, we can represent the f of each of the above k samples. i The feature data and the element S in the indicator vector S i Multiply the values.

[0065] In step S403, user B's device 102 inputs the feature data of each of the m second features of the k samples into the local second model. The second model selects the m second features based on the values ​​of the m second feature selection parameters corresponding to the m second features, and obtains the feature data of each of the t second features of the k samples. Based on the feature data of each of the t second features of the k samples, it performs calculations to obtain k second intermediate results.

[0066] It should be noted that when the second model includes the second feature selection model and the underlying model as described above, the second feature selection model can specifically select the m second features based on the values ​​of the m second feature selection parameters corresponding to the m second features, obtaining the feature data of t second features for each of the k samples. Then, the underlying model calculates based on the feature data of the t second features for each of the k samples to obtain k second intermediate results. It should be understood that these t second features are selected from the m second features.

[0067] In one implementation, when the first model selects n first features using a random gate feature selection algorithm, the second model can select m second features based on the values ​​of m second feature selection parameters according to the random gate feature selection algorithm, thereby obtaining the feature data of t second features for each of the above k samples.

[0068] In step S405, device 102 sends k second intermediate results to device 101.

[0069] In practice, localized differential privacy technology can perturb data on the user's device before sending it to the server for statistical analysis, providing user-level privacy protection. Considering the user-level privacy protection offered by localized differential privacy technology, to effectively combat feature-based inference attacks, in one implementation, device 102 can employ a localized differential privacy mechanism to add noise to k second intermediate results before sending the noisy k second intermediate results to device 101. It should be noted that using a localized differential privacy mechanism to protect data security reduces encryption and decryption operations compared to homomorphic encryption, secret sharing, and other encryption algorithms, thereby reducing computational overhead and making model training more efficient.

[0070] Furthermore, device 102 can employ the Gaussian mechanism in localized differential privacy mechanisms to add noise to the k second intermediate results. The definition of the Gaussian mechanism can include: given a dataset D, let there be a function f with l2-sensitivity Δ2f, for any δ∈(0,1), The randomized algorithm A = f(D) + L provides (ε,δ)-differential privacy protection, where ε represents the privacy budget, δ represents the relaxation term, and σ represents the standard deviation. L represents the added noise, which follows a pattern with a mean of 0 and a variance of σ. 2 The Gaussian distribution.

[0071] Below, taking k second intermediate results as an example of intermediate results with h dimensions, we will combine... Figure 5 This section describes the noise addition process for k second intermediate results. Figure 5 This is a flowchart of the noise-adding process for k second intermediate results in the embodiments of this specification.

[0072] like Figure 5 As shown, firstly, in step S501, k second intermediate results are cropped based on the second cropping threshold to obtain k cropped results.

[0073] The second pruning threshold can be preset or determined based on k intermediate second results. Specifically, when determining the second pruning threshold based on k intermediate second results, for example, the L2 norm of each intermediate second result can be calculated to obtain the corresponding fourth calculation result, and then the second pruning threshold can be determined based on each of these fourth calculation results. The L2 norm can be defined as the square root of the sum of the squares of all elements of a vector. Further, the average of each fourth calculation result can be calculated and used as the second pruning threshold; alternatively, the fourth calculation results can be sorted, and the fourth calculation result in the middle position can be selected as the second pruning threshold.

[0074] Let C represent the second cropping threshold, and resultB j Let j represent the j-th second intermediate result out of k second intermediate results, where j∈[k] and [k] represents the set of positive integers {1, 2, ..., k}. For example, the following formula (5) can be used based on C for resultB. j Cutting:

[0075]

[0076] In step S503, the noise value corresponding to each of the h dimensions is sampled from the Gaussian distribution, and the noise value is added to the element of that dimension in the k clipping results to obtain k noisy clipping results.

[0077] Specifically, the second cropping threshold C can be used as... -Sensitivity: For each of the h dimensions, calculate the standard deviation of the Gaussian noise. From Gaussian distribution The noise value corresponding to this dimension is sampled, and this noise value is added to the elements of this dimension in k clipping results, thus obtaining k noisy clipping results.

[0078] In step S505, the target coefficient is calculated based on the sum of squares of each element in the k noisy cropping results, the variance of k and the k samples, and the square of the second cropping threshold.

[0079] In step S507, the product of each element in the k noisy cropping results and the target coefficient is calculated to obtain k noisy intermediate results.

[0080] It should be noted that differential privacy noise may have some impact on model accuracy. By executing steps S505-S507, the k noisy pruning results can be optimized, thereby effectively balancing the usability and privacy of the model.

[0081] In practice, the James-Stein Estimator is a method for estimating the mean of a multidimensional Gaussian distribution. It is an estimation method that, given the variance, adjusts the sample mean by considering the relationships between the components. Specifically, let X1, X2, ..., Xn be a subset of the components from a multidimensional Gaussian distribution. A sample, where μ is the unknown mean vector and ∑ is the known covariance matrix. The James-Stein estimate is expressed as shown in formula (6) below:

[0082]

[0083] in, It can represent an estimate of the variance of a sample. It can represent the mean vector of a sample.

[0084] Considering that the James-Stein estimation algorithm can perform estimations on vectors (such as...) To optimize the vectors, adjustments are made in steps S505 and S507. The James-Stein estimation is performed on the k noisy clipping results using formula (7) to obtain the k intermediate noisy results.

[0085]

[0086] Where, σ 2 Let σ represent the variance of the k samples above, C represent the second pruning threshold, and σ represent the variance of the k samples above. 2 C 2 This represents the variance estimator for the above k samples. This represents k noisy cropping results. express The square of the L2 norm, equivalent to The sum of squares of all elements in the matrix. This represents the target coefficient (also known as the shrinkage factor), calculated as follows. It can represent k noisy intermediate results.

[0087] Device 102 performs, for example Figure 5 The noise-adding process shown completes the noise-adding processing of k second intermediate results. After obtaining k noise-adding intermediate results, the k noise-adding intermediate results can be sent to device 101. For example, the k noise-adding intermediate results can be sent to device 101 as k noise-adding second intermediate results.

[0088] Continue reading Figure 4 In step S407, device 101 inputs the aggregated result of k first intermediate results and k second intermediate results into the local top-level model to obtain k model outputs.

[0089] It should be noted that the k second intermediate results received by device 101 from device 102 can be either the original k second intermediate results or k second intermediate results with added noise. After receiving the k second intermediate results from device 102, device 101 can aggregate the k first intermediate results and the k second intermediate results, and then input the aggregated result into its local top-level model to obtain k model outputs. Taking RL as an example to represent the k model outputs, RL = F AT (w AT (Agg). Among them, w AT The parameters of the top-level model are specifically the function F corresponding to the top-level model. ATThe parameter of (); Agg represents the aggregated result of k first intermediate results and k second intermediate results, and its function is F AT Input data for ().

[0090] When aggregating k first intermediate results and k second intermediate results, device 101 can determine the aggregation method to be used based on the number of columns in each of the first and second intermediate results. For example, when the number of columns in the first and second intermediate results is the same, a weighted average aggregation method can be used, such as by weighting and summing the k first and k second intermediate results to obtain the aggregated result. When the number of columns in the first and second intermediate results is different, a concatenation aggregation method can be used, such as by concatenating the k first and k second intermediate results to obtain the aggregated result, where the number of columns in the aggregated result is the sum of the number of columns in the first and second intermediate results.

[0091] In step S409, device 101 calculates the gradient values ​​corresponding to the first model and the second model based on the outputs of k models and the first label data of each of the k samples.

[0092] Specifically, when the first model includes the first feature selection model, the gradient value corresponding to the first feature selection model can be calculated based on the outputs of the k models and the first label data of the k samples.

[0093] Furthermore, the prediction loss value L can be calculated based on the outputs of the k models and the first label data of each of the k samples. Taking RL as the output of the k models and Label as the first label data of each of the k samples as an example, L = (RL - Label) can be calculated. 2 Then, the gradient value corresponding to the first feature selection model can be calculated based on the predicted loss value L. For example, when it is determined that the model convergence condition is not met based on the predicted loss value L, the gradient value corresponding to the first feature selection model can be calculated based on the predicted loss value L. Further, the gradient value corresponding to the first feature selection model can be calculated based on the loss function Loss used to calculate the predicted loss value L, the predicted loss value L, and n first feature selection parameters. Where Loss = (F... AT (w aT (Agg)-Label) 2 Here, μ is used. A Let n be the first feature selection parameters, and the gradient value corresponding to the first feature selection model can be denoted as:

[0094] When calculating the gradient value corresponding to the second model, the gradient values ​​corresponding to the k second intermediate results can be calculated based on the outputs of the k models and the first label data of the k samples mentioned above. Further, the gradient value corresponding to resultB can be calculated based on the loss function Loss used to calculate the predicted loss value L, the predicted loss value L, and the k second intermediate results (denoted as resultB). This gradient value can be denoted as...

[0095] In step S411, device 101 updates n first feature selection parameters based on the gradient values ​​corresponding to the first model.

[0096] Specifically, the n first feature selection parameters can be updated based on the gradient values ​​corresponding to the first feature selection model, for example, through the formula... Update the n first feature selection parameters. Here, η can represent the learning rate.

[0097] In step S413, device 101 sends the gradient value corresponding to the second model to device 102.

[0098] Specifically, when calculating the gradient values ​​corresponding to each of the k second intermediate results, device 101 can send the gradient values ​​corresponding to each of the k second intermediate results to device 102.

[0099] As mentioned earlier, localized differential privacy protection technology can provide user-level privacy protection strength. With this in mind, in order to effectively combat label inference attacks, in one implementation, device 101 can employ a localized differential privacy mechanism to add noise to the gradient values ​​corresponding to each of the k second intermediate results, and then send the noisy gradient values ​​corresponding to each of the k second intermediate results to device 102.

[0100] Furthermore, device 101 can employ the Gaussian mechanism in the localized differential privacy mechanism to add noise to the gradient values ​​corresponding to each of the k second intermediate results. Taking the gradient values ​​corresponding to each of the k second intermediate results as gradient values ​​in d dimensions as an example, device 101 can prune the gradient values ​​corresponding to each of the k second intermediate results based on a first pruning threshold to obtain pruned results for each of the k second intermediate results. Then, device 101 can sample the noise value corresponding to each of the d dimensions from the Gaussian distribution and add the noise value to the elements of that dimension in the pruned results for each of the k second intermediate results, thereby obtaining noisy pruned results for each of the k second intermediate results. Next, device 101 can calculate the target coefficient based on the sum of squares of each element in the noisy pruned results for each of the k second intermediate results, k, the variance of the k samples, and the square of the first pruning threshold. Finally, device 101 can calculate the product of each element in the noisy pruned results for each of the k second intermediate results and the target coefficient to obtain the noisy gradient value for each of the k second intermediate results.

[0101] The first pruning threshold can be preset or determined based on the gradient values ​​corresponding to each of the k second intermediate results. Specifically, when determining the first pruning threshold based on the gradient values ​​corresponding to each of the k second intermediate results, for example, the L2 norm of the gradient value corresponding to each of the k second intermediate results can be calculated to obtain the third calculation result corresponding to that gradient value, and then the first pruning threshold can be determined based on each of the third calculation results. Further, the average value of each third calculation result can be calculated and the average value can be determined as the first pruning threshold; or, the third calculation results can be sorted and the third calculation result in the middle position can be selected as the first pruning threshold.

[0102] It should be noted that the noise addition process for the gradient values ​​corresponding to the k second intermediate results is similar to the noise addition process for the k second intermediate results described above. Please refer to the relevant explanation of the noise addition process for the k second intermediate results. It will not be elaborated on here.

[0103] In step S415, device 102 updates m second feature selection parameters based on the gradient values ​​corresponding to the second model.

[0104] It should be noted that the gradient value corresponding to the second model received by device 102 from device 101 can be the original gradient value or the gradient value with noise added, and no specific limitation is made here.

[0105] When the second model includes a second feature selection model, after receiving the gradient values ​​corresponding to the k second intermediate results sent by the device 101, the device 102 can calculate the gradient value corresponding to the second feature selection model based on the gradient values ​​corresponding to the k second intermediate results and the m second feature selection parameters, and then update the m second feature selection parameters based on the gradient values.

[0106] Specifically, device 102 can employ a chain rule to calculate the gradient value corresponding to the second feature selection model based on the gradient values ​​corresponding to k second intermediate results and m second feature selection parameters. Let μ B This represents m second feature selection parameters. The gradient value corresponding to the second feature selection model is represented by inputB, which represents the feature data of the t second features of each of the k samples mentioned above. Taking this as an example, device 102 can calculate the gradient value using the formula... The gradient value corresponding to the second feature selection model is obtained. Then, device 102 can, for example, use the formula... Update the m second feature selection parameters.

[0107] Figure 4 The solution provided in the corresponding embodiments enables multiple participants with the same sample space but different feature spaces to collaboratively train the model, ensuring that feature data does not leave the domain, thereby maximizing data security and preventing privacy leaks. Furthermore, both the first and second models have feature selection parameters, which can be adjusted during model training to optimize feature selection. This allows feature data from several important features of k samples to participate in the calculation of intermediate results, avoiding the use of feature data from all features of those k samples. This reduces computational overhead and effectively improves model training efficiency.

[0108] In one implementation, devices 101 and 102 can also calculate L0 regularization terms based on their respective feature selection parameters, and thus calculate the gradient values ​​corresponding to the L0 regularization terms to update the feature selection parameters.

[0109] Specifically, see Figure 6 This is a timeline diagram of a training method for a vertical federated learning model based on privacy protection and split learning in the embodiments of this specification.

[0110] like Figure 6As shown, in step S601, user A's device 101 inputs the feature data of n first features of each of k samples from multiple samples into the local first model. The first model selects the n first features based on the values ​​of the n first feature selection parameters corresponding to the n first features, and obtains the feature data of s first features of each of the k samples. Based on the feature data of the s first features of each of the k samples, it performs calculations to obtain k first intermediate results.

[0111] For an explanation of step S601, please refer to the relevant explanation of step S401 above, which will not be repeated here.

[0112] In step S603, device 101 calculates the L0 regularization term based on n first feature selection parameters to obtain the first calculation result.

[0113] With Reg A Taking the first calculation result as an example, Reg A It can be done through formula The calculation yields the result. Here, λ represents the regularization parameter, σ represents the standard deviation, and μ... i This represents the i-th first feature selection parameter among n first feature selection parameters.

[0114] In step S605, user B's device 102 inputs the feature data of each of the m second features of the k samples into the local second model. The second model selects the m second features based on the values ​​of the m second feature selection parameters corresponding to the m second features, and obtains the feature data of each of the t second features of the k samples. Based on the feature data of each of the t second features of the k samples, it performs calculations to obtain k second intermediate results.

[0115] For an explanation of step S605, please refer to the relevant explanation of step S403 above, which will not be repeated here.

[0116] In step S607, device 102 calculates the L0 regularization term based on m second feature selection parameters to obtain the second calculation result.

[0117] The method for calculating the L0 regularization term based on m second feature selection parameters can be found in the relevant description of step S603, and will not be repeated here.

[0118] In step S609, device 102 sends the second calculation result and k second intermediate results to device 101 respectively.

[0119] Specifically, the second calculation result in plaintext can be sent to device 101. In addition, in order to effectively combat feature inference attacks, reduce computational overhead, and thus effectively improve model training efficiency, device 102 can adopt a localized differential privacy mechanism to add noise to k second intermediate results, and then send the k noisy second intermediate results to device 101.

[0120] In step S611, device 101 inputs the aggregated result of k first intermediate results and k second intermediate results into the local top-level model to obtain k model outputs.

[0121] In step S613, device 101 calculates the gradient value corresponding to the first feature selection model included in the first model and the gradient value corresponding to the k second intermediate results based on the first label data of the k model outputs and the k samples respectively.

[0122] For an explanation of steps S611 and S613, please refer to the relevant explanations of steps S407 and S409 in the previous text, which will not be repeated here.

[0123] In step S615, device 101 calculates the gradient value corresponding to the L0 regularization term based on the target aggregation result of the first calculation result and the second calculation result and n first feature selection parameters.

[0124] Specifically, device 101 can determine the sum of the first and second calculation results as the target aggregation result, and then calculate the gradient value corresponding to the L0 regularization term based on the target aggregation result and n first feature selection parameters. Here, Reg represents the target aggregation result, and μ... A Taking n first feature selection parameters as an example, the gradient value can be denoted as:

[0125] In step S617, device 101 updates n first feature selection parameters based on the aggregation result of the gradient values ​​corresponding to the first feature selection model and the gradient values ​​corresponding to the L0 regularization term.

[0126] Specifically, with This indicates the gradient value corresponding to the first feature selection model. Taking the gradient value corresponding to the L0 regularization term as an example, it can be expressed by the formula Update the n first feature selection parameters.

[0127] In step S619, device 101 sends the target aggregation result and the gradient values ​​corresponding to the k second intermediate results to device 102 respectively.

[0128] Specifically, the target aggregation result in plaintext can be sent to device 102. In addition, in order to effectively combat label inference attacks, reduce computational overhead, and thus effectively improve model training efficiency, device 101 can adopt a localized differential privacy mechanism to add noise to the gradient values ​​corresponding to each of the k second intermediate results, and then send the noisy gradient values ​​corresponding to each of the k second intermediate results to device 102.

[0129] In step S621, device 102 calculates the gradient value corresponding to the second feature selection model based on the gradient values ​​corresponding to the k second intermediate results and the m second feature selection parameters.

[0130] For an explanation of step S621, please refer to the relevant explanation of step S415 above, which will not be repeated here.

[0131] In step S623, device 102 calculates the gradient value corresponding to the L0 regularization term based on the target aggregation result and m second feature selection parameters.

[0132] Where Reg represents the target aggregation result, μ B Taking m second feature selection parameters as an example, the gradient value corresponding to the L0 regularization term can be denoted as:

[0133] In step S625, device 102 updates m second feature selection parameters based on the aggregation result of the gradient values ​​corresponding to the second feature selection model and the gradient values ​​corresponding to the L0 regularization term.

[0134] Specifically, with This represents the gradient value corresponding to the second feature selection model. Taking the gradient value corresponding to the L0 regularization term as an example, it can be expressed by the formula Update the m second feature selection parameters.

[0135] In one implementation, device 101 can further calculate the gradient value corresponding to the top-level model based on the outputs of k models and the first label data of each of the k samples, and update the parameters of the top-level model based on the gradient value. Further, the gradient value corresponding to the top-level model can be calculated based on the loss function Loss used to calculate the predicted loss value L, the predicted loss value L, and the parameters of the top-level model, as described above. (The last sentence appears to be incomplete and possibly refers to a different implementation.) AT Taking the parameters of the top-level model as an example, the gradient value corresponding to the top-level model can be denoted as: Device 101 can be used via formula Update the parameters of the top-level model.

[0136] Furthermore, when calculating the gradient value corresponding to the first model, the gradient value corresponding to the underlying model within the first model can also be calculated, thereby allowing the parameters of the underlying model to be updated based on the gradient value of the underlying model. Further, the gradient value corresponding to the underlying model can be calculated based on the loss function Loss used to calculate the predicted loss value L, the predicted loss value L, and the parameters of the underlying model, as described above. Let w... AL Taking the parameters of the underlying model as an example, the gradient value corresponding to the underlying model can be denoted as: Device 101 can be used via formula Update the parameters of the underlying model.

[0137] In one implementation, after receiving the gradient values ​​corresponding to each of the k second intermediate results sent by device 101, device 102 can further calculate the gradient value corresponding to the underlying model based on the gradient values ​​corresponding to the k second intermediate results and the parameters of the underlying model in the second model, and then update the parameters of the underlying model based on the gradient value. BL This represents the parameters of the underlying model. This represents the gradient value corresponding to the underlying model. Let inputB represent the gradient values ​​corresponding to the k intermediate second results, and inputB represent the feature data of the t second features of the k samples. Taking this as an example, device 102 can calculate using the formula... The gradient values ​​corresponding to the underlying model are obtained. Then, device 102 can, for example, use the formula... Update the parameters of the underlying model.

[0138] Figure 7 This is a schematic diagram of the structure of a vertical federated learning model training device based on privacy protection and split learning, as described in the embodiments of this specification. The device can be applied to a first user's device, which stores first label data and n first feature data for each of multiple samples corresponding to the target model. The model training also includes a second user, whose device stores m second feature data for each of the multiple samples. The device can, for example, execute... Figure 3 , Figure 4 , Figure 6The steps performed by device 101. The device includes: a model training unit 701, configured to input feature data of n first features of k samples from a plurality of samples into a local first model, wherein the first model selects n first features based on the values ​​of n first feature selection parameters corresponding to the n first features to obtain feature data of s first features of each of the k samples, and calculates k first intermediate results based on the feature data of the s first features of each of the k samples; a receiving unit 702, configured to receive k second intermediate results from a second user's device, wherein the k second intermediate results are obtained by the second user's device inputting feature data of m second features of each of the k samples into a local second model; the model training unit 701 is further configured to input the aggregated result of the k first intermediate results and the k second intermediate results into a local top-level model to obtain k model outputs; calculate the gradient values ​​corresponding to the first model and the second model based on the k model outputs and the first label data of each of the k samples; update the n first feature selection parameters based on the gradient values ​​corresponding to the first model; and a sending unit 703, configured to send the gradient values ​​corresponding to the second model to the second user's device.

[0139] Figure 8 This is a schematic diagram of the structure of a vertical federated learning model training device based on privacy protection and split learning, as described in the embodiments of this specification. This device can be applied to a second user's device, which stores feature data of m second features corresponding to multiple samples of the target model. The model training also includes a first user, whose device stores first label data and n first feature data for each of the multiple samples. For example, this device can perform... Figures 3 to 6The steps performed by device 102. The device includes: a model training unit 801, configured to input feature data of m second features of k samples from a plurality of samples into a local second model; the second model selects m second features based on the values ​​of m second feature selection parameters corresponding to the m second features to obtain feature data of t second features of each of the k samples; and calculates k second intermediate results based on the feature data of the t second features of each of the k samples; a sending unit 802, configured to send the k second intermediate results to a first user's device; and a receiving unit 803, configured to receive gradient values ​​corresponding to the second model from the first user's device; the gradient values ​​are calculated by the first user's device based on the k model outputs and the first label data of each of the k samples after inputting the aggregated result of the k first intermediate results and the k second intermediate results into a local top-level model to obtain k model outputs; the k first intermediate results are obtained by the first user's device inputting feature data of n first features of each of the k samples into a local first model; and the model training unit 801 is further configured to update the m second feature selection parameters based on the gradient values ​​corresponding to the second model.

[0140] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform actions such as... Figures 3 to 6 The method described.

[0141] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements, as shown in the embodiment. Figures 3 to 6 The method described.

[0142] This specification also provides a computer program product, wherein when the computer program product is executed in a computer, it causes the computer to perform the following: Figures 3 to 6 The method described.

[0143] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0144] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0145] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0146] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0147] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0148] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0152] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0153] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0154] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0156] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0157] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for training a longitudinal federated learning model based on privacy protection and split learning, executed by a first user's device, the device storing first label data and n first features of multiple samples corresponding to the target model, wherein the model training also includes a second user, the second user's device storing m second features of each of the multiple samples, the method comprising: The feature data of each of the n first features of k samples from the plurality of samples are input into the local first model. The first model selects the n first features based on the values ​​of the n first feature selection parameters corresponding to the n first features, thereby obtaining the feature data of each of the s first features of the k samples. Based on the feature data of each of the s first features of the k samples, k first intermediate results are obtained. The device receives k second intermediate results, which are obtained by the device of the second user by inputting the feature data of m second features of each of the k samples into a local second model; The aggregated result of the k first intermediate results and the k second intermediate results is input into the local top-level model to obtain k model outputs; Calculate the gradient values ​​corresponding to the first model and the second model based on the outputs of the k models and the first label data of the k samples respectively; The n first feature selection parameters are updated based on the gradient values ​​corresponding to the first model, and the gradient values ​​corresponding to the second model are sent to the device of the second user.

2. The method according to claim 1, wherein, Before inputting the feature data of each of the n first features of k samples from the plurality of samples into the local first model, the process further includes: Based on the first label data of each of the multiple samples and the feature data of n first features, the Gini impurity corresponding to each of the n first features is determined; Based on the Gini impurity corresponding to each of the n first features, the initial values ​​of the selection parameters for the n first features are determined.

3. The method according to claim 2, wherein, The first model selects the n first features based on the values ​​of the n first feature selection parameters corresponding to the n first features, including: The first model selects the n first features based on the values ​​of the n first feature selection parameters corresponding to the n first features according to the random gate feature selection algorithm.

4. The method according to claim 1, wherein, The first label data of each of the multiple samples is obtained by one-hot encoding the category label to which each of the multiple samples belongs in a number of category labels; the second model includes m second feature selection parameters corresponding to the m second features; as well as The method further includes: The first label data of each of the plurality of samples is perturbed to obtain the second label data of each of the plurality of samples, and the second label data conforms to the requirements of one-hot encoding; The second label data of each of the plurality of samples is sent to the device of the second user for the initialization of the m second feature selection parameters.

5. The method according to claim 4, wherein, The step of perturbing the first label data of each of the plurality of samples to obtain the second label data of each of the plurality of samples includes: According to a preset perturbation rule, each element in the first label data of each of the multiple samples is perturbed to obtain the perturbed label data of each of the multiple samples. For each of the plurality of samples, if the perturbation label data of the sample meets the requirements of one-hot encoding, then the perturbation label data is determined as the second label data of the sample; if the perturbation label data does not meet the requirements of one-hot encoding, then the perturbation label data is modified to obtain the second label data of the sample.

6. The method according to claim 1, wherein, The first model includes a first feature selection model and a bottom-level model corresponding to the target model, wherein the first feature selection model includes the n first feature selection parameters; the second model includes a second feature selection model and a bottom-level model corresponding to the target model, wherein the second feature selection model includes m second feature selection parameters corresponding to the m second features; as well as The calculation of the gradient values ​​corresponding to the first model and the second model respectively includes: Calculate the gradient value corresponding to the first feature selection model and the gradient value corresponding to each of the k second intermediate results; The step of updating the n first feature selection parameters based on the gradient values ​​corresponding to the first model and sending the gradient values ​​corresponding to the second model to the second user's device includes: The n first feature selection parameters are updated based on the gradient values ​​corresponding to the first feature selection model, and the gradient values ​​corresponding to the k second intermediate results are sent to the device of the second user.

7. The method according to claim 6, further comprising: The L0 regularization term is calculated based on the n first feature selection parameters to obtain the first calculation result; The second calculation result is received from the second user's device, and the second calculation result is obtained by the second user's device calculating the L0 regularization term based on the m second feature selection parameters; Based on the target aggregation result of the first calculation result and the second calculation result and the n first feature selection parameters, calculate the gradient value corresponding to the L0 regularization term; The step of updating the n first feature selection parameters based on the gradient values ​​corresponding to the first feature selection model includes: Based on the aggregation result of the gradient value corresponding to the first feature selection model and the gradient value corresponding to the L0 regularization term, the n first feature selection parameters are updated. The method further includes: The target aggregation result is sent to the second user's device.

8. The method according to claim 6, wherein, The gradient values ​​corresponding to each of the k second intermediate results are gradient values ​​in d dimensions; as well as Sending the gradient values ​​corresponding to each of the k second intermediate results to the second user's device includes: Based on the first pruning threshold, the gradient values ​​corresponding to each of the k second intermediate results are pruned to obtain the pruning results corresponding to each of the k second intermediate results; The noise value corresponding to each of the d dimensions is sampled from the Gaussian distribution, and the noise value is added to the element of that dimension in the clipping results corresponding to the k second intermediate results, thereby obtaining the noisy clipping results corresponding to the k second intermediate results. The target coefficient is calculated based on the sum of squares of each element in the noisy cropping result corresponding to each of the k second intermediate results, k, the variance of the k samples, and the square of the first cropping threshold. Calculate the product of each element in the noisy cropping result corresponding to each of the k second intermediate results and the target coefficient to obtain the noisy gradient value corresponding to each of the k second intermediate results; Return the noise gradient values ​​corresponding to each of the k second intermediate results to the device of the second user.

9. The method according to claim 8, wherein, Before pruning the gradient values ​​corresponding to the k second intermediate results based on the first pruning threshold, the method further includes: For each of the k second intermediate results, calculate the L2 norm of the gradient value corresponding to the second intermediate result to obtain the third calculation result corresponding to the gradient value; The third calculation results are sorted, and the third calculation result in the middle position is selected as the first clipping threshold.

10. A method for training a longitudinal federated learning model based on privacy protection and split learning, executed by a second user's device, the device storing feature data of m second features corresponding to multiple samples of the target model, wherein the model training participants also include a first user, the first user's device storing first label data and n first feature data of the multiple samples, the method comprising: The feature data of m second features of each of the k samples in the plurality of samples are input into the local second model. The second model selects the m second features based on the values ​​of the m second feature selection parameters corresponding to the m second features, to obtain the feature data of t second features of each of the k samples. Based on the feature data of the t second features of each of the k samples, k second intermediate results are obtained. Send the k second intermediate results to the first user's device; The gradient value corresponding to the second model is received from the device of the first user. The gradient value is calculated by the device of the first user based on the first label data of each of the k model outputs and the first label data of each of the k samples after the aggregation result of k first intermediate results and k second intermediate results is input into the local top-level model to obtain k model outputs. The k first intermediate results are obtained by the device of the first user by inputting the feature data of each of the n first features of the k samples into the local first model. The m second feature selection parameters are updated based on the gradient values ​​corresponding to the second model.

11. The method according to claim 10, wherein, The first label data of each of the multiple samples is obtained by one-hot encoding of the category label to which each of the multiple samples belongs in a number of category labels; as well as Before inputting the feature data of each of the m second features of k samples from the plurality of samples into the local second model, the process further includes: The device of the first user receives the second label data of each of the k samples, which conforms to the requirements of one-hot encoding and is obtained by the device of the first user by perturbing the first label data of each of the k samples; Based on the second label data of each of the multiple samples and the feature data of m second features, determine the Gini impurity corresponding to each of the m second features; Based on the Gini impurity corresponding to each of the m second features, the initial values ​​of the selection parameters for the m second features are determined.

12. The method according to claim 10, wherein, The first model includes a first feature selection model and a bottom-level model corresponding to the target model. The first feature selection model includes n first feature selection parameters corresponding to the n first features. The second model includes a second feature selection model and a bottom-level model corresponding to the target model. The second feature selection model includes the m second feature selection parameters. as well as Receiving the gradient value corresponding to the second model from the first user's device includes: Receive the gradient values ​​corresponding to each of the k second intermediate results from the first user's device; The step of updating the m second feature selection parameters based on the gradient values ​​corresponding to the second model includes: Based on the gradient values ​​corresponding to the k second intermediate results and the m second feature selection parameters, calculate the gradient value corresponding to the second feature selection model; The m second feature selection parameters are updated based on the gradient values ​​corresponding to the second feature selection model.

13. The method of claim 12, further comprising: The L0 regularization term is calculated based on the m second feature selection parameters to obtain the second calculation result; Send the second calculation result to the first user's device; The target aggregation result is received from the device of the first user. The target aggregation result is obtained by the device of the first user after calculating the L0 regularization term based on the n first feature selection parameters to obtain the first calculation result, and then aggregating the first calculation result and the second calculation result. Based on the target aggregation result and the m second feature selection parameters, calculate the gradient value corresponding to the L0 regularization term; The step of updating the m second feature selection parameters based on the gradient values ​​corresponding to the second feature selection model includes: Based on the aggregation result of the gradient value corresponding to the second feature selection model and the gradient value corresponding to the L0 regularization term, the m second feature selection parameters are updated.

14. The method of claim 10, wherein, The k second intermediate results are intermediate results of h dimensions; as well as Sending the k second intermediate results to the first user's device includes: The k intermediate results are cropped based on the second cropping threshold to obtain k cropped results; The noise value corresponding to each of the h dimensions is sampled from the Gaussian distribution, and the noise value is added to the element of that dimension in the k clipping results to obtain k noisy clipping results; The target coefficient is calculated based on the sum of squares of each element in the k noisy cropping results, k, the variance of the k samples, and the square of the second cropping threshold. Calculate the product of each element in the k noisy cropping results with the target coefficient to obtain k noisy intermediate results; Send the k noisy intermediate results to the first user's device.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-14.

16. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-14.

Images