Privacy protection method and system for vertical federated learning considering data leakage among allies

Abstract

The present invention discloses a privacy protection method and system for vertical federated learning that takes into account data leakage between allies, belonging to the field of privacy protection for vertical federated learning. The initial accuracy of the server-side top-level model of the vertical federated learning system is recorded, and the server maintains a data subset sampled from the local data of each client; each client is regarded as a second client in turn, and the remaining clients are regarded as first clients, and the risk of each first client stealing the data of the second client is calculated, and the maximum risk is taken as the overall privacy leakage risk of the second client. If the overall privacy leakage risk exceeds the risk threshold, the updated gradient sent by the server to the second client is noised, and the overall privacy leakage risk is recalculated based on the noised result until the privacy protection condition is met; all clients are traversed; the present invention solves the problem of enhancing the privacy protection capability for data leakage risks between allies in vertical federated learning that does not rely on labels, intermediate results and top-level models.

Description

Technical Field

[0001] The present invention relates to the field of privacy protection of vertical federated learning, and in particular to a privacy protection method and system for vertical federated learning that takes into account data leakage between allies. Background Art

[0002] With increasingly stringent privacy protection requirements, data silos have emerged to avoid privacy leaks, driving the development of federated learning. Vertical Federated Learning (VFL) uses a coordinator (typically a third-party server) to guide multiple clients in joint training. As a distributed learning paradigm that balances privacy and model availability, it provides the fundamental privacy guarantee of data localization and "available but invisible." This means that each participant's data remains hidden from other participants or the server. Its primary application scenario involves clients sharing the same sample space but different feature spaces. For example, traffic cameras from different organizations capture the same object from different angles and jointly predict the object's category to train autonomous driving algorithms. Their shared sample space is the same object, while their different feature spaces are due to different high-dimensional features resulting from different camera angles. As partners in federated learning, these two systems can jointly predict image categories. However, due to data security regulations and the fundamental privacy guarantees of vertical federated learning, their data must remain localized and hidden from each other.

[0003] To achieve data protection, on the one hand, from the perspective of the training architecture, clients in vertical federated learning can only access some features related to predictions, and cannot obtain the overall features and labels required for predictions, nor can they understand the feature information owned by other clients. On the other hand, the vertical federated learning architecture performs noise protection on the content of communications between the server and the client to combat potential data theft attacks between allies.

[0004] Existing privacy protection schemes do not yet account for data leakage between allies in the presence of a server. Specifically, existing privacy risk assessment frameworks often rely on privacy leakage caused by two of the three entities: known labels, intermediate results, and top-level models, while ignoring the risks when all three are unknown. Privacy protection schemes that consider only two of the three known entities for data leakage risk assessment have been widely implemented, by introducing trusted third parties to protect labels and top-level models and encrypting intermediate results. However, the privacy leakage risk caused by allies whose all three are unknown is particularly severe, and because this privacy leakage process does not affect the training process of vertical federated learning, it is difficult to detect. Therefore, how to assess the data leakage risk between allies and strengthen privacy protection in vertical federated learning is a pressing technical issue in this field. Summary of the Invention

[0005] To address the above issues, the present invention provides a vertical federated learning privacy protection method and system that takes into account data leakage between allies, which can effectively enhance the privacy protection capabilities of the vertical federated learning system against the risk of data leakage between allies.

[0006] The technical solution adopted in the present invention is as follows:

[0007] In the first aspect, the present invention proposes a privacy protection method for vertical federated learning that considers data leakage between allies, including:

[0008] Record the initial accuracy of the server-side top-level model of the longitudinal federated learning system. The server maintains a data subset sampled from the local data of each client.

[0009] Consider each client in turn as a second client, and the remaining clients as first clients. Calculate the risk of each first client stealing the second client's data, and use the maximum risk as the overall privacy leakage risk of the second client. If the overall privacy leakage risk exceeds the risk threshold, add noise to the updated gradient sent by the server to the second client, and recalculate the overall privacy leakage risk based on the noise result until the privacy protection condition is met. Traverse all clients;

[0010] When calculating the risk of a first client stealing data of a second client, a label reconstructor is trained based on a data subset of the first client maintained by a server; a first semantically similar image set of the first client data subset and a second semantically similar image set of the second client data subset are obtained, and the sample spaces of the two semantically similar image sets are aligned; the label reconstructor is used to generate pseudo labels for first samples from the first semantically similar image set, and a conditional diffusion model is used to reconstruct a candidate set of second samples from the second semantically similar image set using the first sample and its pseudo label as conditional guidance, and reconstructed samples are screened from the candidate set. The risk of the first client stealing data of the second client is obtained based on the peak signal-to-noise ratio of the reconstructed sample and the second sample.

[0011] Furthermore, in the vertical federated learning system, each client maintains a local underlying model, the intermediate features generated by each local underlying model are spliced ​​on the server and the target category is predicted by the top model, and the update gradient of each local underlying model is calculated on the server.

[0012] Furthermore, in the vertical federated learning system, the local data maintained by the client are object images from different perspectives, and the server fuses the intermediate features of the object images from different perspectives to predict the object category.

[0013] Furthermore, the step of training a label reconstructor based on the first client's data subset maintained by the server includes:

[0014] Inputting the data subset of the first client maintained by the server into the local underlying model of the first client, reducing the dimension of the intermediate features generated by the local underlying model and clustering them to generate cluster labels;

[0015] A fully connected layer is trained using the data subset of the first client and its cluster labels, so that the fully connected layer can predict the cluster labels based on the intermediate features generated by the local underlying model of the first client, and the prediction results are used as pseudo labels.

[0016] Furthermore, the process of acquiring the first semantically similar image set and the second semantically similar image set includes:

[0017] Sampling from the first and second client data subsets respectively, and using a search engine to collect image sets that are semantically similar to the sampling results as first and second semantically similar image sets;

[0018] The sample spaces of the first and second semantically similar image sets are aligned to ensure that the corresponding images in the two similar image sets are images of the same object from different perspectives.

[0019] Furthermore, the method of reconstructing a candidate set of second samples from a second semantically similar image set using a conditional diffusion model with the first sample and its pseudo-label as conditional guides includes:

[0020] Perform forward diffusion on a second sample from the second semantically similar image set, gradually adding noise to generate a noisy image; then, using the first sample from the first semantically similar image set and its pseudo-label as conditional input, gradually denoise the noisy image under the conditional guidance to restore the second sample, and train a conditional diffusion model; the second sample and the first sample are images of the same object from different perspectives;

[0021] Taking a pure noise image as input, the trained conditional diffusion model is used to generate a predicted image under the guidance of the first sample and its pseudo-label. The predicted image is used as a candidate image of the second sample. The conditional diffusion model is run multiple times to generate a candidate set of the second sample.

[0022] Furthermore, the process of screening and reconstructing samples from the candidate set includes:

[0023] Calculate the fidelity score based on the distribution of the predicted labels of each sample in the candidate set and the first sample, and eliminate candidate samples with low fidelity scores;

[0024] Calculate the perceptual similarity based on the normalized Hamming distance between each sample in the candidate set and the perceptual hash sequence of the first sample, and eliminate candidate samples with low perceptual similarity;

[0025] For each sample in the candidate set, the pixel values ​​of the same position area as the first sample are randomly sampled, and the L2 distance is calculated element by element to obtain the matching score. The sample with the highest matching score from the candidate set is used as the reconstructed sample.

[0026] Furthermore, the process of adding noise to the updated gradient sent by the server to the second client includes:

[0027] Gradually add small-scale noise to the gradient calculated by the server and sent to the second client to update the local bottom-level model to obtain the noisy gradient, record the accuracy of the top-level model on the server after the noise addition, and calculate the difference with the initial accuracy;

[0028] If the accuracy difference does not exceed the accuracy fluctuation threshold, the noised result is retained and the overall privacy leakage risk of the second client after noised is further evaluated;

[0029] If the accuracy difference exceeds the accuracy fluctuation threshold, the newly added noise is removed, and the elements with smaller absolute values ​​in the gradient are gradually replaced with zero. The overall privacy leakage risk of the second client after the noise is added is continuously evaluated.

[0030] Furthermore, the privacy protection condition means that the accuracy difference of the top model on the server side does not exceed the accuracy fluctuation threshold, and the overall privacy leakage risk of each client does not exceed the risk threshold.

[0031] In a second aspect, the present invention proposes a vertical federated learning privacy protection system that takes into account data leakage between allies, which is used to implement the above-mentioned vertical federated learning privacy protection method that takes into account data leakage between allies.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] This invention constructs a privacy risk assessment system for unknown labels / intermediate results / top-level models. Through semantically similar image set alignment and conditional diffusion model reconstruction technology, it pioneers a three-level evaluation mechanism of "label reconstruction-candidate generation-optimal screening", uses clustered label reconstruction to achieve semantic guidance, and combines multi-dimensional screening with perceptual hashing and pixel matching. It breaks through the traditional method's dependence on intermediate results and label information, and realizes the privacy leakage risk assessment in scenarios where all three are unknown for the first time. It fills the technical gap in the field of privacy protection in vertical federated learning that considers data leakage between allies, and accurately quantifies the risk of data theft between allies in vertical federated learning.

[0034] This paper uses a dual strategy of gradient noising and compression, combined with real-time monitoring of top-level model accuracy fluctuations, to achieve an optimal balance between privacy protection and model performance, effectively resolving the dilemma between privacy reinforcement and model usability. This paper can comprehensively assess the privacy leakage risks of vertical federated learning and implement targeted reinforcement. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the overall process for enhancing the privacy protection capabilities of a single client.

[0036] Figure 2 This is a schematic diagram of the vertical federated learning architecture (taking the enhanced privacy protection capability of client B among the three clients as an example).

[0037] Figure 3 Provides a method for assessing the risk of an ally stealing client data (taking client A stealing client B's data as an example).

[0038] Figure 4 Experimental results for evaluating the effectiveness of the optimal selection module. DETAILED DESCRIPTION

[0039] The present invention will be further described and illustrated below in conjunction with specific embodiments. The embodiments are merely illustrative of the present disclosure and do not limit its scope. The technical features of the various embodiments of the present invention may be combined accordingly, provided that there is no conflict between them.

[0040] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0042] like Figure 1 As shown in Figure 2, the main steps of the privacy-preserving method for longitudinal federated learning considering data leakage between allies include:

[0043] (1) The server adds noise or performs gradient compression on the gradients returned to the client for local model updates;

[0044] (2) Based on the steps of “label reconstruction – candidate generation – optimal selection”, the privacy leakage risk among allies is evaluated;

[0045] (3) Based on the evaluation results, adjust the scale of the noise and repeat step (2) until the data leakage risk is effectively avoided.

[0046] For step (1), where the server adds noise or performs gradient compression on the gradients returned to the client for local model update, an optional implementation process is as follows:

[0047] (1-1) For K clients participating in vertical federated learning, the final classification accuracy obtained by training is The top model; for each client, calculate the risk of the remaining K-1 clients stealing their data , where the risk of the kth client stealing its data is ; The overall data leakage risk faced by the client is .

[0048] For the convenience of description, Figure 2 As shown in the figure, the vertical federated learning system consists of three clients and a third-party server. Taking the example of the server evaluating the data leakage risk of clients A and C to client B and strengthening B's privacy, the privacy leakage risk faced by B among allies is calculated. , the overall data leakage risk is .

[0049] (1-2) The gradient calculated by the server and sent to client B to update the local underlying model is , gradually adding small-scale noise to it ,get , thereby achieving privacy protection for the client, and recording the classification accuracy of the top model after adding noise , calculate the accuracy difference ;when and When the client's privacy protection is completed; note It needs to be added gradually in small scale because excessive noise will affect the performance of the vertical federated learning system.

[0050] (1-3) When and When removing noise, gradually implement small-scale gradient compression , until when and The scope of implementation is The gradient compression method is specifically implemented as follows: The absolute value of each element in and sort it from large to small to determine the front index of elements; create an Zero tensors of the same shape , the absolute value of the largest elements are copied to the .

[0051] For step (2), based on the steps of “label reconstruction - candidate generation - optimal selection”, the privacy leakage risk between allies is evaluated. Here, the server evaluates the data leakage risk caused by client A to client B. Take this as an example to introduce.

[0052] In one embodiment of the present invention, the data leakage risk assessment system includes three modules: label reconstruction module, candidate generation module and optimal selection module. Figure 3 As shown, the server keeps a small validation dataset Implement verification phase, including 、 and The local data sets from three clients are 、 and The validation data subset is randomly sampled from ; the specific modules and the functions of each module are as follows:

[0053] (2-1) The label reconstruction module is mainly used to verify the data subset Generate pseudo labels for each sample in the dataset to achieve two purposes: on the one hand, the pseudo labels serve as guidance for the candidate generation module to ensure the semantic consistency between the generated data and the target data; on the other hand, the pseudo labels align the candidate generation module training data (from external data: Semantically similar image set) and longitudinal federated learning dataset ( and ) to improve the quality of generation. Therefore, it is necessary to use the label reconstruction module to generate The pseudo labels for each sample in .

[0054] The specific process of generating pseudo labels is as follows:

[0055] Server splicing underlying model Fully connected layer As a label reconstructor, it will simulate the attacker's underlying model The generated intermediate high-dimensional feature vector is subjected to t-SNE dimensionality reduction processing and clustering operation to assign pseudo labels to each sample. ,in, express The pseudo label of the i-th sample in .

[0056] In the tag reconstructor, enter , the output is an intermediate high-dimensional feature vector, which is reduced in dimension and clustered to generate pseudo labels; then, and pseudo-label training fully connected layer, so that Able to predict pseudo labels for each input sample.

[0057] (2-2) The candidate generation module uses a small-scale validation data subset and the underlying model , generate multiple candidate reconstructed images, the specific process is as follows:

[0058] (2-2-1) Training dataset preparation: From the local dataset of the simulated attacker Randomly sample data from the search engine and collect A collection of semantically similar images , and use search engines to collect and A collection of semantically similar images ; and To satisfy the sample space alignment, that is, Images in and Images in It is a different perspective of the same object taken at the same time and place, and the sample index is i.

[0059] (2-2-2) Implement the forward diffusion process using the noise manager to add Gaussian noise to The process can be modeled as a Markov process, which iterates through T rounds to Add Gaussian noise:

[0060]

[0061] Among them, q represents the probability distribution in the forward diffusion process, represents the data at the t-th time step, is a hyperparameter of noise scheduling, N represents Gaussian distribution, represents the identity matrix, represents the total amount of noise from the 1st time step to the tth time step. Each step can be marginalized as:

[0062]

[0063] in, Indicates the current noise level, The image without noise added, i.e. .

[0064] (2-2-3) Design a denoising network to gradually remove noise. As a conditional guide, restore the original input . In a specific implementation of the present invention, the category information and the sinusoidal time step embedding are spliced ​​in the embedding layer to enhance the semantic similarity in the generation process, so as to better capture the potential structure; then the residual blocks are stacked to extract multi-scale features; skip connections are constructed to preserve spatial information; attention blocks are used in the bottleneck part connecting the downsampling step and the upsampling step to dynamically focus on key features; the denoising network can be designed according to the well-known technology in the field, and the present invention does not limit it. After conventional conditional guided training, the conditional diffusion model (denoising network) can be made to take a pure noise image as input. The predicted image is generated under the guidance of the conditions.

[0065] (2-2-4) Use the trained candidate generation module to generate data M times, each time with the same input As a condition, the mth generation result is expressed as:

[0066]

[0067] Finally, we get the candidate set As The candidate generation results.

[0068] (2-3) The purpose of the optimal selection module is to select Determine the reconstruction results .

[0069] For each candidate image , which has a certain degree of difference from other pictures in the candidate set, based on Select the best candidate images at the perceptual and feature point levels, and process the candidate set sequentially through coarse-grained and fine-grained filters The coarse-grained filter calculates each candidate and The fidelity score (FS) after splicing is then calculated and In this embodiment, after discarding the candidate images with the lowest FS value ranking of 50%, the PS values ​​of the remaining images are calculated, and the candidate images with the lowest PS value ranking of 50% are further discarded.

[0070] Fine-grained filters randomly sample With each The matching score (MS) of the two is calculated based on the pixel values ​​of the area, and the image with the highest MS value is finally selected as the reconstruction result. The specific process of selecting the best data is as follows:

[0071] (2-3-1) The calculation process of the fidelity score (FS) is as follows: Use the general open source model Inception-V3 network to predict each candidate image The label is obtained ,in Represents the probability distribution of all labels. Inception-V3 is trained on a large-scale real-world dataset to effectively capture high-level semantic information and evaluate the realism of the image; based on this, high-fidelity images will produce a distribution concentrated on a specific label, and this "concentration" is quantified by calculating the Shannon entropy. The FS is shown in the following formula:

[0072]

[0073] in, express The j-th element of , smaller FS value indicates higher fidelity.

[0074] (2-3-2) Perceptual Similarity (PS) Evaluation Using Perceptual Hashing and Because perceptual hashing focuses more on low-frequency information, which can capture overall color changes, smooth areas, contours and shapes, while ignoring high-frequency details. This method is particularly suitable for evaluating the global perceptual characteristics of images. To calculate perceptual similarity, first calculate the and pHash sequences of the same length, i.e. and , and then calculate the normalized Hamming distance between them. The calculation formula is as follows:

[0075]

[0076] in, express The jth element of Indicates the length of the pHash sequence, Represents the "exclusive OR" operation (i.e., if the two element values ​​are the same, they are 1, and if they are different, they are 0). A smaller PS value indicates a higher similarity.

[0077] (2-3-3) Matching score (MS) as a fine-grained filter evaluation of randomly sampled identical locations With each The similarity of the pixel values ​​of the area; and Respectively and At the pixels in the sampling area, by calculating element-by-element Distance is used to evaluate their matching degree. The calculation formula is as follows:

[0078]

[0079] in, express The jth element of Indicates the number of pixels in the area. A smaller MS value indicates a higher similarity.

[0080] (2-3-4) Select the image with the highest MS as the final reconstruction result .

[0081] (2-4) Evaluate reconstruction results Indicated privacy leakage risks , specifically, calculate the reconstruction result With the original picture The peak signal-to-noise ratio between and is used to determine the degree of privacy leakage:

[0082]

[0083] Among them, MAX is the maximum pixel value, MSE is and Pixel-by-pixel mean square error; The larger the value, the greater the risk of privacy leakage.

[0084] For step (3), evaluate the privacy leakage risk caused by each ally to client B, that is, and , take the maximum value as the privacy leakage risk faced by client B among allies Repeat until the privacy protection condition in step (1) is met: and .

[0085] By traversing all clients and performing privacy leakage risk assessment on each client, the gradient sent from the server to the client is noised according to the risk assessment results until the privacy protection conditions are met. This results in a vertical federated learning system with enhanced privacy protection capabilities against data leakage risks among allies.

[0086] As shown in Table 1, the proposed method shows excellent reconstruction quality in all tasks, even when the attacker only has 1 / 3 of the features. In particular, the case where the attacker has 1 / 2 of the features is analyzed. Specifically, the mean square error (MSE) values ​​of the four tasks are very low, all within 10 -3The peak signal-to-noise ratio (PSNR) values ​​are 32, 22, 28, and 42, respectively. In addition, the structural similarity (SSIM) values ​​are close to 0.9 in the three tasks, which all indicate excellent reconstruction results.

[0087] On the ImageNet task, our method performs particularly well, with an extremely low MSE (1.17e-4) and a PSNR of 41.67. Although its performance on the medical task is relatively weak, it still achieves satisfactory reconstruction results, with an MSE of 7.94e-3 and a PSNR of 21.89.

[0088] There are three main reasons for this performance difference:

[0089] The ImageNet dataset contains all categories in Tiny-ImageNet, so the generated quality is higher;

[0090] Specific patterns related to human tissue in medical images can weaken the guiding role of generating semantically meaningful outputs;

[0091] The current design of perception-level evaluation metrics does not fully consider this type of data with special patterns.

[0092] Table 1: Experimental data on performance of privacy leakage assessment methods

[0093]

[0094] Verifying the Effectiveness of the Label Reconstruction Module: As shown in Table 2, the label reconstruction module demonstrates high effectiveness in label reconstruction, even achieving 100% accuracy in the traffic task. Furthermore, it demonstrates satisfactory label alignment when preprocessing the candidate generation module training dataset, thereby facilitating the generation of semantically consistent candidate samples. The performance of the label reconstruction module in both tasks shows no significant fluctuations in performance as the proportion of features possessed by the attacker decreases, demonstrating that the module maintains strong stability even with limited features.

[0095] Table 2: Experimental data on the effectiveness of the label reconstruction module

[0096]

[0097] Effectiveness of the Candidate Generation Module: As shown in Table 3, the effectiveness of the candidate generation module is evaluated by measuring its performance on its design goals, including the integration of the guidance mechanism, the fidelity of the generated results, and the moderate diversity among the candidate samples.

[0098] To evaluate guidance and fidelity, a classifier was trained using the VFL dataset, using the prediction accuracy of candidate samples as a measure of their similarity to the corresponding real data. Therefore, higher classification accuracy for candidate samples indicates better fidelity. Specifically, for the binary classification task (medical), outputs with prediction probabilities below 0.75 (the average confidence level in real-data tasks) were considered zero, as any random input would only achieve an accuracy of 0.5, equivalent to random guessing. As shown in the "Guidance vs. Fidelity" column in Table 3, the effectiveness of the guidance mechanism was evaluated by comparing the classifier's prediction accuracy for candidate samples generated with and without guidance. Higher accuracy indicates better fidelity. Results show that candidate samples generated with the guidance mechanism perform significantly better, achieving, for example, an accuracy improvement of over 90% on the CIFAR task. In contrast, without guidance, the generated results are extremely poor, with accuracy even below 2% on the ImageNet task and close to 0 on the medical task, primarily due to the large number of classes in ImageNet. This comparison highlights the important role of the guidance mechanism.

[0099] We further compared the classification accuracy of the guided candidate samples with that of real data to assess their generation fidelity. As shown in Table 3, the average difference between the guided candidate samples and the real data was no more than 6.14%. In the CIFAR task, even using half the features exceeded the real data. This fully demonstrates the strong generation fidelity of the candidate generation module.

[0100] To evaluate the diversity among candidate samples, the standard deviation (std.) within the candidate sample set is compared with the standard deviation of similar samples in the target dataset. As shown in the “Diversity” column in Table 3, the standard deviation of the candidate sample set is approximately arrive The standard deviation of samples within the class is reduced by an average of 75.19%, indicating that the generated candidate samples have moderate discrimination.

[0101] Table 3: Experimental data on the effectiveness of the candidate generation module

[0102]

[0103] Effectiveness of the optimal selection module: In order to evaluate the effectiveness of the optimal selection module, the trend of the average PSNR, SSIM, MSE, MAE and LPIPS of the candidate samples after filtering at each layer is shown, as shown in the figure below. Figure 4As shown in the figure, as candidate samples pass through each layer of screening, their average PSNR and SSIM continue to increase, while MSE, MAE, and LPIPS continue to decrease. These trends indicate that each layer of filtering effectively eliminates poor samples, thereby converging to the optimal reconstruction result. Even when the overall quality of all candidate samples has reached a good level (for example, an average PSNR of approximately 20), the coarse-grained filter still provides an average performance improvement of 9.23%, while the fine-grained filter further improves this by an average of 4.89%. Overall, the application of the optimal selection module improves performance by an average of 14.62%, demonstrating its significant value in screening candidate reconstruction results.

[0104] Based on the same inventive concept, this embodiment also provides a vertical federated learning privacy protection system that considers data leakage between allies, including:

[0105] The server side is used to record the initial accuracy of the server-side top model of the longitudinal federated learning system. The server maintains a data subset sampled from the local data of each client;

[0106] A privacy leakage risk assessment module is used to treat each client as a second client and the remaining clients as first clients, calculate the risk of each first client stealing the second client's data, and use the maximum risk as the overall privacy leakage risk of the second client;

[0107] When calculating the risk of a first client stealing data from a second client, a label reconstructor is trained based on a data subset of the first client maintained by a server; a first semantically similar image set from the first client data subset and a second semantically similar image set from the second client data subset are obtained, and the sample spaces of the two semantically similar image sets are aligned; the label reconstructor is used to generate pseudo labels for first samples from the first semantically similar image set, and a conditional diffusion model is used to reconstruct a candidate set of second samples from the second semantically similar image set using the first samples and their pseudo labels as conditional guides. Reconstructed samples are screened from the candidate set, and the risk of the first client stealing data from the second client is calculated based on the peak signal-to-noise ratio of the reconstructed samples and the second samples.

[0108] A gradient noising module is used to add noise to the updated gradient sent by the server to the second client. If the overall privacy leakage risk exceeds a risk threshold, the updated gradient sent by the server to the second client is also added with noise.

[0109] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0110] Embodiments of the system of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, a software implementation, as a logical device, is implemented by a processor of any device with data processing capabilities, reading corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0111] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they are not to be construed as limiting the scope of the present invention. Persons skilled in the art will readily appreciate that variations and modifications may be made without departing from the spirit of the present invention, and all such variations and modifications fall within the scope of protection of the present invention.

Claims

1. A privacy protection method for vertical federated learning considering data leakage between allies, characterized by: include: Record the initial accuracy of the server-side top-level model of the longitudinal federated learning system. The server maintains a data subset sampled from the local data of each client. Each client is considered a second client in turn, and the remaining clients are considered first clients. The risk of each first client stealing the second client's data is calculated, and the maximum risk is used as the overall privacy leakage risk of the second client. If the overall privacy leakage risk exceeds the risk threshold, the update gradient sent by the server to the second client is noised, and the overall privacy leakage risk is recalculated based on the noised result until the privacy protection condition is met; Traverse all clients; When calculating the risk of a first client stealing data of a second client, a label reconstructor is trained based on a data subset of the first client maintained by a server; a first semantically similar image set of the first client data subset and a second semantically similar image set of the second client data subset are obtained, and the sample spaces of the two semantically similar image sets are aligned; the label reconstructor is used to generate pseudo labels for first samples from the first semantically similar image set, and a conditional diffusion model is used to reconstruct a candidate set of second samples from the second semantically similar image set using the first sample and its pseudo label as conditional guidance, and reconstructed samples are screened from the candidate set. The risk of the first client stealing data of the second client is obtained based on the peak signal-to-noise ratio of the reconstructed sample and the second sample.

2. The privacy protection method for vertical federated learning considering data leakage between allies according to claim 1 is characterized in that: In the vertical federated learning system, each client maintains a local underlying model, the intermediate features generated by each local underlying model are spliced ​​on the server and the target category is predicted by the top model, and the update gradient of each local underlying model is calculated on the server.

3. The privacy protection method for vertical federated learning considering data leakage between allies according to claim 2 is characterized in that: In the vertical federated learning system, the local data maintained by the client are object images from different perspectives, and the server fuses the intermediate features of the object images from different perspectives to predict the object category.

4. The privacy protection method for vertical federated learning considering data leakage between allies according to claim 1 is characterized in that: The step of training a label reconstructor based on the first client's data subset maintained by the server includes: Inputting the data subset of the first client maintained by the server into the local underlying model of the first client, reducing the dimension of the intermediate features generated by the local underlying model and clustering them to generate cluster labels; A fully connected layer is trained using the data subset of the first client and its cluster labels, so that the fully connected layer can predict the cluster labels based on the intermediate features generated by the local underlying model of the first client, and the prediction results are used as pseudo labels.

5. The privacy protection method for vertical federated learning considering data leakage between allies according to claim 1 is characterized in that: The process of acquiring the first semantically similar image set and the second semantically similar image set includes: Sampling from the first and second client data subsets respectively, and using a search engine to collect image sets that are semantically similar to the sampling results as first and second semantically similar image sets; The sample spaces of the first and second semantically similar image sets are aligned to ensure that the corresponding images in the two similar image sets are images of the same object from different perspectives.

6. The privacy protection method for vertical federated learning considering data leakage between allies according to claim 1 is characterized in that: The method of reconstructing a candidate set of second samples from a second semantically similar image set using a conditional diffusion model with the first sample and its pseudo-label as conditional guidance includes: Perform forward diffusion on a second sample from the second semantically similar image set, gradually adding noise to generate a noisy image; then, using the first sample from the first semantically similar image set and its pseudo-label as conditional input, gradually denoise the noisy image under the conditional guidance to restore the second sample, and train a conditional diffusion model; the second sample and the first sample are images of the same object from different perspectives; Taking a pure noise image as input, the trained conditional diffusion model is used to generate a predicted image under the guidance of the first sample and its pseudo-label. The predicted image is used as a candidate image of the second sample. The conditional diffusion model is run multiple times to generate a candidate set of the second sample.

7. The privacy protection method for vertical federated learning considering data leakage between allies according to claim 1 or 6, characterized in that: The process of screening and reconstructing samples from the candidate set includes: Calculate the fidelity score based on the distribution of the predicted labels of each sample in the candidate set and the first sample, and eliminate candidate samples with low fidelity scores; Calculate the perceptual similarity based on the normalized Hamming distance between each sample in the candidate set and the perceptual hash sequence of the first sample, and eliminate candidate samples with low perceptual similarity; For each sample in the candidate set, the pixel values ​​of the same position area as the first sample are randomly sampled, and the L2 distance is calculated element by element to obtain the matching score. The sample with the highest matching score from the candidate set is used as the reconstructed sample.

8. The privacy protection method for vertical federated learning considering data leakage between allies according to claim 1 is characterized in that: The process of adding noise to the updated gradient sent by the server to the second client includes: Gradually add small-scale noise to the gradient calculated by the server and sent to the second client to update the local bottom-level model to obtain the noisy gradient, record the accuracy of the top-level model on the server after the noise addition, and calculate the difference with the initial accuracy; If the accuracy difference does not exceed the accuracy fluctuation threshold, the noised result is retained and the overall privacy leakage risk of the second client after noised is further evaluated; If the accuracy difference exceeds the accuracy fluctuation threshold, the newly added noise is removed, and the elements with smaller absolute values ​​in the gradient are gradually replaced with zero. The overall privacy leakage risk of the second client after the noise is added is continuously evaluated.

9. The privacy protection method for vertical federated learning considering data leakage between allies according to claim 8 is characterized in that: The privacy protection condition means that the accuracy difference of the top model on the server side does not exceed the accuracy fluctuation threshold, and the overall privacy leakage risk of each client does not exceed the risk threshold.

10. A vertical federated learning privacy protection system considering data leakage between allies, used to implement the vertical federated learning privacy protection method described in claim 1, characterized in that: The system comprises: The server side is used to record the initial accuracy of the server-side top model of the longitudinal federated learning system. The server maintains a data subset sampled from the local data of each client; A privacy leakage risk assessment module is used to treat each client as a second client and the remaining clients as first clients, calculate the risk of each first client stealing the second client's data, and use the maximum risk as the overall privacy leakage risk of the second client; When calculating the risk of a first client stealing data from a second client, a label reconstructor is trained based on a data subset of the first client maintained by a server; a first semantically similar image set from the first client data subset and a second semantically similar image set from the second client data subset are obtained, and the sample spaces of the two semantically similar image sets are aligned; the label reconstructor is used to generate pseudo labels for first samples from the first semantically similar image set, and a conditional diffusion model is used to reconstruct a candidate set of second samples from the second semantically similar image set using the first samples and their pseudo labels as conditional guides. Reconstructed samples are screened from the candidate set, and the risk of the first client stealing data from the second client is calculated based on the peak signal-to-noise ratio of the reconstructed samples and the second samples. A gradient noising module is used to add noise to the updated gradient sent by the server to the second client. If the overall privacy leakage risk exceeds a risk threshold, the updated gradient sent by the server to the second client is also added with noise.

Images