A gradient-based data set completeness evaluation method

Abstract

The application discloses a gradient-based data set completeness evaluation method, comprising the following steps: setting a basic data set on a client side, participating in federated training by using the basic data set, and obtaining gradient information; performing gradient inversion on the gradient information to obtain data information of the client side; packing the data information into a data set and putting the data set into a basic model to test the accuracy of the basic model; and judging the data set completeness according to the accuracy. The method can use the information existing in the gradient uploaded by the client to inversely obtain the original real data set of the client, and then make the inversely obtained data into a test set and put the test set into the basic model to test the accuracy, so that the data set completeness of the client can be accurately judged.

Description

Technical Field

[0001] This invention belongs to the field of data quality assessment, and in particular relates to a gradient-based method for assessing dataset completeness. Background Technology

[0002] Federated learning is an emerging distributed machine learning paradigm used to collaboratively train a global model with a massive number of clients. To attract potential high-quality data holders to participate in federated learning, incentives need to be provided based on the expected contribution of each client's data to the collaborative effort. Unfortunately, some clients exploit the limitation of existing evaluation metrics that cannot fully quantify their contributions, falsely claiming to possess compliant data or falsely claiming to have genuinely completed local deep learning. These clients deceive the server by fabricating their contributions to obtain incentives that do not rightfully belong to them.

[0003] Considering that gradients uploaded by clients contain a wealth of information, for example, L Zhu et al. found that sharing gradients in distributed machine learning is insecure; they designed several steps that could easily obtain the original dataset, thus compromising client privacy. J Geiping et al. implemented the reconstruction of high-resolution images using shared gradients in federated learning and demonstrated that trained neural networks also face this risk. Building on previous work, H Yin et al. further confirmed that gradients can be used to recover data even in neural networks with more than 50 layers and at larger batch sizes. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a gradient-based dataset completeness evaluation method to overcome the shortcomings of existing technologies that cannot fully quantify the contribution of datasets.

[0005] To achieve the above objectives, this invention provides a gradient-based dataset completeness evaluation method, comprising the following steps:

[0006] A base dataset is set up on the client side, and the base dataset is used to participate in federated training to obtain gradient information;

[0007] The gradient information is inverted to obtain the client's data information;

[0008] The data information is packaged into a dataset and put into the basic model to test its accuracy. The completeness of the dataset is judged based on the accuracy.

[0009] Preferably, the basic dataset includes a noisy dataset, a partially complete dataset, and a complete dataset;

[0010] The completeness of the partially complete dataset includes 0%, 20%, 40%, 60%, 80%, and 100%.

[0011] Preferably, the gradient inversion process includes:

[0012] The gradient of the data in the client is calculated using the gradient formula, the gradient is divided into equal parts, and the divided gradients are used to update the gradient for the next round.

[0013] Randomly initialize a virtual input and a label input, and then input the initialized virtual input and label input into the model to obtain the virtual gradient;

[0014] The virtual gradient is optimized based on the obtained gradient in the next round to obtain the real training data.

[0015] Preferably, the gradient inversion process further includes:

[0016] When the gradient of a certain step is available, the recovered dataset samples and labels are obtained by minimization based on the virtual gradient.

[0017] Preferably, the method for obtaining the basic model includes: constructing a model and training the constructed model using a basic dataset to obtain the basic model.

[0018] Preferably, the packaging format includes CIFAR-10.

[0019] Preferably, the method for determining the completeness of a dataset based on the accuracy rate includes: setting an accuracy threshold of 85%; when the accuracy rate is greater than or equal to 85%, the dataset is determined to be complete; when the accuracy rate is less than 85%, the dataset is determined to be incomplete.

[0020] Compared with the prior art, the present invention has the following advantages and technical effects:

[0021] This invention proposes a gradient-based method for evaluating dataset completeness, which utilizes the gradient information uploaded by the client to reconstruct the original client's actual dataset. In addition, we add a base dataset and a base model to the federated learning server. The reconstructed data is used to create a test set and fed into the base model to test its accuracy, thereby accurately determining the dataset completeness of the client. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart of the gradient-based dataset completeness evaluation method according to an embodiment of the present invention;

[0024] Figure 2This is a graph showing the change in accuracy as a function of completeness in an embodiment of the present invention. Detailed Implementation

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0027] Example 1

[0028] like Figure 1 As shown, this invention proposes a gradient-based dataset completeness evaluation method. This method adds a basic dataset D to the traditional federated learning framework. base Basic Model M base The underlying dataset D base Unlike traditional federated learning where all data comes from the client, this framework provides the server with a small, clean, and reliable base dataset D. base To obtain a basis for evaluating the completeness of client data, specifically, the server collects datasets from each client before the federated learning training begins. The base dataset D... base It contains approximately 10% of the total data from various clients, and the collected data samples will not participate in local model calculations on the clients. Furthermore, to protect client privacy, this dataset is opaque to other clients. Base Model M base It is the server that utilizes the underlying dataset D base The model obtained through training has a high accuracy rate. The base model is based on the base dataset D. base The high-accuracy model obtained after training with ResNet18 achieved an accuracy of 93.8%.

[0029] The specific steps are as follows:

[0030] 1. Set up datasets with different proportions: noisy dataset, partially complete dataset, and complete dataset. Six datasets with different completeness levels are set: 0%, 20%, 40%, 60%, 80%, and 100%. The client will use these six different datasets for federated learning training and send gradient information to the server.

[0031] 2. Before the server performs formal aggregation: The server uses this gradient information to invert the client's data information. Afterwards, the server packages the inverted data into a dataset according to the CIFAR-10 format and puts it into M. base The accuracy was tested, and the results were as follows: Figure 2 As shown:

[0032] 3. If the dataset created by the client through gradient inversion achieves an accuracy higher than 85% in the base model, federated learning will continue; otherwise, the gradients uploaded by the client will be removed from the global aggregation in this round.

[0033] The gradient inversion algorithm steps are as follows:

[0034] The client's set of data is (x t，i y t，i ), where x t,i Let y be the data sample of client i in round t. t,i This corresponds to the label. Therefore, the gradient of client i in round t can be expressed as Equation 1:

[0035]

[0036] in, The loss function l represents the loss function l with respect to the model output F(x). t，i W t ) and tag y t，i For model parameters W t The partial derivatives. The purpose of this calculation is to determine the direction and extent of the change of the loss function relative to the model parameters. Let W represent the partial derivative, l represent the loss function, and W represent the partial derivative. t F(x) represents the model parameters issued by the server in round t. t,i W t ) represents a differentiable model.

[0037] After obtaining the gradient of client i in round t, the gradient is averaged into N parts, and then the gradient of the next round is updated, as shown in formulas (2) and (3):

[0038]

[0039]

[0040] Here, η represents the learning rate, a hyperparameter used to control the step size or speed of each model parameter update. It determines the magnitude of the model parameter updates in each iteration.

[0041] The gradient information of client k can now be obtained from the federation. The goal is based on gradients Recover data (x) t,k ,y t,k To recover the data from the gradient, the evaluation layer first randomly initializes a virtual input x' and a label input y', i.e., line 1 of Algorithm 1. These "virtual data" are then input into the model to obtain the "virtual gradient," as detailed in Equation (4):

[0042]

[0043] Here, W represents the randomly initialized model parameters, the purpose of which is to initialize the virtual gradient.

[0044] Next, we optimize the virtual gradient to approximate the original gradient. This optimization also makes the virtual data closer to the real training data x'. * ,y' * If the gradient of a certain step already exists... Then, by minimizing equation (5), we can obtain the recovered dataset samples and labels x'. * ,y' * .

[0045]

[0046] in, This represents the value of the parameter that makes the function reach its minimum value. arg min is a combination of the parameter (x', y') and min (minimum value), representing the parameter value that makes the function reach its minimum value.

[0047] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A gradient-based data set completeness evaluation method, characterized by, The method comprises the following steps: Setting a basic data set on a client side, participating in federated training using the basic data set, and obtaining gradient information; Obtaining data information of the client by gradient inversion of the gradient information: calculating the gradient of the data in the client through a gradient formula, dividing the gradient averagely, updating the divided gradient to obtain the gradient of the next round, randomly initializing a virtual input and a label input, inputting the initialized virtual input and label input into a model to obtain a virtual gradient, optimizing the virtual gradient based on the obtained gradient of the next round to obtain real training data; When having the gradient of a certain step, obtaining the recovered data set sample and label based on the virtual gradient by minimizing the formula; When having the gradient of a certain step, obtaining the recovered data set sample and label based on the virtual gradient by minimizing the formula; Packaging the data information into a data set and putting it into a basic model to test the accuracy thereof, and judging the completeness of the data set according to the accuracy.

2. The gradient-based data set completeness evaluation method according to claim 1, wherein The basic data set comprises a noise data set, a partially complete data set, and a complete data set. The completeness of the partially complete data set comprises 0%, 20%, 40%, 60%, 80%, and 100%.

3. The gradient-based data set completeness evaluation method according to claim 1, wherein The method for obtaining the basic model comprises constructing a model, training the constructed model through the basic data set, and obtaining the basic model.

4. The gradient-based data set completeness evaluation method according to claim 1, wherein The format of the packaged data set comprises CIFAR-10.

5. The gradient-based data set completeness evaluation method according to claim 1, wherein The method for judging the completeness of the data set according to the accuracy comprises setting an accuracy threshold of 85%, determining that the data set is complete when the accuracy is greater than or equal to 85%, and determining that the data set is incomplete when the accuracy is less than 85%.

Images