A method, device and storage medium for forgetting learning based on distributed conversion
By introducing distributed transformation and EWC loss functions in the forgetting learning cloud service, the problem of users being unable to customize forgetting tools and privacy data leakage is solved, and the effective protection of user privacy and the robustness of the forgetting process is achieved.
Patent Information
- Application Number
- CN202411364722.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-09-29
AI Technical Summary
In the forgetful learning cloud service scenario, users in the existing technology cannot customize forgetting tools, resulting in privacy data leakage and "catastrophic forgetting" phenomena, making it difficult to effectively protect user privacy.
A forgetful learning method based on distributed transformation DT-UPD is proposed. By obtaining user privacy data and dividing it into training set, verification set and forgetful set, the forgetful module is designed for feature mapping and random prediction, and the EWC loss function is introduced to prevent Disaster forgetting and optimize the forgetting process through tool format conversion and optimization functions.
It realizes the protection of user privacy data, avoids data leakage, reduces the loss of verification sets, prevents catastrophic forgetting, and improves the robustness and privacy protection effect of the forgetting process.
Smart Images
Figure CN118886065B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of forgetting learning, and more specifically, relates to a forgetting learning method, device and storage medium based on distributed transformation. Background Art
[0002] Machine learning has become essential in many real-world applications, such as medical diagnosis, risk assessment, etc. Therefore, privacy issues also arise in these scenarios. As an effective privacy protection method, machine unlearning has become a research hotspot in recent years. It allows users to eliminate the memory of their private data from a trained machine learning model.
[0003] Chinese invention patent CN117892843A discloses a machine learning data forgetting method based on game theory and cryptography, which includes: obtaining a training set and constructing a learning system thereof; initializing and iterating the learning system; constructing necessary functions and constructing a judgment game matrix based on the necessary functions; inputting all data and performing model training according to the SISA framework; users play games with them and allocate their own game space; using PKI to encrypt their own game space to complete information category forgetting protection; and extracting the results of machine forgetting learning.
[0004] In summary, in the forgetting learning cloud service scenario, existing forgetting learning methods require forgetting cloud servers to provide users with forgetting tools. Before users forget, they use the forgetting tools to process their local private data. These methods implicitly assume that the forgetting learning cloud service is trustworthy. Many cases have shown that this assumption threatens user privacy. Therefore, there is a problem that the forgetting learning cloud server leaks user data privacy through the forgetting tool with loopholes. In the existing technology, users cannot define their own forgetting tools without relying on cloud servers, and the data set loss is large, making it difficult to protect their privacy. Summary of the invention
[0005] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a forgetting learning method based on distributed transformation to solve problems such as user privacy protection in the forgetting learning cloud service scenario.
[0006] The detailed technical scheme of the present invention is as follows:
[0007] This paper proposes a user privacy protection method based on distributed transformation in the forgetting learning cloud service (MUaaS) scenario, namely a forgetting learning method based on distributed transformation, which is defined as DT-UPD. The method includes:
[0008] S1. Obtain user privacy data X and the data label Y corresponding to X to form a data set, and divide the data set into a training set, a validation set, and a forgetting set;
[0009] The training set is used to train the forgetting learning cloud model, the test set is used to test the forgetting learning cloud model, and the forgetting set is used to test the accuracy of forgetting learning.
[0010] S2. Design a forgetting module and train it using user privacy data X. The forgetting module includes a forgetting tool, a tool format conversion, and a forgetting operation.
[0011] The forgetting tool extracts the feature M of the private data X before sampling and maps it to the distribution Q, and then randomly samples from the distribution Q to obtain the low-dimensional feature M´;
[0012] The tool format conversion forgets and reconstructs the sampled feature M' to obtain the sampled data X', further diluting the feature so that the forgetting tool and the forgetting operation can be smoothly performed.
[0013] The forgetting operation, the forgetting learning cloud model performs random prediction on the sampled data X' to complete the forgetting learning operation, and the random prediction includes completely random distribution prediction and uniform distribution prediction.
[0014] S3. Input the user privacy data X into the forget module and optimize it through the optimization function. At the same time, use the EWC algorithm to optimize the loss. While inputting the current forget operation optimization model, retain the content learned by the forget cloud model in the previous forget operation training process:
[0015] (1);
[0016] In formula (1), is the value of the variable that makes the formula in () reach the minimum value; and They are the regularization term of user privacy data and the utility regularization term, is the weight parameter used to control the privacy protection utility regularization term;
[0017] represents mutual information, is a function that maps X to X´, that is (X) = X´; Dp represents the minimization and The KL divergence distance between the two, () is the processing function of the forgetting learning cloud model on the sampled data X´; is the original P distribution of X; is the Q distribution of X´ in the low-dimensional space, M is the feature of the user’s private data X before sampling, and the loss of the forgetting process and EWC needs to be minimized; For balance and The weight parameter, To forget the loss in the learning process, that is, the total loss, the current forgetting operation model is obtained through the EWC algorithm, while retaining the old task knowledge to prevent catastrophic forgetting.
[0018] S4: The training ends when the set training threshold or the set forgetting accuracy rate is reached.
[0019] Furthermore, the S2 specifically includes:
[0020] S21. Design a forgetting tool. First, extract the feature M of the private data X, i.e., the feature before sampling, and map M to an unrelated distribution Q. The decisive parameter of distribution Q is i ;
[0021] Furthermore, if the distribution Q is a normal distribution, then ,in , represent the mean and variance respectively, i are the decisive parameters of Q, namely the mean and variance.
[0022] S22. When the determining parameter of the distribution Q i When determined, from Q ( i ) is randomly sampled to obtain low-dimensional features M´, i It is the decisive parameter of Q, and the low-dimensional feature M´ is the feature after sampling;
[0023] For the same input X, each sampled M′ can be different. This random sampling process further enhances the privacy protection effect. However, directly sending M′ to the Forgotten Cloud Server will cause format problems and make the data unrecognizable. Therefore, the following tool format conversion is used to solve this problem.
[0024] S23, tool format conversion. In order to make the format of the sampled data X' consistent with the data input format of the forgetting learning operation, the sampled feature M' needs to be forgotten and reconstructed to obtain the sampled data X':
[0025] (2);
[0026] In formula (2), E is the mean value, It is the mean of the information difference between the sampled feature M' and the sampled data X' in high and low dimensions, S is the sampling operation, and the loss in the forgetting operation is minimized. ; Ensure that the overall impact on the forgetting learning cloud model is minimized during the forgetting process. The forgetting cloud model refers to the cloud model required in the forgetting operation, which is used to forget the data after forgetting reconstruction.
[0027] S24. The sampled data X' constitutes a sampled data set for training the forgetting learning cloud model.
[0028] S25, the forgetting operation is performed through random prediction, and the sampled data X' is randomly predicted, wherein the random prediction includes a completely random distribution prediction and a uniform distribution prediction. The sampled data X' randomly selects one of the completely random distribution prediction and the uniform distribution prediction for prediction, thereby reducing the prediction deviation, helping to improve the robustness of the forgetting process, and reducing the risk caused by relying on a single method;
[0029] The sampled data X´ is predicted to be completely randomly distributed:
[0030] (3);
[0031] In formula (3), a completely randomly generated value is used for the sampled data X´ Instead, it is randomly selected from the entire input space. In the input space Uniform distribution on the input space It refers to the set of all possible input values, which is achieved by replacing the original data with uniformly distributed random values to achieve the effect of forgetting these data.
[0032] Make uniform distribution predictions for sampled data X´:
[0033] (4);
[0034] In formula (4), the sampled data X' is replaced by a uniform distribution prediction. The uniform distribution prediction replaces X' with a value drawn from a uniform distribution within a preset range. represents the value of the random prediction after replacement, Represents a uniform distribution on the interval [c,d], where [c,d] represents a specific range of values used to generate predicted values.
[0035] Then, formula (4) is specified as:
[0036] (5);
[0037] In formula (5), represents the uniformly distributed predicted value of the i-th sample data feature, Indicates that the i-th sample data is in the interval [ here,this ] is uniformly distributed on .
[0038] Furthermore, using only the forget set for training during the forgetting process can lead to "catastrophic forgetting", in which the model forgets everything it has trained, including the training set, validation set, and forget set. To avoid this, the present invention implements an optimized "EWC loss" to help prevent this from happening:
[0039] (6);
[0040] In formula (6), represents the EWC loss function, represents the loss function of the current forget operation, Represents the loss of the previous forget operation, which is a loss term introduced to retain the knowledge of the old task.
[0041] Furthermore, we optimize the regularization term , that is, minimize the mutual information of the private data X and the sampled data X´ to minimize their mutual information;
[0042] Optimizing mutual information Converted to JS divergence optimization, the resulting function is expressed as maximizing JS divergence optimization:
[0043] (7);
[0044] In formula (7), JS on the right side of the equation represents Jensen-Shannon divergence. By maximizing the similarity between Q and P through JS, X and X' are guaranteed to be independent of each other in terms of privacy information expression. Optimize
[0045] Expand the JS on the left side of the equation:
[0046] (8);
[0047] Forgetting the loss in the learning process , and expand it to get:
[0048] (9);
[0049] In formula (9), is the forgetting weight coefficient, which is used to adjust the loss of the forgetting operation process. and They represent the features of the i-th sample data before forgetting and reconstructing and the features of the i-th sample data after forgetting and reconstructing respectively;
[0050] Furthermore, in optimizing When , minimize the information difference between the reconstructed feature set and the original feature set to ensure that the two are parallel in utility. And optimize it to maintain high forgetting performance during the forgetting process:
[0051] (10);
[0052] In formula (10), the optimization It is divided into two parts, one of which is the optimization of the current forget operation loss, namely , and the other part is the loss term introduced to retain what the forget cloud model has learned in the previous forget operation training process, namely ;in, is the weight coefficient, which is used to control the balance between the EWC loss function and the current forgetting operation loss. Represents the weight parameter of the i-th task. The main function of the weight parameter is to ensure that the model can successfully forget the specified data while maintaining the performance and stability of other data to the greatest extent. represents the optimal parameters for the i-th task, is the weight coefficient used to control the balance between the EWC loss function and the loss of the previous forget operation, represents the weight parameter of the previous forget operation, Represents the optimal weight parameter of the previous forget operation.
[0053] Furthermore, we optimize the utility regularization term , that is, minimize the KL divergence distance between the two:
[0054] When mapping X to X′, we ensure And make (X) = (X´), Generalized to:
[0055] (11);
[0056] In formula (11), Y´ is the sample data label corresponding to the output of X´, Pf is the distribution of the output of the forgetting learning cloud model, and d represents the distance metric.
[0057] For the privacy-utility tradeoff of utility regularization, we need to find a mapping method (), map X to X′, ensuring and (X) = (X′), so that X and X´ are as parallel as possible in utility, and the utility regularization is simplified to:
[0058] (12);
[0059] In formula (12), is to minimize the following formula; Formula (12) is called cross entropy. In many classification task scenarios, cross entropy is usually used as a loss function.
[0060] In another aspect of the present invention, a forgetting learning device based on distributed transformation is provided, comprising:
[0061] at least one processor; and
[0062] A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to perform the above-mentioned forgetting learning method based on distributed transformation.
[0063] In another aspect of the present invention, a machine-readable storage medium is provided, which stores executable instructions. When the instructions are executed, the machine performs the above-mentioned forgetting learning method based on distributed transformation.
[0064] Compared with the prior art, the present invention has the following beneficial effects:
[0065] The present invention provides a forgetting learning method, device and storage medium based on distributed conversion. Users can obtain forgetting learning services while protecting their privacy data. DT-UPD allows users to obtain forgetting learning services without directly using their own privacy data. At the same time, EWC loss is introduced in the forgetting process, which can effectively alleviate the "catastrophic forgetting" phenomenon in forgetting learning experiments and minimize our losses in the verification set. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 It is a flow chart of a forgetting learning method based on distributed transformation described in the present invention.
[0067] Figure 2 It is a schematic diagram of t-SNE visualization of low-dimensional space in Example 1 of the present invention. DETAILED DESCRIPTION
[0068] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0069] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0070] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0071] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0072] Example 1
[0073] Ginseng Figure 1 , this embodiment provides a forgetting learning method based on distributed transformation, namely DT-UPD, the method comprising:
[0074] S1. Obtain user privacy data X and the data label Y corresponding to X to form a data set, and divide the data set into a training set, a validation set, and a forgetting set.
[0075] The training set is used to train the forgetting learning cloud model, the test set is used to test the forgetting learning cloud model, and the forgetting set is used to verify the accuracy of forgetting learning.
[0076] Preferably, two datasets are used to verify the effectiveness of DT-UPD: MNIST and CIFAR-10, where MNIST is a handwriting dataset containing 70,000 images of size 28*28, and CIFAR10 contains 60,000 images of 10 different objects, each of which is 32*32*3 in size; the MNIST dataset is mainly used to compare images with and without distributed transformations, and the CIFAR-10 dataset is mainly used for the forgetting learning experiment of DT-UPD.
[0077] Specifically, the first 1000 samples from the CIFAR-10 dataset are selected as the validation set, then the 1000th to the last samples are selected as the training set, and then the first 48 samples are selected from the training set as the forgetting set.
[0078] S2. Design a forgetting module and train it using user privacy data X. The forgetting module includes a forgetting tool, a tool format conversion, and a forgetting operation.
[0079] The forgetting tool extracts the feature M of the private data X before sampling through a distributed transformation method and maps it to the distribution Q, and then randomly samples from the distribution Q to obtain a low-dimensional feature M´;
[0080] The tool format conversion forgets and reconstructs the sampled feature M' to obtain the sampled data X', further diluting the feature so that the forgetting tool and the forgetting operation can be smoothly performed;
[0081] The forgetting operation performs random prediction on the sampled data X' and completes a forgetting learning operation on the sampled data. The random prediction includes a completely random distribution prediction and a uniform distribution prediction.
[0082] Specifically, the S2 specifically includes:
[0083] S21. Design a forgetting tool. First, extract the feature M of the private data X, i.e., the feature before sampling, and map M to an unrelated distribution Q. The decisive parameter of distribution Q is i ;
[0084] Specifically, the distribution Q may be a Gaussian distribution, a uniform distribution, or a normal distribution. Preferably, Q represents a normal distribution. Then ,in , represent the mean and variance respectively, i are the decisive parameters of Q, namely the mean and variance.
[0085] S22. When the determining parameter θ of the distribution Q is determined, from Q ( i ) is randomly sampled to obtain low-dimensional features M´, i It is the decisive parameter of Q, and the low-dimensional feature M´ is the feature after sampling;
[0086] For the same input X, each sampled M′ can be different. This random sampling process further enhances the privacy protection effect. However, directly sending M′ to the Forgotten Cloud Server will cause format problems and make the data unrecognizable. Therefore, the following tool format conversion is used to solve this problem.
[0087] S23, tool format conversion. In order to make the format of the sampled data X' consistent with the data input format of the forgetting learning operation, the sampled feature M' is forgotten and reconstructed to obtain the sampled data X':
[0088] (2);
[0089] In formula (2), E is the mean value, It is the mean of the information difference between the maximized sampled feature M' and the sampled data X' in high and low dimensions, and S is the sampling operation;
[0090] At the same time, minimize the loss during the forgetting operation to ensure that the overall impact on the forgetting learning cloud model is minimized during the forgetting process. The forgotten cloud model refers to the cloud model required in the forgetting operation, which is used to forget the data set after forgetting and reconstruction. Ensure that the overall impact on the forgetting learning cloud model is minimized during the forgetting process. The forgotten cloud model refers to the cloud model required in the forgetting operation, which is used to forget the data set after forgetting and reconstruction.
[0091] S24, the sampled data X' is used to form a sampled data set for training the forgetting learning cloud model;
[0092] S25. The forgetting operation is performed through random prediction. The sampled data X' is randomly predicted. The random prediction includes a completely random distribution prediction and a uniform distribution prediction. The sampled data X' randomly selects one of the completely random distribution prediction and the uniform distribution prediction for prediction.
[0093] The sampled data X´ is predicted to be completely randomly distributed:
[0094] (3);
[0095] In formula (3), a completely randomly generated value is used for the sampled data X´ Instead, this value is randomly chosen from the entire input space; In the input space Uniform distribution on the input space It refers to the set of all input values, which is achieved by replacing the original data with uniformly distributed random values to achieve the effect of forgetting these data.
[0096] Make uniform distribution predictions for sampled data X´:
[0097] (4);
[0098] In formula (4), the sampled data X' is replaced by a uniform distribution prediction. The uniform distribution prediction replaces X' with a value drawn from a uniform distribution within a preset range. represents the value of the random prediction after replacement, Represents a uniform distribution on the interval [c,d], where [c,d] represents a specific range of values used to generate predicted values.
[0099] Specifically in practice, uniform distribution prediction is concretized as:
[0100] (5);
[0101] In formula (5), Indicates i The uniformly distributed predicted value of the sampled data features, Indicates i The sampling data is in the interval [c i ,d i ] is uniformly distributed on .
[0102] Furthermore, in the forgetting process, a large number of experiments have found that using only the forgetting set for training will lead to "catastrophic forgetting", in which the model will forget everything it has trained, including the training set, the validation set, and the forgetting set. To avoid this, the present invention implements an optimized "EWC loss" to help prevent this from happening:
[0103] (6);
[0104] In formula (6), represents the EWC loss function, represents the loss function of the current forget operation, Represents the loss of the previous forget operation, which is a loss term introduced to retain the knowledge of the old task.
[0105] S3. Input the user privacy data X into the forget module and optimize it through the optimization function. At the same time, use the EWC algorithm to optimize the loss. While inputting the current forget operation optimization model, retain the content learned by the forget cloud model in the previous forget operation training process:
[0106] (1);
[0107] In formula (1), is the value of the variable that makes the formula in () reach the minimum value; and They are the regularization term of user privacy data and the utility regularization term, is the weight parameter used to control the privacy protection utility regularization term;
[0108] represents mutual information, is a function that maps X to X´, that is (X) = X´; Dp represents the minimization and The KL divergence distance between the two, () is the processing function of the forgetting learning cloud model on the sampled data X´; is the original P distribution of X; is the Q distribution of X´ in the low-dimensional space, M is the feature of the user’s private data X before sampling, and the loss of the forgetting process and EWC needs to be minimized; For balance and The weight parameter, To forget the loss in the learning process, that is, the total loss, the current forgetting operation model is obtained through the EWC algorithm, while retaining the old task knowledge to prevent catastrophic forgetting.
[0109] Furthermore, we optimize the regularization term , that is, minimize the mutual information of the private data X and the sampled data X´ to minimize their mutual information;
[0110] Optimizing mutual information Converted to JS divergence optimization, the resulting function is expressed as maximizing JS divergence optimization:
[0111] (7);
[0112] In formula (7), the right side of the equation is JS represents Jensen-Shannon divergence. By maximizing the similarity between Q and P through JS, X and X' are guaranteed to be independent of each other in expressing private information. Optimize
[0113] The left side of the equation The JS is expanded to:
[0114] (8).
[0115] Forgetting the loss in the learning process , and expand it to get:
[0116] (9);
[0117] In formula (9), is the forgetting weight coefficient, which is used to adjust the loss of the forgetting operation process. and They represent the features of the i-th sample data before forgetting and reconstructing and the features of the i-th sample data after forgetting and reconstructing respectively;
[0118] Preferably, in the optimization When , we need to minimize the information difference between the reconstructed feature set and the original feature set to ensure that the two are parallel in utility. And optimize it to maintain high forgetting performance during the forgetting process:
[0119] (10);
[0120] In formula (10), the optimization It is divided into two parts, one of which is the optimization of the current forget operation loss, namely , and the other part is the loss term introduced to retain what the forget cloud model has learned in the previous forget operation training process, namely ;in, is the weight coefficient, which is used to control the balance between the EWC loss function and the current forgetting operation loss. Represents the weight parameter of the i-th task. The main function of the weight parameter is to ensure that the model can successfully forget the specified data while maintaining the performance and stability of other data to the greatest extent. represents the optimal parameters for the i-th task, is the weight coefficient used to control the balance between the EWC loss function and the loss of the previous forget operation, represents the weight parameter of the previous forget operation, Represents the optimal weight parameter of the previous forgetting operation. After adding the EWC loss process, the phenomenon of "catastrophic forgetting" in the forgetting learning process can be significantly prevented.
[0121] Specifically, when optimizing JS, P is known and fixed. By expanding JS, maximizing the similarity between Q and P is transformed into maximizing The entropy of M sampled from The difference in entropy between samples:
[0122] (13);
[0123] In formula (13), To maximize the degree of confusion of the joint distribution of Q distribution and P distribution, is the degree of disorder of the Q distribution.
[0124] from Start minimizing, that is, minimizing the entropy of the Q distribution:
[0125] (14).
[0126] In formula (14), when M is a fixed value, the entropy of Q is the smallest. However, this will make the forgetting tool lose its generalization ability. Therefore, we make Q a distribution represented by a simple mathematical expression, such as normal distribution N, uniform distribution U, Poisson distribution π, etc. Preferably, the normal distribution is selected:
[0127] (15);
[0128] Formula (15) indicates that the feature M of the user privacy data X before sampling is mapped to a normal distribution Q, where Q represents the normal distribution. Then ,in , represent the mean and variance of the normal distribution respectively.
[0129] Preferably, the parameter Q i It should not be determined manually, but should be optimized. In the training of the forgetting tool, it can be adjusted according to different inputs X. i , to control the uncertainty of Q, since the information contained in different X is diverse, different X correspond to different certainty levels of Q (i.e., some X require high certainty of Q in order for the Forget Learning Cloud model to output the correct label, and vice versa):
[0130] (16);
[0131] Formula (16) is to minimize the entropy of the joint distribution of P and Q. P is the user's privacy data distribution, P is a constant. Due to the additivity of entropy, Q and P are theoretically two completely independent sources, so:
[0132] (17);
[0133] In formula (17), To maximize the entropy of the Q distribution, P is a complex chaotic distribution, so the value of C is large. Therefore, as long as the entropy of Q is increased, that is, in Establish a trade-off with formula (12);
[0134] This trade-off depends on i , so use X to determine i ,when i When determined, from Q ( i ) to obtain low-dimensional features M′, i.e., the features after sampling.
[0135] Preferably, optimize the utility regularization term , that is, minimize the KL divergence distance between the two:
[0136] When mapping X to X′, we ensure And make (X) = (X´), Generalized to:
[0137] (11);
[0138] In formula (11), Y´ is the sample data label corresponding to the output of X´, Pf is the distribution of the output of the forgetting learning cloud model, and d represents the distance metric. is the weight parameter used to control the privacy-preserving utility regularization term.
[0139] Specifically, for the privacy-utility trade-off of utility regularization, we need to find a mapping method (), map X to X′, ensuring and (X) = (X′), so that X and X´ are as parallel as possible in utility, and the utility regularization is simplified to:
[0140] (12);
[0141] In formula (12), is to minimize the following formula; Formula (12) is called cross entropy. In many classification task scenarios, cross entropy is usually used as a loss function;
[0142] Furthermore, we can rewrite the utility regularization as follows, which is a more general representation:
[0143] (18);
[0144] In formula (18), Y´ is the sample data label corresponding to the output of X´, Pf is the distribution of the output of the forgetting learning cloud model, and d represents the distance metric. Preferably, d is the Euclidean distance.
[0145] S4: The training ends when the set training threshold or the set forgetting accuracy rate is reached.
[0146] Preferably, the forgetting learning effect is evaluated. Before the forgetting operation, the data is visualized in low-dimensional space by using the t-SNE dimensionality reduction technology, such as Figure 2 As shown, Figure 2 Horizontal axis: the first coordinate axis in the two-dimensional space generated by the t-SNE algorithm after dimensionality reduction, indicating the first feature dimension generated after t-SNE dimensionality reduction; Figure 2 Vertical axis: The second coordinate axis in the two-dimensional space generated by t-SNE, representing the second feature dimension generated after t-SNE dimensionality reduction.
[0147] t-SNE maintains the local similarity of data points by embedding data into a low-dimensional space, ensuring that similar data points are as close as possible and dissimilar data points are as far away as possible.
[0148] For the data before and after the transformation, t-SNE visualizes the distribution pattern of high-dimensional data in low-dimensional space, so that the similarities and differences before and after the distribution transformation can be observed. The transformed data is orthogonal to the data before the transformation in expression and parallel to the data before the transformation in utility; the forgetting learning effect is tested and evaluated through the CIFAR-10 data set, and the forgetting effect and the accuracy of the validation set are evaluated, wherein the forgetting effect is fed back through the forgetting accuracy, and the low forgetting accuracy indicates that the forgetting effect is relatively good; after 200 rounds of experimental evaluation, the forgetting method of the present invention can achieve a forgetting accuracy of 18.22% in the forgetting set, while only 4% of the validation set accuracy decreases. By converting user privacy data into sampled data for forgetting, user privacy data can be effectively protected, and a good utility trade-off can be achieved.
[0149] Furthermore, the comparison results of the present invention and other methods in the forgetting learning cloud service scenario are shown in Table 1:
[0150] Table 1 Comparison results
[0151]
[0152] Among them, the reasoning time is the process from model input to output. The present invention only needs the process from input to output. However, the encryption method requires the process of encryption first, reasoning, and finally decryption. The present invention can shorten the time a lot.
[0153] Example 2
[0154] This embodiment provides a device for implementing a forgetting learning method based on distributed transformation, the device comprising:
[0155] at least one processor; and
[0156] A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to perform the above-mentioned forgetting learning method based on distributed transformation.
[0157] In this embodiment, the device includes but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.
[0158] Example 3
[0159] This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the above-mentioned forgetting learning method based on distributed transformation.
[0160] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.
[0161] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.
[0162] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code may be downloaded from a server computer or a cloud via a communication network.
[0163] Embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0164] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.
Claims
1. A forgetting learning method based on distributed transformation, characterized in that: The method comprises: S1. Obtain user privacy data X and the data label Y corresponding to X to form a data set, and divide the data set into a training set, a validation set, and a forgetting set; S2. Design a forgetting module and train it using user privacy data X. The forgetting module includes a forgetting tool, a tool format conversion, and a forgetting operation. The forgetting tool extracts the feature M of the user's private data X before sampling and maps it to the distribution Q, and then randomly samples from the distribution Q to obtain a low-dimensional feature M´; The tool format conversion forgets and reconstructs the sampled feature M' to obtain the sampled data X'; The forgetting operation, the forgetting learning cloud model performs random prediction on the sampled data X' to complete the forgetting learning operation, and the random prediction includes completely random distribution prediction and uniform distribution prediction; S3. Input the user privacy data X into the forget module and optimize it through the optimization function; at the same time, use the EWC algorithm to optimize the loss: （1）； In formula (1), is the value of the variable that makes the formula in () reach the minimum value; and They are the regularization term of user privacy data and the utility regularization term, is the weight parameter used to control the privacy protection utility regularization term; represents mutual information, is a function that maps X to X´, that is (X) = X´; Dp represents the minimization and The KL divergence distance between the two, () is the processing function of the forgetting learning cloud model on the sampled data X´; is the original P distribution of X; is the Q distribution of X´ in the low-dimensional space, M is the feature of the user’s private data X before sampling; For balance and The weight parameter, To forget the loss during the operation, Losses for EWC; S4: The training ends when the set training threshold or the set forgetting accuracy rate is reached; The S2 specifically includes: S21. Design a forgetting tool. First, extract the feature M of the user's privacy data X, that is, the feature before sampling, and map M to an unrelated distribution Q. The decisive parameter of distribution Q is θ ; S22. When the determining parameter of the distribution Q θ When determined, from Q ( θ ) is randomly sampled to obtain low-dimensional features M´, θ It is the decisive parameter of Q, and the low-dimensional feature M´ is the feature after sampling; S23, tool format conversion, forget the sampled feature M' and reconstruct it to obtain the sampled data X': （2）； In formula (2), E is the mean value, It is the mean of the information difference between the sampled feature M' and the sampled data X' in high and low dimensions, S is the sampling operation, and the loss in the forgetting operation is minimized. ; S24, the sampled data X' is used to form a sampled data set for training the forgetting learning cloud model; S25, the forgetting operation is performed by random prediction, and the sampled data X' is randomly predicted, wherein the random prediction includes a completely random distribution prediction and a uniform distribution prediction, and the sampled data X' randomly selects one of the completely random distribution prediction and the uniform distribution prediction for prediction; The sampled data X´ is predicted to be completely randomly distributed: （3）； In formula (3), a completely randomly generated value is used for the sampled data X´ Instead, it is randomly selected from the entire input space. In the input space Uniform distribution on the input space refers to the set of all possible input values; Make uniform distribution predictions for sampled data X´: （4）； In formula (4), the sampled data X' is replaced by a uniform distribution prediction. The uniform distribution prediction replaces X' with a value drawn from a uniform distribution within a preset range. represents the value of the random prediction after replacement, Indicates that in the interval [ c,d ] uniform distribution on [ c,d ] indicates a specific value range, which is used to generate predicted values; Then, formula (4) is specified as: （5）； In formula (5), represents the uniformly distributed predicted value of the i-th sample data feature, Indicates that the i-th sample data is in the interval [ c i , d i ] is uniformly distributed on .
2. The method for forgetting learning based on distributed transformation according to claim 1, characterized in that: The distribution Q is a normal distribution, then ,in , represent the mean and variance respectively, θ is the determining parameter of Q.
3. A forgetting learning method based on distributed transformation according to claim 1 or 2, characterized in that: Said for: （6）； In formula (6), represents the EWC loss function, represents the loss function of the current forget operation, Represents the loss of the previous forget operation.
4. The method for forgetting learning based on distributed transformation according to claim 2, characterized in that: The optimization regularization term specifically includes: Optimizing the regularization term , that is, minimize the mutual information of the private data X and the sampled data X´ to minimize their mutual information; Optimizing mutual information Converted to JS divergence optimization, the resulting function is expressed as maximizing JS divergence optimization: （7）； In formula (7), JS on the right side of the equation represents Jensen-Shannon divergence. By maximizing the similarity between Q and P through JS, X and X' are guaranteed to be independent of each other in terms of privacy information expression. Optimize Expand the JS on the left side of the equation: （8）； Forgetting the loss in the learning process , and expand it to get: （9）； In formula (9), is the forgetting weight coefficient, which is used to adjust the loss of the forgetting operation process. and They respectively represent the features of the i-th sample data before forgetting and reconstructing and the features of the i-th sample data after forgetting and reconstructing.
5. The method for forgetting learning based on distributed transformation according to claim 4, characterized in that: In Optimization When , the information difference between the reconstructed feature set and the original feature set is minimized: （10）； In formula (10), the optimization It is divided into two parts, one of which is the optimization of the current forget operation loss, namely , and the other part is the loss term introduced to retain what the forget cloud model has learned in the previous forget operation training process, namely ;in, is the weight coefficient, which is used to control the balance between the EWC loss function and the current forgetting operation loss. represents the weight parameter of the bth task, represents the optimal parameters for the bth task, is the weight coefficient used to control the balance between the EWC loss function and the loss of the previous forget operation, represents the weight parameter of the previous forget operation, Represents the optimal weight parameter of the previous forget operation.
6. The method for forgetting learning based on distributed transformation according to claim 2, characterized in that: The optimization utility regularization term specifically includes: Optimizing the utility regularization term , that is, minimize the KL divergence distance between the two: When mapping X to X′, we ensure And make (X) = ,Will Generalized to: （11）； In formula (11), Y´ is the sample data label corresponding to the output of X´, Pf is the distribution of the output of the forgetting learning cloud model, and d represents the distance metric; For the privacy-utility tradeoff of utility regularization, we need to find a mapping method () is the processing function of the forgetting learning cloud model on the sampled data X', mapping X to X' to ensure and (X) = (X′), so that X and X´ are as parallel as possible in utility, and the utility regularization is simplified to: （12）； In formula (12), is to minimize the following formula.
7. A forgetting learning device based on distributed transformation, characterized in that: The device comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of a forgetting learning method based on distributed transformation as described in any one of claims 1 to 6 are implemented.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.