Privacy erasure model training method, privacy erasure method and device
By training the privacy erasure model, using the model structure and weight data of the big model to build a loss function, directly modifying the big model parameters, solving the problem of big model privacy data leakage, realizing safe and efficient privacy data deletion, and reducing costs.
Patent Information
- Application Number
- CN202411025575.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-07-29
AI Technical Summary
Large models may leak private data during training, and existing solutions may be costly or easily bypassed.
By training the privacy erase model, use the model structure of the large model to describe text and weight data to build a loss function, update the privacy erase model parameters, directly modify the parameters of the large model to delete the privacy data, and there is no need to retrain the large model.
It realizes the secure deletion of large-scale private data, reduces optimization costs and improves security.
Smart Images
Figure CN118568775B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a privacy erasure model training method, a privacy erasure method and a privacy erasure device. Background Art
[0002] During the training process of the Large Language Model (LLM), the training data may contain a large amount of private data. An attacker can cause privacy leakage by asking questions to the large model and extracting private data from the response content. There are two main existing solutions to the problem of privacy leakage. One is to desensitize or directly delete the private data in the training data, and then retrain the large model. This method requires retraining the large model, which is extremely costly. The other is to perform content filtering in the request and response links of the large model, blocking or desensitizing malicious questions and responses containing private information, but this method is very easy to be bypassed by attackers. Summary of the invention
[0003] One or more embodiments of this specification provide a privacy erasure model training method, a privacy erasure method and a device, which can solve the problem of privacy leakage of large models at a very low cost.
[0004] In a first aspect, a method for training a privacy erasure model is provided, comprising:
[0005] Obtain at least one large model to be trained;
[0006] Obtaining a first training sample set containing private data;
[0007] Desensitizing the private data in the first training sample set to obtain a desensitized second training sample set;
[0008] For each of the large models, the large model is trained using the first training sample set, and first weight data of the large model after training is obtained;
[0009] Adopting a completely consistent training method, using the second training sample set to train the large model, and obtaining second weight data after the large model is trained;
[0010] The model structure description text of the large model and the first weight data of the large model are used as training samples, and the second weight data of the large model is used as a label to train the privacy erasure model.
[0011] As an optional implementation manner of the method of the first aspect, training the privacy erasure model specifically includes:
[0012] The model structure description text of the large model and the first weight data of the large model are input into the privacy erasure model, and a loss function is constructed based on the model weight data predicted by the privacy erasure model and the second weight data of the large model to update the parameters of the privacy erasure model.
[0013] Specifically, the loss function is constructed based on the cross entropy of the predicted model weight data and the second weight data of the large model.
[0014] In a second aspect, a privacy erasure method is provided, the method comprising:
[0015] Obtain the weight data of the target large model after training based on the training task;
[0016] Inputting the model structure description text of the target large model and the weight data into a privacy erasure model to obtain the weight data after privacy erasure; the privacy erasure model is pre-trained according to the above-mentioned privacy erasure model training method;
[0017] The target large model is reconstructed according to the model structure description text and the weight data after privacy erasure.
[0018] As an optional implementation of the method of the second aspect, obtaining weight data of the target large model after training based on the training task specifically includes:
[0019] According to the training task of the target large model, a third training sample set is obtained;
[0020] Using the third training sample set to train the target large model;
[0021] Obtain the weight data of the trained target large model.
[0022] In a third aspect, a training device for a privacy erasure model is provided, the device comprising:
[0023] A first data acquisition module, configured to acquire at least one large model to be trained;
[0024] A second data acquisition module is configured to acquire a first training sample set containing private data;
[0025] A desensitization module, configured to perform desensitization processing on the privacy data in the first training sample set to obtain a desensitized second training sample set;
[0026] A first training module is configured to, for each of the large models, train the large model using the first training sample set and obtain first weight data of the large model after training; and, using a completely consistent training method, train the large model using the second training sample set and obtain second weight data of the large model after training;
[0027] The second training module is configured to use the model structure description text of the large model and the first weight data of the large model as training samples, and the second weight data of the large model as labels to train the privacy erasure model.
[0028] As an optional implementation manner of the device described in the third aspect, the second training module is specifically used to input the model structure description text of the large model and the first weight data of the large model into the privacy erasure model, and construct a loss function based on the model weight data predicted by the privacy erasure model and the second weight data of the large model to update the parameters of the privacy erasure model.
[0029] Specifically, the second training module is specifically used to construct the loss function according to the cross entropy of the predicted model weight data and the second weight data of the large model.
[0030] In a fourth aspect, a privacy erasure device is provided, the device comprising:
[0031] A third data acquisition module is configured to obtain weight data of the target large model after training based on the training task;
[0032] The privacy erasure module is configured to input the model structure description text of the target large model and the weight data into the privacy erasure model to obtain the weight data after privacy erasure; reconstruct the target large model according to the weight data after privacy erasure; the privacy erasure model is pre-trained according to the above-mentioned privacy erasure model training method.
[0033] As an optional implementation of the device described in the fourth aspect, the third data acquisition module is specifically used to obtain a third training sample set according to the training task of the target large model; use the third training sample set to train the target large model, and obtain the weight data of the trained target large model.
[0034] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program runs on an electronic device, the electronic device executes the above-mentioned privacy erasure model training method, or executes the above-mentioned privacy erasure method.
[0035] In a fifth aspect, an electronic device is provided, including:
[0036] at least one memory for storing a program;
[0037] At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above-mentioned privacy erasure model training method, or execute the above-mentioned privacy erasure method.
[0038] The beneficial effect of the privacy erasure model training method described in the embodiments of this specification is that the method directly modifies the parameters of the large model by training the privacy erasure model, fundamentally deleting the privacy data in the memory of the large model, and has extremely high security. This method also does not require retraining the large model, which greatly reduces the optimization cost of the large model. The privacy erasure model training device, privacy erasure method and device described in the embodiments of this specification also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0040] Figure 1 A flowchart of a method for training a privacy erasure model described in an embodiment of this specification is exemplarily shown.
[0041] Figure 2 The training framework of a privacy erasure model described in an embodiment of this specification is exemplarily shown.
[0042] Figure 3 A structural diagram of a training device for a privacy erasure model described in an embodiment of this specification is exemplarily shown.
[0043] Figure 4 A flowchart of a privacy erasure method described in an embodiment of this specification is exemplarily shown.
[0044] Figure 5 The structural diagram of a privacy erasing device described in an embodiment of this specification is exemplarily shown.
[0045] Figure 6 The structural diagram of an electronic device described in an embodiment of this specification is exemplarily shown. DETAILED DESCRIPTION
[0046] First of all, it should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise.
[0047] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0048] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.
[0049] Large Language Model (LLM) is a deep learning model trained on massive text data that can generate natural language text or understand the meaning of language text. Such models can perform a variety of natural language processing tasks, including but not limited to text classification, question answering, and conversation. As the model size grows, LLM is able to generate more accurate and coherent outputs while processing more complex and longer input sequences. In addition, larger models can cover a wider range of knowledge and language contexts, thereby providing more comprehensive and targeted answers and solutions.
[0050] The training of large models requires a large amount of training data, which contains a large amount of private data, especially personal privacy data. The most common of these is PII, or personally identifiable information, which refers to information that can identify an individual when used alone or with other related data, such as mobile phone number, ID number, driver's license, mailing address, etc.
[0051] In the application process of the big model, attackers can ask questions to the big model and then extract personal privacy data from the answers, thus causing personal privacy leakage. To avoid this problem, the industry currently generally adopts two solutions:
[0052] One method is to desensitize or directly delete the personal privacy data in the training data, and then retrain the large model. However, this method requires retraining the large model, which is extremely costly.
[0053] Another method is to perform content filtering in the request and response stages of the large model, blocking or desensitizing malicious questions and responses containing private information, but this method can be easily bypassed by attackers by modifying the attack method.
[0054] In summary, there is an urgent need in the existing technology for a safe, effective and low-cost large model privacy leakage solution.
[0055] In view of this, one or more embodiments of the present specification propose a privacy erasure model training method, a privacy erasure method and a device, which can solve the problem of privacy leakage of large models at a very low cost.
[0056] The following will further describe in detail the privacy erasure model training method, privacy erasure method and device described in one or more embodiments of this specification in combination with the drawings and specific embodiments of the specification, but this detailed description does not constitute a limitation on the embodiments of this specification.
[0057] Please refer to Figure 1 , Figure 1 The present invention is a flowchart of a method for training a privacy erasure model proposed in one or more embodiments of the present specification. Figure 1 The training method shown can be performed by a training device for a privacy erasure model, which can be deployed on a device terminal or on a server. The device terminal can be a smart device such as a mobile phone, a tablet computer, a desktop computer, a portable notebook, etc. The server can be an independent server or a server cluster composed of multiple servers.
[0058] like Figure 1 As shown, in some embodiments, the training method of the privacy erasure model may include steps S100 to S106:
[0059] S100: Obtain at least one large model to be trained.
[0060] S102: Obtain a first training sample set containing private data.
[0061] S104: Desensitizing the private data in the first training sample set to obtain a desensitized second training sample set.
[0062] S106: For each large model, use the first training sample set to train the large model, and obtain first weight data after the large model is trained.
[0063] S108: Using a completely consistent training method, using the second training sample set to train the large model, and obtaining second weight data after the large model is trained.
[0064] S110: Use the model structure description text of the large model and the first weight data of the large model as training samples, and use the second weight data of the large model as labels to train the privacy erasure model.
[0065] It can be seen that in Figure 1 In the training method of the privacy erasure model shown, the large model is trained on the original training data and the desensitized data of the original training data, respectively, wherein the training algorithm, number of rounds, etc. of the training process are completely consistent. The weights of the same large model on the original training data and the desensitized data are recorded to form a new erased data training set, which can be used to train a privacy erasure model to erase the weight data related to the privacy data in the large model.
[0066] The following is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings and specific embodiments. Figure 1 The method shown is described.
[0067] First, in step S100: obtain at least one large model to be trained.
[0068] The above-mentioned big model can be any open source big model. The type and number of open source big models can be adaptively selected according to needs, and this implementation does not impose any restrictions on this.
[0069] Next, in step S102 , a first training sample set containing private data is obtained.
[0070] Corresponding to the above-mentioned open source large model, the first training sample set here can also be the original training data composed of common open source data sets, which may include common personal privacy data types, such as mobile phone number, ID number, driver's license, mailing address, etc.
[0071] Next, in step S104 , the private data in the first training sample set is desensitized to obtain a desensitized second training sample set.
[0072] In some implementations, the type of private data to be desensitized may be determined first, and then a regular matching rule may be constructed according to the determined type of private data, and finally the private data in the first training sample set may be determined by regular matching. After finding the private data in the first training sample set, the private data may be desensitized by rewriting the data or directly deleting it, thereby obtaining the above-mentioned second training sample set.
[0073] Next, in step S106 : for each large model, the large model is trained using the first training sample set, and first weight data after the large model is trained is obtained.
[0074] The training tasks and task loss functions of the above-mentioned large model can be set according to the needs, such as identity recognition tasks, consistency comparison tasks, etc. Correspondingly, the above-mentioned first training sample set should also be a training sample set for the training task. When training the above-mentioned large model, the parameters of the above-mentioned large model are updated according to the set task loss function until the training is completed. At this time, the weight coefficient of the large model is the above-mentioned first weight data.
[0075] Next, for step S108: adopt a completely consistent training method, use the second training sample set to train the large model, and obtain the second weight data after the large model is trained.
[0076] In this step, the training of the large model is the same as the training process described in step S106 except that the training data set used is different, and the rest of the training process, such as the task loss function, training method, number of training rounds, etc. After the large model is trained based on the second training sample set until completion, the weight coefficient of the large model is the second weight data mentioned above.
[0077] Next, for step S110: the model structure description text of the large model and the first weight data of the large model are used as training samples, and the second weight data of the large model is used as a label to train the privacy erasure model.
[0078] Please refer to Figure 2 , Figure 2 A training framework for a privacy erasure model is shown. The network structure of the privacy erasure model can be selected according to the needs, and this embodiment does not limit this. For example, a convolutional neural network CNN can be used as a privacy erasure model. Figure 2 Let’s explain the training process of the privacy erasure model.
[0079] like Figure 2 As shown, the input of the privacy erasure model composed of CNN is the model structure description text and the first weight data of the large model, and the output of the privacy erasure model is the predicted weight coefficient of the large model. A loss function is constructed based on the difference between the predicted weight system and the second weight system of the large model, and the loss function is used to update the network parameters of CNN until the training is completed. At this time, a trained privacy erasure model is obtained.
[0080] In some embodiments, cross entropy can be used to characterize the difference between the predicted weight system and the second weight system of the large model, that is, the loss function can be the cross entropy of the predicted model weight data and the second weight data of the large model.
[0081] In some implementations, the above-mentioned model structure description text can be constructed manually.
[0082] Corresponding to the above-mentioned privacy erasure model training method, this embodiment also provides a privacy erasure model training device. Figure 3 , Figure 3 The exemplary embodiment shows a privacy erasure model training device, which can be used to implement the above-mentioned privacy erasure model training method. It should be noted that the privacy erasure model training method described in one or more embodiments of the present application can rely on Figure 3 The training device of the privacy erasure model shown is implemented, but not limited to this training device.
[0083] like Figure 3 As shown, the training device of the privacy erasure model includes:
[0084] The first data acquisition module 301 is configured to acquire at least one large model to be trained.
[0085] The second data acquisition module 302 is configured to acquire a first training sample set containing private data.
[0086] The desensitization module 303 is configured to perform desensitization processing on the private data in the first training sample set to obtain a desensitized second training sample set.
[0087] The first training module 304 is configured to, for each large model, train the large model using the first training sample set and obtain the first weight data after the large model is trained; and, adopt a completely consistent training method, train the large model using the second training sample set and obtain the second weight data after the large model is trained.
[0088] The second training module 305 is configured to use the model structure description text of the large model and the first weight data of the large model as training samples, and the second weight data of the large model as labels to train the privacy erasure model.
[0089] For the first data acquisition module 301, the big model it acquires can be any open source big model. The type and quantity of the open source big model can be adaptively set and selected according to the needs, and the present embodiment does not limit this.
[0090] For the second data acquisition module 302, the first training sample set acquired can be original training data composed of common open source data sets, which can include common personal privacy data types, such as mobile phone number, ID number, driver's license, mailing address, etc.
[0091] For the desensitization module 303, the desensitization module 303 can determine the privacy data in the first training sample set by regular matching according to the pre-constructed regular matching rules, and then desensitize the privacy data by rewriting or directly deleting the data, so as to obtain the above-mentioned second training sample set. The above-mentioned regular matching rules can be constructed according to the type of privacy data that needs to be desensitized.
[0092] For the first training module 304, the model is mainly used to use the first training sample set containing privacy data and the second training sample set not containing privacy data to respectively train the above-mentioned large model. Specifically, when the first training module 304 uses the first training sample set to train the above-mentioned large model, it can update the parameters of the above-mentioned large model according to the pre-set task loss function until the training is completed. At this time, the weight coefficient of the large model is the above-mentioned first weight data. Correspondingly, when the first training module 304 uses the second training sample set to train the above-mentioned large model, except for the different training data sets used, the rest of the training process, such as the task loss function, training method, number of training rounds, etc., are the same as the above-mentioned training process based on the first training sample set. After the training of the large model based on the second training sample set is completed, the weight coefficient of the large model is the above-mentioned second weight data.
[0093] It should be noted that when the first training module 304 uses the above two different training sample sets to train the same large model, the training can be performed sequentially or synchronously, and this embodiment does not impose any limitation on this.
[0094] For the second training module 305, the second training module 305 can input the model structure description text and the first weight data of the large model into the privacy erasure model, and then construct a loss function based on the difference between the weight system of the large model predicted by the privacy erasure model and the second weight coefficient of the large model, and use the loss function to update the network parameters of the privacy erasure model until the training is completed.
[0095] The structure of the privacy erasure model can be adaptively selected according to the needs. For example, in some embodiments, a convolutional neural network (CNN) can be used as a privacy erasure model.
[0096] In some embodiments, cross entropy can be used to characterize the difference between the predicted weight system and the second weight system of the large model, that is, the loss function can be the cross entropy of the predicted model weight data and the second weight data of the large model.
[0097] In some implementations, the above-mentioned model structure description text may be pre-constructed manually.
[0098] For the above-mentioned training device of a privacy erasure model, taking a module as an example of a software functional unit, the first data acquisition module 301 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first data acquisition module 301 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.
[0099] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0100] As an example of a hardware functional unit, the first data acquisition module 301 may include at least one computing device, such as a server, etc. Alternatively, the first data acquisition module 301 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
[0101] The multiple computing devices included in the first data acquisition module 301 can be distributed in the same region or in different regions. The multiple computing devices included in the first data acquisition module 301 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first data acquisition module 301 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0102] In other embodiments, the first data acquisition module 301 can be used to execute any step in the training method of the privacy erasure model, the second data acquisition module 302 can be used to execute any step in the training method of the privacy erasure model, the desensitization module 303 can be used to execute any step in the training method of the privacy erasure model, the first training module 304 can be used to execute any step in the training method of the privacy erasure model, and the second training module 305 can be used to execute any step in the training method of the privacy erasure model. The steps that the first data acquisition module 301, the second data acquisition module 302, the desensitization module 303, the first training module 304 and the second training module 305 are responsible for implementing can be specified as needed, and the first data acquisition module 301, the second data acquisition module 302, the desensitization module 303, the first training module 304 and the second training module 305 respectively implement different steps in the training method of the privacy erasure model to realize all the functions of the training device of the privacy erasure model.
[0103] In this implementation, the privacy erasure training device can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to realize the training function of the privacy erasure model.
[0104] Based on the privacy erasure model trained by the above-mentioned privacy erasure model training method, this embodiment also proposes a privacy erasure method, which can be executed by a privacy erasure device, which can be deployed on a device terminal or on a server. The device terminal can be a smart device such as a mobile phone, a tablet computer, a desktop computer, a portable notebook, etc. The server side can be an independent server or a server cluster composed of multiple servers.
[0105] like Figure 4 As shown, in some embodiments, the privacy erasure method may include steps S400 to S404:
[0106] S400: Obtain weight data of the target large model after training based on the training task.
[0107] The target large model in step S400 is a model that has been trained for its training task. The target large model can be trained in the following ways:
[0108] According to the training task of the target large model, a third training sample set is obtained.
[0109] The target large model is trained using the third training sample set until a target large model that meets preset conditions is obtained.
[0110] After the target large model is trained, the weight data of the target large model can be obtained
[0111] S402: Input the model structure description text and weight data of the target large model into the privacy erasure model to obtain the weight data after privacy erasure.
[0112] The privacy erasure model used in step S402 is pre-trained based on the above-mentioned privacy erasure model method, and the specific training process will not be repeated here.
[0113] The model structure description text of the target large model can be constructed and input manually.
[0114] S404: Reconstruct the target large model according to the model structure description text and the weight data after privacy erasure.
[0115] After obtaining the privacy-erased weight data, the privacy-erased weight data can be used to replace the original weight data of the target large model, thereby obtaining the privacy-erased target large model.
[0116] Corresponding to the above-mentioned privacy erasure method, this embodiment also provides a privacy erasure device. Figure 5 , Figure 5 An exemplary privacy erasure device is shown, which can be used to implement the above-mentioned privacy erasure method. It should be noted that a privacy erasure method described in one or more embodiments of the present application can rely on Figure 5 The privacy erasure device shown implements, but is not limited to, this device.
[0117] like Figure 5 As shown, the privacy erasing device includes:
[0118] The third data acquisition module 501 is configured to obtain weight data of the target large model after training based on the training task.
[0119] The privacy erasure module 502 is configured to input the model structure description text and weight data of the target large model into the privacy erasure model to obtain the weight data after privacy erasure; and reconstruct the target large model according to the weight data after privacy erasure.
[0120] Regarding the third data acquisition module 501, the third data acquisition module 501 can be specifically used to obtain a third training sample set according to the training task of the target large model; use the third training sample set to train the target large model, and obtain the weight data of the trained target large model.
[0121] Regarding the privacy erasure module 502, the privacy erasure model used by the privacy erasure module 502 is pre-trained based on the above-mentioned privacy erasure model method, and the specific training process is not repeated here.
[0122] The model structure description text of the target large model can be constructed manually and input into the privacy erasure module 502 .
[0123] After obtaining the weight data after privacy erasure, the privacy erasure module 502 may replace the original weight data of the target large model with the weight data after privacy erasure, thereby obtaining the target large model after privacy erasure.
[0124] For the above-mentioned privacy erasure device, taking a module as an example of a software functional unit, the third data acquisition module 501 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the third data acquisition module 501 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.
[0125] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0126] As an example of a hardware functional unit, the third data acquisition module 501 may include at least one computing device, such as a server, etc. Alternatively, the third data acquisition module 501 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
[0127] The multiple computing devices included in the third data acquisition module 501 can be distributed in the same region or in different regions. The multiple computing devices included in the third data acquisition module 501 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the third data acquisition module 501 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0128] In other embodiments, the third data acquisition module 501 can be used to execute any step in the above-mentioned privacy erasure method, and the privacy erasure module 502 can be used to execute any step in the above-mentioned privacy erasure method. The steps that the third data acquisition module 501 and the privacy erasure module 502 are responsible for implementing can be specified as needed, and the third data acquisition module 501 and the privacy erasure module 502 respectively implement different steps in the above-mentioned privacy erasure method to realize all the functions of the above-mentioned privacy erasure device.
[0129] In this implementation, the privacy erasing device can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to implement the privacy erasing function.
[0130] In some embodiments, an electronic device is also provided. Figure 6 , the electronic device includes: a bus 601, a processor 602, a memory 603 and a communication interface 604. The processor 602, the memory 603 and the communication interface 604 communicate with each other through the bus 601. The electronic device can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the electronic device.
[0131] The bus 601 may be a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The use of only one line does not mean that there is only one bus or one type of bus. Bus 601 may include a path for transmitting information between various components of the electronic device (eg, processor 602, memory 603, and communication interface 604).
[0132] The processor 602 may include any one or more of a processor such as a CPU, a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0133] The memory 603 may include a volatile memory, such as a random access memory (RAM). The memory 603 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0134] The memory 603 stores executable program codes, and the processor 602 executes the executable program codes to respectively implement the functions of the first data acquisition module 301, the second data acquisition module 302, the desensitization module 303, the first training module 304, and the second training module 305, that is, to implement the functions of the training device of the privacy erasure model, thereby implementing the training method of the privacy erasure model. Alternatively, the processor 602 executes the executable program codes to respectively implement the functions of the third data acquisition module 501 and the privacy erasure module 502, that is, to implement the functions of the privacy erasure device, thereby implementing the privacy erasure method.
[0135] The communication interface 604 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the electronic device and other devices or a communication network.
[0136] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned privacy erasure model training method, or implements the above-mentioned privacy erasure method.
[0137] The computer-readable storage medium may be any available medium that can be stored in an electronic device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the electronic device to execute the above-mentioned privacy erasure model training method, or to execute the above-mentioned privacy erasure method.
[0138] It is to be understood that the structure illustrated in the embodiments of this specification does not constitute a specific limitation on the system of the embodiments of this specification. In other embodiments of the specification, the above system may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0139] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0140] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0141] It should be noted that the above examples are only specific embodiments of the present invention, and the present invention is obviously not limited to the above examples, and there are many similar variations. All variations directly derived or associated from the contents disclosed by the technicians in this field should fall within the protection scope of the present invention.
Claims
1. A privacy erasure method, comprising: Obtain the weight data of the target large model after training based on the training task; Inputting the model structure description text of the target large model and the weight data into a privacy erasure model to obtain the weight data after privacy erasure; Reconstructing the target large model according to the model structure description text and the weight data after privacy erasure; The training method of the privacy erasure model includes: obtaining at least one large model to be trained; obtaining a first training sample set containing privacy data; desensitizing the privacy data in the first training sample set to obtain a desensitized second training sample set; for each of the large models, training the large model using the first training sample set, and obtaining the first weight data of the large model after training; using a completely consistent training method, training the large model using the second training sample set, and obtaining the second weight data of the large model after training; using the model structure description text of the large model and the first weight data of the large model as training samples, and using the second weight data of the large model as a label to train the privacy erasure model.
2. The method according to claim 1, wherein training the privacy erasure model comprises: The model structure description text of the large model and the first weight data of the large model are input into the privacy erasure model, and a loss function is constructed based on the model weight data predicted by the privacy erasure model and the second weight data of the large model to update the parameters of the privacy erasure model.
3. The method as claimed in claim 2, wherein the loss function is constructed based on the cross entropy of the predicted model weight data and the second weight data of the large model.
4. The method according to claim 1, obtaining weight data of the target large model after training based on the training task, specifically comprises: According to the training task of the target large model, a third training sample set is obtained; Using the third training sample set to train the target large model; Obtain the weight data of the trained target large model.
5. A privacy erasing device, comprising: A first data acquisition module, configured to acquire at least one large model to be trained; A second data acquisition module is configured to acquire a first training sample set containing private data; a desensitization module configured to perform desensitization processing on the privacy data in the first training sample set to obtain a desensitized second training sample set; A first training module is configured to train each of the large models using the first training sample set and obtain first weight data of the large model after training; And, using a completely consistent training method, using the second training sample set to train the large model, and obtaining second weight data after the large model is trained; A second training module is configured to use the model structure description text of the large model and the first weight data of the large model as training samples, and the second weight data of the large model as labels to train a privacy erasure model; A third data acquisition module is configured to obtain weight data of the target large model after training based on the training task; The privacy erasure module is configured to input the model structure description text of the target large model and the weight data into the privacy erasure model to obtain the weight data after privacy erasure; and reconstruct the target large model according to the weight data after privacy erasure.
6. In the device as described in claim 5, the third data acquisition module is specifically used to obtain a third training sample set according to the training task of the target large model; use the third training sample set to train the target large model, and obtain the weight data of the trained target large model.
7. In the device as described in claim 5, the second training module is specifically used to input the model structure description text of the large model and the first weight data of the large model into the privacy erasure model, and construct a loss function based on the model weight data predicted by the privacy erasure model and the second weight data of the large model to update the parameters of the privacy erasure model.
8. In the device as described in claim 7, the second training module is specifically used to construct the loss function based on the cross entropy of the predicted model weight data and the second weight data of the large model.
9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 4.
10. An electronic device comprising: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 4.