Pre-training model fine tuning method, application method and device for data protection

Abstract

The invention discloses a pre-training model fine tuning method for data protection, and an application method and device. The method comprises the following steps: creating an agent model for data protection; according to a preset face image set and the description text thereof, performing iteration fine tuning of a preset time step on the proxy model to obtain a proxy personalized generation model; attacking the proxy personalized generation model by using a preset adversarial sample set and the description text to obtain a disturbance value; adding the disturbance value to a preset confrontation sample set, and continuing to execute the step of attacking the proxy personalized generation model until the number of attacks reaches a preset number threshold, and obtaining a target confrontation sample; and using the target adversarial sample to carry out iteration fine tuning of a preset time step on the proxy model again, and obtaining a final proxy personalized generation model under the condition that the current fine tuning frequency reaches a preset disturbance threshold. By adopting the embodiment of the invention, the calculation demand in an actual training scene is less, and the waste of calculation resources is reduced.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a pre-trained model fine-tuning method, application method and device for data protection. Background Art

[0002] In the digital age, PCS (Personalized Content Synthesis) technology plays an increasingly important role in the field of text-to-image generation. Users provide a short description text and use PCS technology to generate high-quality images related to a specific topic or object.

[0003] DM (Diffusion Model) is a mainstream method to implement PCS technology. Users provide images related to SoI (Subject of Interest), and use algorithms such as DreamBooth to fine-tune the pre-trained diffusion model to generate new images containing SoI. Malicious users can use this technology to generate realistic synthetic images and spread false information through social media platforms, posing a threat to personal safety and privacy.

[0004] In the related art, in order to solve the privacy and security issues caused by the malicious use of facial images in PCS, Anti-DreamBooth is proposed in the prior art to guide the model to generate unrelated results. This method interferes with DreamBooth by learning an alternative model, thereby enhancing the protection effect of personal images. However, the current Anti-DreamBooth training requires the construction of prior class samples and all parameters of the training model. At the same time, it currently requires a large number of interferences to complete the learning, resulting in large computing requirements and high computing costs in actual training scenarios. Summary of the invention

[0005] The embodiments of the present application provide a pre-trained model fine-tuning method, application method and device for data protection. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0006] In a first aspect, an embodiment of the present application provides a pre-trained model fine-tuning method for data protection, the method comprising: Create a proxy model for data protection, which is obtained by disabling all parameters in the pre-trained diffusion model except the parameters of the text encoder and the cross-attention layer of the U-Net structure; According to the preset face image set and its description text, the proxy model is iteratively fine-tuned at a preset time step to obtain a proxy personalized generation model; Use the preset adversarial sample set and description text to attack the agent personalized generation model to obtain the perturbation value, where the preset adversarial sample set is obtained by preprocessing the preset face image set, and the perturbation step length when calculating the perturbation value is calculated according to the number of time steps; Add the perturbation value to the preset adversarial sample set, and continue to execute the steps of attacking the agent personalized generation model until the number of attacks reaches the preset threshold, and then obtain the target adversarial sample; Using the target adversarial sample, the proxy model is iteratively fine-tuned for a preset time step again. When the current number of fine-tuning times reaches the preset perturbation threshold, the final proxy personalized generation model is obtained.

[0007] Optionally, use a preset adversarial sample set and description text to attack the agent personalized generation model to obtain perturbation values, including: Input the preset adversarial sample set and the description text into the agent personalized generation model to calculate the perturbation value; Output the processing results corresponding to the preset adversarial sample set; Gets the disturbance value included in the processing result.

[0008] Optionally, a preset adversarial sample set and a description text are input into the agent personalized generation model to calculate the perturbation value, including: The agent personalized generation model uses the fractional distillation sampling algorithm to calculate and process the preset adversarial sample set and description text to obtain the current gradient; The agent personalization generation model obtains the time step number; The agent personalized generation model calculates the perturbation step size based on the number of time steps; The agent personalized generation model calculates the perturbation value based on the current gradient and perturbation step size.

[0009] Optionally, the perturbation step size is calculated as:

[0010] in, is the perturbation step length, It is a hyperparameter used to adjust the step size and can be adjusted according to the needs of the model and the performance during training. In the time step The noise intensity at this time is related to the noise variance of the diffusion model. is the number of time steps, which is used to dynamically adjust the step size so that the update amplitude gradually decreases as the time step increases; The calculation formula of the disturbance value is:

[0011] in, is the disturbance value, is a scaling factor of 0.1, which is used to control the strength of the adversarial perturbation. is the perturbation step length, is the sign function of the gradient of the loss function, which returns the sign of the gradient vector (positive or negative). In adversarial attacks, the sign of the gradient is used to determine the direction of the perturbation. is the current gradient obtained using the fractional distillation sampling algorithm.

[0012] Optionally, the current gradient calculation formula is:

[0013] in, Express The gradient operator is used to calculate the loss function L relative to the input preset adversarial sample set image The gradient of Indicated in the model function Next, image After disturbance The loss function after ; Indicates the first The image of the iteration, represents the loss function calculated using the fractional distillation sampling algorithm, Represents the expected value operator, which is used to calculate the The average value of the gradient under the combination, represents the original input image, before iteration and perturbation, is the number of time steps, is the set of model parameters, is the number of time steps Add to image The actual noise in Indicates that the model is based on the current model parameter set , time steps and description text The predicted noise, Indicates the number of time steps When Relative to The gradient of is the number of time steps The output of the model is is the number of time steps The input of the model.

[0014] Optionally, the loss function of the proxy model is:

[0015] in, is the loss function, which is used to measure the given model parameters and preset face image sets The loss value calculated under the condition of is the expected value operator, which is used to calculate all possible The average value of the loss function under the combination, is the number of time steps, is the set of model parameters, is the number of time steps Add to image The actual noise in Indicates that the model is based on the current model parameter set , time steps and description text Predicted noise.

[0016] Optionally, the preset time step is 3 steps, the preset number threshold is greater than 6, the preset disturbance threshold is 50 times, and the fine-tuning method of the proxy model adopts the training method of the DreamBooth algorithm.

[0017] In a second aspect, an embodiment of the present application provides a model application method, the method comprising: receiving an image processing request, the image processing request carrying a user's face image; The face image is input into the final proxy personalized generation model, and the target image with added privacy protection information is output. The privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest. The final proxy personalized generation model is fine-tuned by the pre-trained model fine-tuning method for data protection. Display the target image.

[0018] In a third aspect, an embodiment of the present application provides a pre-trained model fine-tuning device for data protection, the device comprising: Create a module for creating a proxy model for data protection. The proxy model is obtained by disabling all parameters in the pre-trained diffusion model except the parameters of the text encoder and the cross-attention layer of the U-Net structure; A first fine-tuning module is used to iteratively fine-tune the proxy model at a preset time step according to a preset face image set and its description text, so as to obtain a proxy personalized generation model; A first attack module is used to attack the agent personalized generation model using a preset adversarial sample set and a description text to obtain a perturbation value, wherein the preset adversarial sample set is obtained by preprocessing a preset face image set, and the perturbation step length when calculating the perturbation value is calculated according to the number of time steps; The second attack module is used to add the perturbation value to the preset adversarial sample set, and continue to execute the step of attacking the agent personalized generation model until the number of attacks reaches a preset number threshold, thereby obtaining a target adversarial sample; The second fine-tuning module is used to use the target adversarial sample to iteratively fine-tune the proxy model for a preset time step again, and obtain the final proxy personalized generation model when the current number of fine-tuning times reaches a preset disturbance threshold.

[0019] In a fourth aspect, an embodiment of the present application provides a model application device, the device comprising: A receiving module, used for receiving an image processing request, wherein the image processing request carries a user's face image; An input module is used to input a face image into the final proxy personalized generation model and output a target image with privacy protection information added. The privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest. The final proxy personalized generation model is fine-tuned by a pre-trained model fine-tuning method for data protection. The display module is used to display the target image.

[0020] The technical solution provided by the embodiments of the present application may have the following beneficial effects: In an embodiment of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross-attention layer of the U-Net structure, the model parameters are greatly reduced, and the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and image that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, so that the computing requirements in actual training scenarios are small, and the waste of computing resources is reduced.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] Figure 1 It is a flowchart of a pre-trained model fine-tuning method for data protection provided in an embodiment of the present application; Figure 2A It is a schematic diagram of a technical architecture for fine-tuning a pre-trained model for data protection provided in an embodiment of the present application; Figure 2B It is a schematic diagram of another technical architecture for fine-tuning a pre-trained model for data protection provided in an embodiment of the present application; Figure 3 It is a flow chart of a model application method provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of a pre-trained model fine-tuning device for data protection provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of a model application device provided in an embodiment of the present application; Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0025] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0026] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0027] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0028] The present application provides a method, application method and device for fine-tuning a pre-trained model for data protection to solve the problems existing in the above-mentioned related technical problems. In the embodiment of the present application, by disabling other parameters in the pre-trained diffusion model except the parameters of the text encoder and the cross-attention layer of the U-Net structure, the model parameters are greatly reduced, and the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and image that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, so that the computing requirements in the actual training scenario are small and the waste of computing resources is reduced. The following is a detailed description using an exemplary embodiment.

[0029] The present application provides a pre-training model fine-tuning method, application method and device for data protection to solve the problems existing in the above-mentioned related technical issues. Figure 1 -Attached Figure 3 , a method for fine-tuning a pre-trained model for data protection provided in an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a pre-trained model fine-tuning device for data protection based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0030] See also Figure 1 , a flowchart of a method for fine-tuning a pre-trained model for data protection is provided in an embodiment of the present application, which is applied to a client, the client including a private data set, a general model for federated learning, and a local model, the general model is a model shared by all clients, and the local model is trained for a local service. Figure 1 As shown, the method of the embodiment of the present application may include the following steps: S101, creating a proxy model for data protection, where the proxy model is obtained by disabling all parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross attention layer of the U-Net structure; Among them, the pre-trained diffusion model is obtained by training the DM (Diffusion Model) in advance with a large amount of data. This model already exists, such as the Stable Diffusion model, and can also be retrained. A text encoder is a model component that converts text data into a numerical representation (usually in vector form). In the text-to-image generation task, the text encoder is used to convert the text description into a form that can be combined with image features. U-Net is a convolutional neural network architecture commonly used in image processing tasks (such as segmentation, generation, etc.). It has a special U-shaped structure that can effectively combine the contextual information and location information of the image.

[0031] In some embodiments of the present application, a pre-trained diffusion model is selected, such as a StableDiffusion model, which has been trained on a large amount of image and text data. In this model, except for the parameters of the text encoder and the cross-attention layer in the U-Net structure, all other parameters are set to be untrainable (i.e., their updates are disabled), which is achieved by setting the weights of these layers to fixed values ​​or using a specific training configuration. At this point, a proxy model is obtained that retains the capabilities of the text encoding and cross-attention mechanisms, but the other parts remain unchanged.

[0032] S102, iteratively fine-tuning the proxy model at a preset time step according to a preset face image set and its description text, to obtain a proxy personalized generation model; Among them, the preset face image set refers to a set of pre-selected face images, which will be used to fine-tune the proxy model so that it can generate personalized content similar to these images. The description text is the text description associated with the preset face image set, which is used to guide the model to generate face images that match specific text descriptions. These texts usually contain descriptions of the image content, such as "a smiling young man". In the diffusion model, the preset number of time steps usually refers to the fixed number of time steps required for the model to recover from a noisy image to a clear image during the reverse diffusion process. Iterative fine-tuning refers to gradually adjusting the model parameters through multiple iterations during the model training process so that the model can better adapt to specific training data.

[0033] In some embodiments of the present application, the specific process of iteratively fine-tuning the proxy model at a preset time step according to the preset face image set and its description text to obtain the proxy personalized generation model includes: obtaining the preset face image set and its description text, pre-processing the preset face image set, and uniformly scaling it to a uniform fixed size Afterwards, a clean reference image set is constructed , using the clean image set And the description text, combined with the current Dreambooth training method, fine-tuned the pre-trained diffusion model for 3 steps to obtain the agent personalized generation model. The model parameter set of the pre-trained diffusion model is .

[0034] Specifically, the loss function of the proxy model is:

[0035] in, is the loss function, which is used to measure the given model parameters and preset face image sets The loss value calculated under the condition of is the expected value operator, which is used to calculate all possible The average value of the loss function under the combination, is the number of time steps, is the set of model parameters, is the number of time steps Add to image The actual noise in Indicates that the model is based on the current model parameter set , time steps and description text Predicted noise.

[0036] S103, using a preset adversarial sample set and a description text to attack the agent personalized generation model to obtain a perturbation value, wherein the preset adversarial sample set is obtained by preprocessing a preset face image set, and the perturbation step length when calculating the perturbation value is calculated according to the number of time steps; Among them, the preset adversarial sample set is obtained by preprocessing the preset face image set. These samples are usually created by adding carefully calculated perturbations to the original images. The purpose is to deceive the model to make wrong predictions or classifications without significantly changing the appearance of the image. The perturbation value refers to the small changes or noise added to the original image in the adversarial sample. These changes are designed to maximize the changes in the model output while maintaining the invariance of the image to human observers. Preprocessing refers to a series of processing steps performed on the image. This may include operations such as scaling, cropping, and normalization, which are designed to make the image data suitable for the input requirements of the model. The perturbation step size refers to the amplitude or scale of the perturbation when calculating the perturbation value. In adversarial attacks, dynamically adjusting the perturbation step size is a common strategy that can be used to control the intensity of the perturbation to more effectively deceive the model. The step size may be adjusted according to the time step. The time step number usually refers to the step count in the iteration process. In the context of adversarial attacks, the time step number can be used to dynamically adjust the perturbation step size to adapt to the sensitivity of the model at different training stages.

[0037] Among them, the preset face image set is preprocessed and scaled to a uniform fixed size Constructing adversarial sample sets .

[0038] In some embodiments of the present application, a preset adversarial sample set and description text are used to attack the agent personalized generation model, and the specific process of obtaining the disturbance value includes: inputting the preset adversarial sample set and description text into the agent personalized generation model to calculate the disturbance value; outputting the processing result corresponding to the preset adversarial sample set; and obtaining the disturbance value included in the processing result.

[0039] Specifically, the specific process of inputting the preset adversarial sample set and the description text into the agent personalized generation model to calculate the perturbation value includes: the agent personalized generation model uses the fractional distillation sampling algorithm to calculate and process the preset adversarial sample set and the description text to obtain the current gradient; obtains the number of time steps; calculates the perturbation step length according to the number of time steps; calculates the perturbation value according to the current gradient and the perturbation step length.

[0040] Specifically, the perturbation step length calculation formula is:

[0041] in, is the perturbation step length, It is a hyperparameter used to adjust the step size and can be adjusted according to the needs of the model and the performance during training. In the time step The noise intensity at this time is related to the noise variance of the diffusion model. is the number of time steps, which is used to dynamically adjust the step size so that the update amplitude gradually decreases as the time step increases; The calculation formula of the disturbance value is:

[0042] in, is the disturbance value, is a scaling factor of 0.1, which is used to control the strength of the adversarial perturbation. is the perturbation step length, is the sign function of the gradient of the loss function, which returns the sign of the gradient vector (positive or negative). In adversarial attacks, the sign of the gradient is used to determine the direction of the perturbation. is the current gradient obtained using the fractional distillation sampling algorithm.

[0043] Specifically, the current gradient calculation formula is:

[0044] in, Express The gradient operator is used to calculate the loss function L relative to the input preset adversarial sample set image The gradient of Indicated in the model function Next, image After disturbance The loss function after ; Indicates the first The image of the iteration, represents the loss function calculated using the fractional distillation sampling algorithm, Represents the expected value operator, which is used to calculate the The average value of the gradient under the combination, represents the original input image, before iteration and perturbation, is the number of time steps, is the set of model parameters, is the number of time steps Add to image The actual noise in Indicates that the model is based on the current model parameter set , time steps and description text The predicted noise, Indicates the number of time steps When Relative to The gradient of is the number of time steps The output of the model is is the number of time steps The input of the model.

[0045] S103, adding the disturbance value to the preset adversarial sample set, and continuing to execute the step of attacking the agent personalized generation model until the number of attacks reaches a preset number threshold, thereby obtaining a target adversarial sample; Among them, the target adversarial sample refers to the final adversarial sample obtained after a series of adversarial attack steps, which can successfully deceive or mislead the machine learning model and make it produce wrong output or prediction.

[0046] In some embodiments of the present application, the calculated perturbation value is added to each image in the adversarial sample set to generate a new adversarial sample set. Continue to perform the attack step, using the updated adversarial sample set each time. In each iteration, adjust the perturbation value according to the response of the model. Record the number of attacks and ensure that the number of attacks does not exceed the preset number threshold. When the number of attacks reaches the preset number threshold, stop the attack process. The adversarial sample set obtained at this time is the target adversarial sample set.

[0047] S104, using the target adversarial sample, iteratively fine-tuning the proxy model for a preset time step again, and obtaining the final proxy personalized generation model when the current number of fine-tuning times reaches a preset disturbance threshold.

[0048] In some embodiments of the present application, the target adversarial sample is used, and combined with the current Dreambooth training method, three steps are fine-tuned on the pre-trained diffusion model to regain the agent personalized generation model. The model parameter set of the pre-trained diffusion model is When the current number of fine-tuning times reaches the preset disturbance threshold, the final agent personalized generation model is obtained. Otherwise, the process returns to step S102 for fine-tuning.

[0049] Specifically, the preset time step is 3 steps, the preset number threshold is greater than 6, the preset disturbance threshold is 50 times, and the fine-tuning method of the proxy model adopts the training method of the DreamBooth algorithm.

[0050] For example Figure 2A , 2B As shown, it is a technical architecture diagram of a pre-trained model fine-tuning for data protection provided by the present application, including two parts: agent model training and attack. When training the agent model, the cross-attention layer of the preset face image set and text prompt word input model and the text encoder are used for training. After the training, the agent personalized generation model is obtained. Then, the adversarial sample is used to attack the agent personalized generation model to obtain the target adversarial sample, and finally the target adversarial sample is used to train the agent personalized generation model. After the training conditions meet the preset conditions, the final agent personalized generation model can be obtained.

[0051] For example, define is a collection of face images that need to be protected. For each image in the collection , adding an adversarial perturbation and post the modified image , while preserving the privacy of the original image. The collection of images released is called A malicious user could collect a small collection of images containing Then use this set as the reference image to fine-tune the diffusion model , according to the specific PCS algorithm, to obtain the optimal hyperparameters The overall goal is to optimize adversarial noise:

[0052] in It is the loss function used to evaluate the performance of adversarial examples. The loss function of the diffusion model is usually used: The noise limit is about Paradigm metric sphere. Existing methods widely use Projected Gradient Descent (PGD) to iteratively optimize adversarial examples. The process follows the following formula:

[0053]

[0054] where x is the input image, is a sign function, is the loss function relative to gradient. represents the step size of each iteration, and k is the number of iterations.

[0055] In an embodiment of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross-attention layer of the U-Net structure, the model parameters are greatly reduced, and the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and image that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, so that the computing requirements in actual training scenarios are small, and the waste of computing resources is reduced.

[0056] See also Figure 3 , which is a flow chart of a model application method provided in the embodiment of the present application. Figure 3 As shown, the method of the embodiment of the present application may include the following steps: S201, receiving an image processing request, the image processing request carrying a user's face image; Among them, an image processing request refers to a request from a user to the system to process a specific image, which involves using a certain model to modify the image. A face image is a specific image provided by a user, usually containing face information, for further processing or analysis.

[0057] In some embodiments of the present application, users upload their face images through a website or application and request image processing services. After the system receives the face image, it passes it as input to the final agent personalized generation model.

[0058] S202, inputting the face image into the final proxy personalized generation model, and outputting a target image with added privacy protection information, where the privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest; the final proxy personalized generation model is obtained by fine-tuning the pre-trained model fine-tuning method for data protection; The proxy personalized generation model is a fine-tuned model that can generate new images based on the user's face image, taking into account the user's personalized needs. Privacy-preserving information refers to specific information or disturbances added during image processing to prevent the model from generating images that may violate privacy or be abused.

[0059] In some embodiments of the present application, the proxy personalized generation model analyzes a face image and generates a new image that is visually similar to the original image but contains privacy protection information to prevent abuse. The target image output by the model is processed and contains privacy protection information to prevent other systems using the diffusion model from generating images related to the user's face.

[0060] S203, display the target image.

[0061] Among them, the target image refers to the final image after processing, which not only contains the content of the original image, but also adds privacy protection information to prevent abuse.

[0062] In some embodiments of the present application, the system displays the generated target image to the user, and the user can see the image effect after the privacy protection information is added.

[0063] In an embodiment of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross-attention layer of the U-Net structure, the model parameters are greatly reduced, and the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and image that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, so that the computing requirements in actual training scenarios are small, and the waste of computing resources is reduced.

[0064] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0065] See also Figure 4 , which shows a schematic diagram of the structure of a pre-trained model fine-tuning device for data protection provided by an exemplary embodiment of the present application. The pre-trained model fine-tuning device for data protection can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device includes a private data set, a general model deployed for federated learning, and a local model. The general model is a model shared by all clients, and the local model is trained for local services. The device 1 includes a creation module 10, a first fine-tuning module 20, a first attack module 30, a second attack module 40, and a second fine-tuning module 50.

[0066] Creating module 10, for creating a proxy model for data protection, the proxy model is obtained by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross attention layer of the U-Net structure; A first fine-tuning module 20, configured to iteratively fine-tune the proxy model at a preset time step according to a preset set of face images and their description texts, to obtain a proxy personalized generation model; A first attack module 30 is used to attack the agent personalized generation model using a preset adversarial sample set and a description text to obtain a disturbance value, wherein the preset adversarial sample set is obtained by preprocessing a preset face image set, and the disturbance step length when calculating the disturbance value is calculated according to the number of time steps; The second attack module 40 is used to add the perturbation value to the preset adversarial sample set, and continue to execute the step of attacking the agent personalized generation model until the number of attacks reaches a preset number threshold, thereby obtaining a target adversarial sample; The second fine-tuning module 50 is used to use the target adversarial sample to iteratively fine-tune the proxy model for a preset time step again, and obtain the final proxy personalized generation model when the current number of fine-tuning times reaches a preset disturbance threshold.

[0067] It should be noted that the pre-trained model fine-tuning device for data protection provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the pre-trained model fine-tuning method for data protection. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the pre-trained model fine-tuning device for data protection provided in the above embodiment and the pre-trained model fine-tuning method for data protection belong to the same concept, and its implementation process is detailed in the method embodiment, which will not be repeated here.

[0068] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0069] In an embodiment of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross-attention layer of the U-Net structure, the model parameters are greatly reduced, and the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and image that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, so that the computing requirements in actual training scenarios are small, and the waste of computing resources is reduced.

[0070] See also Figure 5 , which shows a schematic diagram of the structure of a model application device provided by an exemplary embodiment of the present application. The pre-trained model fine-tuning device for data protection can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device 2 includes a receiving module 60, an input module 70, and a display module 80.

[0071] A receiving module 60, configured to receive an image processing request, wherein the image processing request carries a user's face image; An input module 70 is used to input the face image into the final proxy personalized generation model, and output a target image with privacy protection information added, where the privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest; the final proxy personalized generation model is fine-tuned by the above-mentioned pre-trained model fine-tuning method for data protection; The display module 80 is used to display the target image.

[0072] It should be noted that the pre-trained model fine-tuning device for data protection provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the pre-trained model fine-tuning method for data protection. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the pre-trained model fine-tuning device for data protection provided in the above embodiment and the pre-trained model fine-tuning method for data protection belong to the same concept, and its implementation process is detailed in the method embodiment, which will not be repeated here.

[0073] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0074] In an embodiment of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross-attention layer of the U-Net structure, the model parameters are greatly reduced, and the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and image that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, so that the computing requirements in actual training scenarios are small, and the waste of computing resources is reduced.

[0075] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the pre-trained model fine-tuning method for data protection provided by the above-mentioned various method embodiments.

[0076] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the pre-trained model fine-tuning method for data protection of each of the above-mentioned method embodiments.

[0077] See also Figure 6 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0078] The communication bus 1002 is used to realize the connection and communication between these components.

[0079] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0080] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0081] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a model application (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0082] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 6 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a pre-trained model fine-tuning application for data protection.

[0083] exist Figure 6 In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call the pre-trained model fine-tuning application for data protection stored in the memory 1005, and specifically perform the following operations: Create a proxy model for data protection, which is obtained by disabling all parameters in the pre-trained diffusion model except the parameters of the text encoder and the cross-attention layer of the U-Net structure; According to the preset face image set and its description text, the proxy model is iteratively fine-tuned at a preset time step to obtain a proxy personalized generation model; Use the preset adversarial sample set and description text to attack the agent personalized generation model to obtain the perturbation value, where the preset adversarial sample set is obtained by preprocessing the preset face image set, and the perturbation step length when calculating the perturbation value is calculated according to the number of time steps; Add the perturbation value to the preset adversarial sample set, and continue to execute the steps of attacking the agent personalized generation model until the number of attacks reaches the preset threshold, and then obtain the target adversarial sample; Using the target adversarial sample, the proxy model is iteratively fine-tuned for a preset time step again. When the current number of fine-tuning times reaches the preset perturbation threshold, the final proxy personalized generation model is obtained.

[0084] In one embodiment, when the processor 1001 uses the preset adversarial sample set and the description text to attack the agent personalized generation model and obtain the disturbance value, the processor 1001 specifically performs the following operations: Input the preset adversarial sample set and the description text into the agent personalized generation model to calculate the perturbation value; Output the processing results corresponding to the preset adversarial sample set; Gets the disturbance value included in the processing result.

[0085] In one embodiment, when the processor 1001 inputs the preset adversarial sample set and the description text into the agent personalized generation model to calculate the disturbance value, the following operations are specifically performed: The agent personalized generation model uses the fractional distillation sampling algorithm to calculate and process the preset adversarial sample set and description text to obtain the current gradient; The agent personalization generation model obtains the time step number; The agent personalized generation model calculates the perturbation step size based on the number of time steps; The agent personalized generation model calculates the perturbation value based on the current gradient and perturbation step size.

[0086] In an embodiment of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross-attention layer of the U-Net structure, the model parameters are greatly reduced, and the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and image that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, so that the computing requirements in actual training scenarios are small, and the waste of computing resources is reduced.

[0087] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program for fine-tuning the pre-trained model for data protection can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0088] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A pre-trained model fine-tuning method for data protection, characterized in that: The method comprises: Creating a proxy model for data protection, wherein the proxy model is obtained by disabling all parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross-attention layer of the U-Net structure; According to a preset set of face images and their description texts, the proxy model is iteratively fine-tuned at a preset time step to obtain a proxy personalized generation model; Using a preset adversarial sample set and the description text, the agent personalized generation model is attacked to obtain a disturbance value, wherein the preset adversarial sample set is obtained by preprocessing the preset face image set, and the disturbance step length when calculating the disturbance value is calculated according to the number of time steps; Adding the disturbance value to the preset adversarial sample set, and continuing to execute the step of attacking the agent personalized generation model until the number of attacks reaches a preset number threshold, thereby obtaining a target adversarial sample; The target adversarial sample is used to iteratively fine-tune the proxy model for a preset time step again, and when the current number of fine-tuning times reaches a preset disturbance threshold, a final proxy personalized generation model is obtained.

2. The method according to claim 1, characterized in that The using of the preset adversarial sample set and the description text to attack the agent personalized generation model to obtain a disturbance value includes: Inputting the preset adversarial sample set and the description text into the agent personalized generation model to calculate the disturbance value; Output the processing result corresponding to the preset adversarial sample set; A disturbance value included in the processing result is obtained.

3. The method according to claim 2, characterized in that The step of inputting the preset adversarial sample set and the description text into the agent personalized generation model to calculate the disturbance value includes: The agent personalized generation model uses a fractional distillation sampling algorithm to calculate and process the preset adversarial sample set and the description text to obtain a current gradient; The agent personalized generation model obtains the number of time steps; The agent personalized generation model calculates the perturbation step length according to the time step number; The agent personalized generation model calculates a disturbance value according to the current gradient and the disturbance step size.

4. The method according to claim 3, characterized in that The perturbation step length calculation formula is: in, is the perturbation step length, It is a hyperparameter used to adjust the step size and can be adjusted according to the needs of the model and the performance during training. In the time step The noise intensity at this time is related to the noise variance of the diffusion model. is the number of time steps, which is used to dynamically adjust the step size so that the update amplitude gradually decreases as the time step increases; The calculation formula of the disturbance value is: in, is the disturbance value, is a scaling factor of 0.1, which is used to control the strength of the adversarial perturbation. is the perturbation step length, is the sign function of the gradient of the loss function, which returns the sign of the gradient vector (positive or negative). In adversarial attacks, the sign of the gradient is used to determine the direction of the perturbation. is the current gradient obtained using the fractional distillation sampling algorithm.

5. According to the method of claim 4, the current gradient calculation formula is: in, Express The gradient operator is used to calculate the loss function L relative to the input preset adversarial sample set image The gradient of Indicated in the model function Next, image After disturbance The loss function after ; Indicates the first The image of the iteration, represents the loss function calculated using the fractional distillation sampling algorithm, Represents the expected value operator, which is used to calculate the The average value of the gradient under the combination, represents the original input image, before iteration and perturbation, is the number of time steps, is the set of model parameters, is the number of time steps Add to image The actual noise in Indicates that the model is based on the current model parameter set , time steps and the description text The predicted noise, Indicates the number of time steps When Relative to The gradient of is the number of time steps The output of the model is is the number of time steps The input of the model.

6. The method according to claim 1, characterized in that The loss function of the proxy model is: in, is the loss function, which is used to measure the given model parameters and preset face image sets The loss value calculated under the condition of is the expected value operator, which is used to calculate all possible The average value of the loss function under the combination, is the number of time steps, is the set of model parameters, is the number of time steps Add to image The actual noise in Indicates that the model is based on the current model parameter set , time steps and the description text Predicted noise.

7. The method according to claim 1, characterized in that The preset time step is 3 steps, the preset number threshold is greater than 6, the preset disturbance threshold is 50 times, and the fine-tuning method of the proxy model adopts the training method of the DreamBooth algorithm.

8. A model application method, characterized in that: The method comprises: receiving an image processing request, wherein the image processing request carries a facial image of a user; Inputting the face image into a final proxy personalized generation model, and outputting a target image with added privacy protection information, wherein the privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest; the final proxy personalized generation model is obtained by fine-tuning the pre-trained model fine-tuning method for data protection according to any one of claims 1 to 7; The target image is displayed.

9. A pre-trained model fine-tuning device for data protection, characterized in that: The device comprises: A creation module for creating a proxy model for data protection, wherein the proxy model is obtained by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross attention layer of the U-Net structure; A first fine-tuning module is used to iteratively fine-tune the proxy model for a preset time step according to a preset face image set and its description text, so as to obtain a proxy personalized generation model; A first attack module, used to attack the agent personalized generation model using a preset adversarial sample set and the description text to obtain a disturbance value, wherein the preset adversarial sample set is obtained by preprocessing the preset face image set, and the disturbance step length when calculating the disturbance value is calculated according to the number of time steps; A second attack module, used for adding the disturbance value to the preset adversarial sample set, and continuing to execute the step of attacking the agent personalized generation model until the number of attacks reaches a preset number threshold, thereby obtaining a target adversarial sample; The second fine-tuning module is used to use the target adversarial sample to iteratively fine-tune the proxy model for a preset time step again, and obtain the final proxy personalized generation model when the current number of fine-tuning times reaches a preset disturbance threshold.

10. A model application device, characterized in that: The device comprises: A receiving module, used for receiving an image processing request, wherein the image processing request carries a face image of a user; An input module, used to input the face image into a final proxy personalized generation model, and output a target image with added privacy protection information, wherein the privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest; the final proxy personalized generation model is obtained by fine-tuning the pre-trained model fine-tuning method for data protection according to any one of claims 1 to 7; A display module is used to display the target image.

Images