Pre-trained Model Fine-tuning Method, Application Method and Device for Data Protection

Abstract

The present application discloses a method, an application method and a device for fine-tuning a pre-trained model for data protection. The method includes: creating a proxy model for data protection; performing iterative fine-tuning on the proxy model for a preset number of time steps according to a preset face image set and its description text to obtain a proxy personalized generation model; using a preset adversarial sample set and the description text to attack the proxy personalized generation model to obtain a perturbation value; adding the perturbation value to the preset adversarial sample set, and continuing to perform the step of attacking the proxy personalized generation model until the number of attacks reaches a preset number threshold, obtaining a target adversarial sample; using the target adversarial sample to perform iterative fine-tuning on the proxy model for a preset number of time steps again, and obtaining a final proxy personalized generation model when the current fine-tuning number reaches a preset perturbation threshold. By adopting the embodiments of the present application, the computing requirements in the actual training scenario are small, and the waste of computing resources is reduced.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and particularly to a method, application method and device for fine-tuning a pre-trained model for data protection. Background Art

[0002] In the digital age, the PCS (Personalized Content Synthesis) technology plays an increasingly important role in the field of text-to-image generation. Users can generate high-quality images related to a specific theme or object by providing a short description text using the PCS technology.

[0003] The DM (Diffusion Model) is a mainstream method for implementing the PCS technology. Users can fine-tune a pre-trained diffusion model using an algorithm such as DreamBooth by providing an image related to the SoI (Subject of Interest), and thus can generate new images containing the SoI. Malicious users can use this technology to generate realistic synthetic images and spread false information through social media platforms, which poses a threat to personal safety and privacy.

[0004] In related technologies, in order to solve the privacy and security problems caused by the malicious use of facial images in PCS, Anti-DreamBooth has been proposed in the prior art to guide the model to generate irrelevant results. This method interferes with DreamBooth by learning an alternative model, thereby enhancing the protection effect of personal images. However, currently, when training Anti-DreamBooth, prior class samples need to be constructed and all parameters of the training model are required during training. At the same time, currently, a relatively large number of interferences are required to complete the learning, resulting in high computational requirements and high computational costs in actual training scenarios. Summary of the Invention

[0005] Embodiments of this application provide a method, application method and device for fine-tuning a pre-trained model for data protection. To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary part is not a general review, nor is it intended to identify key / important constituent elements or delineate the protection scope of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the subsequent detailed description.

[0006] In a first aspect, embodiments of this application provide a method for fine-tuning a pre-trained model for data protection, the method including:

[0007] Create a proxy model for data protection, which is obtained by disabling other parameters in the pre-trained diffusion model except the parameters of the cross-attention layers of the text encoder and the U-Net structure;

[0008] According to the preset face image set and its description text, perform iterative fine-tuning on the proxy model for a preset number of time steps to obtain a proxy personalized generation model;

[0009] Use the preset adversarial sample set and the description text to attack the proxy personalized generation model to obtain a perturbation value. Among them, the preset adversarial sample set is obtained by preprocessing the preset face image set, and the perturbation step size during the calculation of the perturbation value is calculated according to the number of time steps;

[0010] Add the perturbation value to the preset adversarial sample set, and continue to execute the step of attacking the proxy personalized generation model until the number of attacks reaches the preset number threshold, and then obtain the target adversarial sample;

[0011] Use the target adversarial sample to perform iterative fine-tuning on the proxy model for a preset number of time steps again. When the current number of fine-tuning times reaches the preset perturbation threshold, obtain the final proxy personalized generation model.

[0012] Optionally, using the preset adversarial sample set and the description text to attack the proxy personalized generation model to obtain a perturbation value includes:

[0013] Input the preset adversarial sample set and the description text into the proxy personalized generation model to calculate the perturbation value;

[0014] Output the processing result corresponding to the preset adversarial sample set;

[0015] Obtain the perturbation value included in the processing result.

[0016] Optionally, inputting the preset adversarial sample set and the description text into the proxy personalized generation model to calculate the perturbation value includes:

[0017] The proxy personalized generation model uses the score distillation sampling algorithm to perform computational processing on the preset adversarial sample set and the description text to obtain the current gradient;

[0018] The proxy personalized generation model obtains the number of time steps;

[0019] The proxy personalized generation model calculates the perturbation step size according to the number of time steps;

[0020] The proxy personalized generation model calculates the perturbation value according to the current gradient and the perturbation step size.

[0021] Optionally, the perturbation step size calculation formula is:

[0022]

[0023] Among them, is the perturbation step size, is a hyperparameter used to adjust the size of the step size, which can be adjusted according to the needs of the model and its performance during training. At the time step the noise addition intensity, which is related to the noise variance of the diffusion model. is the time step used to dynamically adjust the step size so that as the time step increases, the update amplitude gradually decreases;

[0024] The formula for calculating the perturbation value is:

[0025]

[0026] Among them, is the perturbation value, is a scaling factor of 0.1 used to control the intensity of adversarial perturbations. is the perturbation step size, is the sign function of the gradient of the loss function, which returns the sign (positive or negative) of the gradient vector. In adversarial attacks, the sign of the gradient is used to determine the direction of the perturbation. is the current gradient obtained using the fractional distillation sampling algorithm.

[0027] Optionally, the formula for calculating the current gradient is:

[0028]

[0029] Among them, represents the gradient operator for used to calculate the gradient of the loss function L with respect to the image in the preset adversarial sample set of the input; represents the loss function of the image under the model function after being perturbed by ; represents the image at the th iteration during the iteration process, represents the loss function calculated using the fractional distillation sampling algorithm, represents the expected value operator used to calculate the average value of the gradients under all possible combinations, represents the original input image, the state before iteration and perturbation, is the time step, is the set of model parameters, is the time step when added to the image The actual noise in indicates that the model, according to the current set of model parameters , the number of time steps and the description text predicts the noise. Indicates the number of time steps When, the model output relative to the gradient of is the number of time steps the output of the model when is the number of time steps the input of the model when.

[0030] Optionally, the loss function of the surrogate model is:

[0031]

[0032] where is the loss function, used to measure the loss value calculated under the given model parameters and the preset face image set The case of calculating the loss value, is the expected value operator, used to calculate the average value of the loss function under all possible combinations, is the number of time steps, is the set of model parameters, is the number of time steps when added to the image the actual noise in indicates that the model, according to the current set of model parameters , the number of time steps and the description text predicts the noise.

[0033] Optionally, the preset number of time steps is 3 steps, the preset number threshold is greater than 6, the preset perturbation threshold is 50 times, and the fine-tuning method of the surrogate model adopts the training method of the DreamBooth algorithm.

[0034] In a second aspect, an embodiment of the present application provides a model application method, the method includes:

[0035] Receiving an image processing request, the image processing request carrying the user's face image;

[0036] Inputting the face image into the final surrogate personalized generation model, and outputting a target image with privacy protection information added, the privacy protection information being used to prevent the diffusion model from generating images related to the subject of interest; the final surrogate personalized generation model is obtained by fine-tuning through a pre-training model fine-tuning method for data protection;

[0037] Display the target image.

[0038] In a third aspect, an embodiment of the present application provides a pre-trained model fine-tuning device for data protection, and the device includes:

[0039] A creation module, configured to create a proxy model for data protection, where the proxy model is obtained after disabling other parameters except the parameters of the cross-attention layer of the text encoder and the U-Net structure in the pre-trained diffusion model;

[0040] A first fine-tuning module, configured to iteratively fine-tune the proxy model for a preset number of time steps according to a preset face image set and its description text to obtain a proxy personalized generation model;

[0041] A first attack module, configured to use a preset adversarial sample set and description text to attack the proxy personalized generation model to obtain a perturbation value, where the preset adversarial sample set is obtained by preprocessing the preset face image set, and the perturbation step size when calculating the perturbation value is calculated according to the number of time steps;

[0042] A second attack module, configured to add the perturbation value to the preset adversarial sample set and continue to execute the step of attacking the proxy personalized generation model until the number of attacks reaches a preset number threshold, and then obtain a target adversarial sample;

[0043] A second fine-tuning module, configured to use the target adversarial sample to iteratively fine-tune the proxy model for a preset number of time steps again, and obtain a final proxy personalized generation model when the current fine-tuning number reaches a preset perturbation threshold.

[0044] In a fourth aspect, an embodiment of the present application provides a model application device, and the device includes:

[0045] A receiving module, configured to receive an image processing request, where the image processing request carries a user's face image;

[0046] An input module, configured to input the face image into the final proxy personalized generation model to output a target image with privacy protection information added, where the privacy protection information is used to prevent the diffusion model from generating images related to the topic of interest; the final proxy personalized generation model is obtained by fine-tuning through the pre-trained model fine-tuning method for data protection;

[0047] A display module, configured to display the target image.

[0048] The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

[0049] In the embodiments of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the cross-attention layer of the text encoder and the U-Net structure, the model parameters are greatly reduced. At the same time, the perturbation update step size is dynamically adjusted according to the time steps, which not only destroys the coupling between the text and the image that malicious users may exploit, but also reduces the time and computing resources required for adversarial sample training, resulting in less computing demand in the actual training scenario and reducing the waste of computing resources.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0052] Figure 1 is a schematic flowchart of a method for fine-tuning a pre-trained model for data protection provided by an embodiment of the present application;

[0053] Figure 2A is a schematic diagram of a technical architecture for fine-tuning a pre-trained model for data protection provided by an embodiment of the present application;

[0054] Figure 2B is another schematic diagram of a technical architecture for fine-tuning a pre-trained model for data protection provided by an embodiment of the present application;

[0055] Figure 3 is a schematic flowchart of a model application method provided by an embodiment of the present application;

[0056] Figure 4 is a schematic diagram of the structure of a device for fine-tuning a pre-trained model for data protection provided by an embodiment of the present application;

[0057] Figure 5 is a schematic diagram of the structure of a model application device provided by an embodiment of the present application;

[0058] Figure 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The following description and the accompanying drawings fully illustrate the specific embodiments of the present application, enabling those skilled in the art to practice them.

[0060] It should be clear that the described embodiments are only a part of the embodiments of this application, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts belong to the scope of protection of this application.

[0061] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. On the contrary, they are only examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0062] In the description of this application, it should be understood that terms such as "first", "second", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, in the description of this application, unless otherwise specified, "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0063] This application provides a fine-tuning method, application method, and device for a pre-trained model for data protection to solve the problems existing in the above-mentioned related technical problems. In the embodiments of this application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the cross-attention layers of the text encoder and the U-Net structure, the model parameters are greatly reduced. At the same time, the perturbation update step size is dynamically adjusted according to the time step, which not only destroys the coupling between text and images that malicious users may utilize, but also reduces the time and computing resources required for adversarial sample training, resulting in less computing demand in the actual training scenario and reducing the waste of computing resources. The following will be described in detail using exemplary embodiments.

[0064] This application provides a fine-tuning method, application method, and device for a pre-trained model for data protection to solve the problems existing in the above-mentioned related technical problems. The following will be combined with the attached Figure 1 -attached Figure 3 , to introduce in detail the fine-tuning method for the pre-trained model for data protection provided by the embodiments of this application. This method can be implemented depending on a computer program and can run on a pre-trained model fine-tuning device for data protection based on the von Neumann architecture. This computer program can be integrated into an application or run as an independent tool class application.

[0065] Please refer to Figure 1, This is a schematic flowchart of a method for fine-tuning a pre-trained model for data protection provided by an embodiment of the present application, which is applied to a client. The client includes a private dataset, a general model deployed for federated learning, and a local model. The general model is a model shared by all clients, and the local model is trained for local services. As Figure 1 shown, the method of the embodiment of the present application may include the following steps:

[0066] S101, Create a proxy model for data protection. The proxy model is obtained by disabling other parameters except the parameters of the cross-attention layer of the text encoder and the U-Net structure in the pre-trained diffusion model;

[0067] Among them, the pre-trained diffusion model is obtained by training the DM (Diffusion Model) in advance with a large amount of data. This model already exists, such as the Stable Diffusion model, or can also be re-trained. The text encoder is a model component that converts text data into a numerical representation (usually in vector form). In the text-to-image generation task, the text encoder is used to convert the text description into a form that can be combined with image features. U-Net is a convolutional neural network architecture commonly used in image processing tasks (such as segmentation, generation, etc.). It has a special U-shaped structure and can effectively combine the context information and location information of the image.

[0068] In some embodiments of the present application, a pre-trained diffusion model is selected, such as the StableDiffusion model, which has been trained on a large amount of image and text data. In this model, except for the parameters of the text encoder and the cross-attention layer in the U-Net structure, all other parameters are set to be non-trainable (that is, their updates are disabled), which is achieved by setting the weights of these layers to fixed values or using specific training configurations. At this time, a proxy model is obtained, which retains the capabilities of text encoding and cross-attention mechanisms, but the other parts remain unchanged.

[0069] S102, Iteratively fine-tune the proxy model according to the preset face image set and its description text for a preset number of time steps to obtain a proxy personalized generation model;

[0070] Among them, the preset face image set refers to a group of pre-selected face images, which will be used to fine-tune the proxy model so that it can generate personalized content similar to these images. The description text is the text description associated with the preset face image set, used to guide the model to generate face images that match the specific text description. These texts usually contain descriptions of the image content, such as "a smiling young man". The preset number of time steps in the diffusion model generally refers to the fixed number of time steps that the model needs to go through in the reverse diffusion process to recover from the noisy image to the clear image. Iterative fine-tuning means that during the model training process, the model parameters are gradually adjusted through multiple iterations so that the model can better adapt to specific training data.

[0071] In some embodiments of the present application, the specific process of performing iterative fine-tuning for a preset number of time steps on the proxy model according to the preset face image set and its description text to obtain the proxy personalized generation model includes: obtaining the preset face image set and its description text, preprocessing the preset face image set, and uniformly scaling it to a unified fixed size After that, a clean reference image set is formed , using the clean image set and the description text, and combining with the current Dreambooth training method, fine-tuning the pre-trained diffusion model for 3 steps to obtain the proxy personalized generation model. The model parameter set of the pre-trained diffusion model is .

[0072] Specifically, the loss function of the proxy model is:

[0073]

[0074] Among them, is the loss function, used to measure the loss value calculated under the given model parameters and the preset face image set , is the expected value operator, used to calculate the average value of the loss function under all possible combinations, is the number of time steps, is the set of model parameters, is the number of time steps when the actual noise added to the image , represents the noise predicted by the model according to the current set of model parameters , the number of time steps and the description text .

[0075] S103. Use the preset adversarial sample set and the description text to attack the proxy personalized generation model to obtain a perturbation value. The preset adversarial sample set is obtained by preprocessing a preset face image set, and the perturbation step size when calculating the perturbation value is calculated according to the number of time steps.

[0076] Among them, the preset adversarial sample set is obtained by preprocessing the preset face image set. These samples are usually created by adding carefully calculated perturbations to the original images, aiming to deceive the model into making wrong predictions or classifications without significantly changing the appearance of the images. The perturbation value refers to the tiny changes or noises added to the original image in the adversarial sample. These changes are designed to maximize the change in the model output while keeping the image invariant to human observers. Preprocessing refers to a series of processing steps performed on the image. This may include operations such as scaling, cropping, and normalization, aiming to make the image data suitable for the input requirements of the model. The perturbation step size refers to the amplitude or scale of updating the perturbation when calculating the perturbation value. In adversarial attacks, dynamically adjusting the perturbation step size is a common strategy, which can be used to control the intensity of the perturbation to more effectively deceive the model. The step size may be adjusted according to the number of time steps. The number of time steps usually refers to the step count in the iterative process. In the context of adversarial attacks, the number of time steps can be used to dynamically adjust the perturbation step size to adapt to the sensitivity of the model at different training stages.

[0077] Among them, preprocess the preset face image set and scale it to a unified fixed size to form an adversarial sample set .

[0078] In some embodiments of the present application, the specific process of using the preset adversarial sample set and the description text to attack the proxy personalized generation model to obtain a perturbation value includes: inputting the preset adversarial sample set and the description text into the proxy personalized generation model to calculate the perturbation value; outputting the processing result corresponding to the preset adversarial sample set; and obtaining the perturbation value included in the processing result.

[0079] Specifically, the specific process of inputting the preset adversarial sample set and the description text into the proxy personalized generation model to calculate the perturbation value includes: the proxy personalized generation model uses the score distillation sampling algorithm to perform calculation processing on the preset adversarial sample set and the description text to obtain the current gradient; obtain the number of time steps; calculate the perturbation step size according to the number of time steps; and calculate the perturbation value according to the current gradient and the perturbation step size.

[0080] Specifically, the perturbation step size calculation formula is:

[0081]

[0082] Among them, is the perturbation step size, is a hyperparameter used to adjust the size of the step, which can be adjusted according to the needs of the model and its performance during training. At the time step the noise addition intensity, which is related to the noise variance of the diffusion model. is the time step used to dynamically adjust the step size so that as the time step increases, the update amplitude gradually decreases.

[0083] The formula for the perturbation value is:

[0084]

[0085] where is the perturbation value, is a scaling factor of 0.1 used to control the intensity of the adversarial perturbation. is the perturbation step size, is the sign function of the loss function gradient, which returns the sign (positive or negative) of the gradient vector. In adversarial attacks, the sign of the gradient is used to determine the direction of the perturbation. is the current gradient obtained using the fractional distillation sampling algorithm.

[0086] Specifically, the formula for the current gradient is:

[0087]

[0088] where represents the gradient operator for used to calculate the gradient of the loss function L with respect to the image in the input preset adversarial sample set; represents the loss function of the image after being perturbed by under the model function ; represents the image at the th iteration during the iteration process, represents the loss function calculated using the fractional distillation sampling algorithm, represents the expectation operator used to calculate the average value of the gradient over all possible combinations, represents the original input image, in the state before iteration and perturbation, is the time step, is the set of model parameters, is the time step when the actual noise added to the image ; represents that the model, according to the current set of model parameters and the time step and the description text Predicted noise, indicating the time step when the model output relative to the gradient of, is the time step the output of the model at, is the time step the input of the model at.

[0089] S103. Add the perturbation value to the preset adversarial sample set, and continue to execute the step of attacking the proxy personalized generation model until the number of attacks reaches the preset number threshold, and then obtain the target adversarial sample;

[0090] Among them, the target adversarial sample refers to the final adversarial sample obtained after a series of adversarial attack steps, which can successfully deceive or mislead the machine learning model to produce incorrect outputs or predictions.

[0091] In some embodiments of the present application, add the calculated perturbation value to each image in the adversarial sample set to generate a new adversarial sample set. Continue to execute the attack step, and use the updated adversarial sample set each time. In each iteration, adjust the perturbation value according to the reaction of the model. Record the number of attacks and ensure that the number of attacks does not exceed the preset number threshold. When the number of attacks reaches the preset number threshold, stop the attack process. The obtained adversarial sample set at this time is the target adversarial sample set.

[0092] S104. Use the target adversarial sample to perform iterative fine-tuning on the proxy model for a preset number of time steps. When the current number of fine-tuning times reaches the preset perturbation threshold, obtain the final proxy personalized generation model.

[0093] In some embodiments of the present application, use the target adversarial sample and combine the current Dreambooth training method to fine-tune the pre-trained diffusion model for 3 steps to re-obtain the proxy personalized generation model. The model parameter set of the pre-trained diffusion model is . When the current number of fine-tuning times reaches the preset perturbation threshold, obtain the final proxy personalized generation model. Otherwise, return to step S102 to perform fine-tuning again.

[0094] Specifically, the preset number of time steps is 3 steps, the preset number threshold is greater than 6, the preset perturbation threshold is 50 times, and the fine-tuning method of the proxy model adopts the training method of the DreamBooth algorithm.

[0095] For example Figure 2A 、 2BAs shown in the figure, it is a schematic diagram of the technical architecture for fine-tuning a pre-trained model for data protection provided by this application, including two parts: proxy model training and attack. When training the proxy model, a preset face image set and text prompt words are used to input the cross-attention layer and text encoder of the model for training. After the training is completed, a proxy personalized generation model is obtained. Then, adversarial samples are used to attack the proxy personalized generation model to obtain target adversarial samples. Finally, the target adversarial samples are used to train the proxy personalized generation model. After the training conditions meet the preset conditions, the final proxy personalized generation model can be obtained.

[0096] For example, define as the set of face images to be protected. For each image in the set , add an adversarial perturbation and publish the modified image , while preserving the privacy of the original image. The set of published images is called . Malicious users can collect a small set of images, that is, the set containing , and then use this set as reference images to fine-tune the diffusion model and obtain the optimal hyperparameters according to the specific PCS algorithm. The overall goal is to optimize the adversarial noise:

[0097]

[0098] where is the loss function used to evaluate the performance of adversarial examples, and usually the loss function of the diffusion model is used: . The noise is restricted within the ball measured by the norm. Existing methods widely use Projected Gradient Descent (PGD) to iteratively optimize adversarial examples. This process follows the following formula:

[0099]

[0100]

[0101] where x is the input image, is the sign function, is the gradient of the loss function with respect to . represents the step size for each iteration, and k is the number of iterations.

[0102] In the embodiments of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the cross-attention layers of the text encoder and the U-Net structure, the model parameters are greatly reduced. At the same time, the perturbation update step size is dynamically adjusted according to the time step, which not only destroys the coupling between the text and the image that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, resulting in less computing demand in the actual training scenario and reducing the waste of computing resources.

[0103] Please refer to Figure 3 , which is a schematic flowchart of a model application method provided by the embodiments of the present application. As Figure 3 shown, the method of the embodiments of the present application may include the following steps:

[0104] S201, receive an image processing request, where the image processing request carries the user's face image;

[0105] Among them, the image processing request refers to a request sent by the user to the system, asking to process a specific image, which involves using a certain model to modify the image. The face image is a specific image provided by the user, usually containing face information for further processing or analysis.

[0106] In some embodiments of the present application, the user uploads their face image through a website or application and requests an image processing service. After receiving the face image, the system passes it as input to the final proxy personalized generation model.

[0107] S202, input the face image into the final proxy personalized generation model, and output a target image with privacy protection information added, where the privacy protection information is used to prevent the diffusion model from generating images related to the topic of interest; the final proxy personalized generation model is obtained by fine-tuning through a pre-trained model fine-tuning method for data protection;

[0108] Among them, the proxy personalized generation model is a fine-tuned model that can generate a new image according to the user's face image while considering the user's personalized needs. The privacy protection information refers to specific information or perturbations added during the image processing process, aiming to prevent the model from generating images that may violate privacy or be misused.

[0109] In some embodiments of the present application, the proxy personalized generation model analyzes the face image and generates a new image that is visually similar to the original image but contains privacy protection information to prevent abuse. The target image output by the model is processed and contains privacy protection information to prevent other systems using the diffusion model from generating images related to the user's face.

[0110] S203, display the target image.

[0111] Among them, the target image refers to the final image after processing, which not only contains the content of the original image but also adds privacy protection information to prevent abuse.

[0112] In some embodiments of the present application, the system displays the generated target image to the user, and the user can see the image effect after adding the privacy protection information.

[0113] In the embodiments of the present application, by disabling other parameters of the pre-trained diffusion model except for the parameters of the cross-attention layer of the text encoder and the U-Net structure, the model parameters are greatly reduced. At the same time, the perturbation update step size is dynamically adjusted according to the time steps, which not only destroys the coupling between the text and the image that may be exploited by malicious users but also reduces the time and computing resources required for adversarial sample training, resulting in less computing demand in the actual training scenario and reducing the waste of computing resources.

[0114] The following are the device embodiments of the present application, which can be used to execute the method embodiments of the present application. For the details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0115] Please refer to Figure 4 , which shows a schematic structural diagram of a pre-trained model fine-tuning device for data protection provided by an exemplary embodiment of the present application. The pre-trained model fine-tuning device for data protection can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device includes a private data set, a general model deployed for federated learning, and a local model. The general model is a model shared by all clients, and the local model is trained for local services. The device 1 includes a creation module 10, a first fine-tuning module 20, a first attack module 30, a second attack module 40, and a second fine-tuning module 50.

[0116] The creation module 10 is used to create a proxy model for data protection, and the proxy model is obtained by disabling other parameters of the pre-trained diffusion model except for the parameters of the cross-attention layer of the text encoder and the U-Net structure;

[0117] The first fine-tuning module 20 is used to perform iterative fine-tuning of the proxy model for a preset number of time steps according to a preset face image set and its description text to obtain a proxy personalized generation model;

[0118] The first attack module 30 is used to attack the proxy personalized generation model with a preset adversarial sample set and description text to obtain a perturbation value, where the preset adversarial sample set is obtained by preprocessing the preset face image set, and the perturbation step size when calculating the perturbation value is calculated according to the time steps;

[0119] The second attack module 40 is used to add perturbation values to a preset adversarial sample set and continue to execute the step of attacking the proxy personalized generation model until the target adversarial sample is obtained when the number of attacks reaches a preset number threshold.

[0120] The second fine-tuning module 50 is used to use the target adversarial sample to iteratively fine-tune the proxy model for a preset number of time steps, and obtain the final proxy personalized generation model when the current fine-tuning times reach a preset perturbation threshold.

[0121] It should be noted that when the pre-trained model fine-tuning device for data protection provided in the above embodiments executes the pre-trained model fine-tuning method for data protection, only the above-mentioned division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the pre-trained model fine-tuning device for data protection provided in the above embodiments and the embodiments of the pre-trained model fine-tuning method for data protection belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0122] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0123] In the embodiments of the present application, by disabling other parameters except the parameters of the text encoder and the cross-attention layer of the U-Net structure in the pre-trained diffusion model, the model parameters are greatly reduced. At the same time, the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and images that may be exploited by malicious users, but also reduces the time and computing resources required for adversarial sample training, resulting in less computing demand in the actual training scenario and reducing the waste of computing resources.

[0124] Please refer to Figure 5 , which shows a schematic structural diagram of a model application device provided by an exemplary embodiment of the present application. The pre-trained model fine-tuning device for data protection can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device 2 includes a receiving module 60, an input module 70, and a display module 80.

[0125] The receiving module 60 is used to receive an image processing request, and the image processing request carries the user's face image.

[0126] An input module 70 for inputting a face image into a final proxy personalized generation model and outputting a target image with privacy protection information added, where the privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest; the final proxy personalized generation model is obtained by fine-tuning through the above-mentioned pre-training model fine-tuning method for data protection;

[0127] A display module 80 for displaying the target image.

[0128] It should be noted that when the above-mentioned pre-training model fine-tuning device for data protection executes the pre-training model fine-tuning method for data protection, only the above-mentioned division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the above-mentioned pre-training model fine-tuning device for data protection and the embodiment of the pre-training model fine-tuning method for data protection belong to the same concept, and the implementation process is detailed in the method embodiment, which will not be repeated here.

[0129] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0130] In the embodiments of the present application, by disabling other parameters in the pre-trained diffusion model except for the parameters of the cross-attention layer of the text encoder and the U-Net structure, the model parameters are greatly reduced. At the same time, the perturbation update step size is dynamically adjusted according to the time steps, which not only destroys the coupling between text and images that malicious users may utilize, but also reduces the time and computing resources required for adversarial sample training, resulting in less computing demand in the actual training scenario and reducing the waste of computing resources.

[0131] The present application also provides a computer-readable medium with program instructions stored thereon, and when the program instructions are executed by a processor, the pre-training model fine-tuning method for data protection provided by the above-mentioned method embodiments is implemented.

[0132] The present application also provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the pre-training model fine-tuning method for data protection in the above-mentioned method embodiments.

[0133] Please refer to Figure 6 which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 6 shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0134] Among them, the communication bus 1002 is used to realize the connection and communication between these components.

[0135] Among them, the user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface.

[0136] Among them, the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0137] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and circuits to connect various parts within the entire electronic device 1000. By running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and by calling the data stored in the memory 1005, it executes various functions of the electronic device 1000 and processes data. Optionally, the processor 1001 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 1001 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for the rendering and drawing of the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 1001 and may be implemented separately by a single chip.

[0138] Among them, the memory 1005 may include a Random Access Memory (RAM), or may include a Read-Only Memory. Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above method embodiments, etc.; the data storage area may store the data involved in the above method embodiments. Optionally, the memory 1005 may also be at least one storage system located far from the aforementioned processor 1001. As Figure 6 shown, in the memory 1005 as a computer storage medium, it may include an operating system, a network communication module, a user interface module, and a pre-trained model fine-tuning application for data protection.

[0139] In Figure 6 the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user to obtain the data input by the user; and the processor 1001 can be used to call the pre-trained model fine-tuning application for data protection stored in the memory 1005 and specifically perform the following operations:

[0140] Create a proxy model for data protection, where the proxy model is obtained by disabling other parameters except the parameters of the cross-attention layers of the text encoder and the U-Net structure in the pre-trained diffusion model;

[0141] According to the preset face image set and its description text, perform iterative fine-tuning on the proxy model for a preset number of time steps to obtain a proxy personalized generation model;

[0142] Use the preset adversarial sample set and the description text to attack the proxy personalized generation model to obtain a perturbation value, where the preset adversarial sample set is obtained by preprocessing the preset face image set, and the perturbation step size when calculating the perturbation value is calculated according to the number of time steps;

[0143] Add the perturbation value to the preset adversarial sample set and continue to perform the step of attacking the proxy personalized generation model until the number of attacks reaches the preset number threshold, and then obtain the target adversarial sample;

[0144] Using the target adversarial samples, iteratively fine-tune the surrogate model for a preset number of time steps again. When the current number of fine-tuning times reaches the preset perturbation threshold, obtain the final surrogate personalized generation model.

[0145] In one embodiment, when the processor 1001 executes to attack the surrogate personalized generation model using the preset adversarial sample set and the description text to obtain the perturbation value, it specifically performs the following operations:

[0146] Input the preset adversarial sample set and the description text into the surrogate personalized generation model to calculate the perturbation value;

[0147] Output the processing result corresponding to the preset adversarial sample set;

[0148] Obtain the perturbation value included in the processing result.

[0149] In one embodiment, when the processor 1001 executes to input the preset adversarial sample set and the description text into the surrogate personalized generation model to calculate the perturbation value, it specifically performs the following operations:

[0150] The surrogate personalized generation model uses the score distillation sampling algorithm to perform computational processing on the preset adversarial sample set and the description text to obtain the current gradient;

[0151] The surrogate personalized generation model obtains the number of time steps;

[0152] The surrogate personalized generation model calculates the perturbation step size according to the number of time steps;

[0153] The surrogate personalized generation model calculates the perturbation value according to the current gradient and the perturbation step size.

[0154] In the embodiments of the present application, by disabling other parameters of the pre-trained diffusion model except for the parameters of the cross-attention layer of the text encoder and the U-Net structure, the model parameters are greatly reduced. At the same time, the perturbation update step size is dynamically adjusted according to the number of time steps, which not only destroys the coupling between text and images that malicious users may utilize, but also reduces the time and computing resources required for adversarial sample training, resulting in less computing demand in the actual training scenario and reducing the waste of computing resources.

[0155] Those of ordinary skill in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware through a computer program. The program for fine-tuning the pre-trained model for data protection can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0156] The above disclosure is only for the preferred embodiments of the present application. Of course, it cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application still fall within the scope covered by the present application.

Claims

1. A pre-trained model fine-tuning method for data protection, characterized in that: The method comprises: Creating a proxy model for data protection, wherein the proxy model is obtained by disabling all parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross-attention layer of the U-Net structure; According to a preset set of face images and their description texts, the proxy model is iteratively fine-tuned at a preset time step to obtain a proxy personalized generation model; Using a preset adversarial sample set and the description text, the agent personalized generation model is attacked to obtain a disturbance value, wherein the preset adversarial sample set is obtained by preprocessing the preset face image set, and the disturbance step length when calculating the disturbance value is calculated according to the time step; Adding the disturbance value to the preset adversarial sample set, and continuing to execute the step of attacking the agent personalized generation model until the number of attacks reaches a preset number threshold, thereby obtaining a target adversarial sample; The target adversarial sample is used to iteratively fine-tune the proxy model for a preset time step again, and when the current number of fine-tuning times reaches a preset disturbance threshold, a final proxy personalized generation model is obtained.

2. The method according to claim 1, characterized in that The using of the preset adversarial sample set and the description text to attack the agent personalized generation model to obtain a disturbance value includes: Inputting the preset adversarial sample set and the description text into the agent personalized generation model to calculate the disturbance value; Output the processing result corresponding to the preset adversarial sample set; A disturbance value included in the processing result is obtained.

3. The method according to claim 2, characterized in that The step of inputting the preset adversarial sample set and the description text into the agent personalized generation model to calculate the disturbance value includes: The agent personalized generation model uses a fractional distillation sampling algorithm to calculate and process the preset adversarial sample set and the description text to obtain a current gradient; The agent personalized generation model obtains a time step; The agent personalized generation model calculates the perturbation step length according to the time step; The agent personalized generation model calculates a disturbance value according to the current gradient and the disturbance step size.

4. The method according to claim 3, characterized in that The perturbation step length calculation formula is: in, is the perturbation step length, It is a hyperparameter used to adjust the step size and can be adjusted according to the needs of the model and the performance during training. At time step The noise intensity at this time is related to the noise variance of the diffusion model. is the time step, which is used to dynamically adjust the step size so that the update amplitude gradually decreases as the time step increases; The calculation formula of the disturbance value is: in, is the disturbance value, is a scaling factor of 0.1, which is used to control the strength of the adversarial perturbation. is the perturbation step length, is the sign function of the gradient of the loss function, which returns the sign of the gradient vector. In adversarial attacks, the sign of the gradient is used to determine the direction of the perturbation. is the current gradient obtained using the fractional distillation sampling algorithm.

5. According to the method of claim 4, the current gradient calculation formula is: in, Express The gradient operator is used to calculate the loss function L relative to the input preset adversarial sample set image The gradient of Indicated in the model function Next, image After disturbance The loss function after ; Indicates the first The image of the iteration, represents the loss function calculated using the fractional distillation sampling algorithm, Represents the expected value operator, which is used to calculate the The average value of the gradient under the combination, represents the original input image, before iteration and perturbation, is the time step, is the set of model parameters, is the time step Add to image The actual noise in Indicates that the model is based on the current model parameter set , time step and the description text The predicted noise, Represents the time step When Relative to The gradient of is the time step The output of the model is is the time step The input of the model.

6. The method according to claim 1, characterized in that The loss function of the proxy model is: in, is the loss function, which is used to measure the given model parameters and preset face image sets The loss value calculated under the condition of is the expected value operator, which is used to calculate all possible The average value of the loss function under the combination, is the time step, is the set of model parameters, is the time step Add to image The actual noise in Indicates that the model is based on the current model parameter set , time step and the description text Predicted noise.

7. The method according to claim 1, characterized in that The preset time step is 3 steps, the preset number threshold is greater than 6, the preset disturbance threshold is 50 times, and the fine-tuning method of the proxy model adopts the training method of the DreamBooth algorithm.

8. A model application method, characterized in that: The method comprises: receiving an image processing request, wherein the image processing request carries a facial image of a user; Inputting the face image into a final proxy personalized generation model, and outputting a target image with added privacy protection information, wherein the privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest; the final proxy personalized generation model is obtained by fine-tuning the pre-trained model fine-tuning method for data protection according to any one of claims 1 to 7; The target image is displayed.

9. A pre-trained model fine-tuning device for data protection, characterized in that: The device comprises: A creation module for creating a proxy model for data protection, wherein the proxy model is obtained by disabling other parameters in the pre-trained diffusion model except for the parameters of the text encoder and the cross attention layer of the U-Net structure; A first fine-tuning module is used to iteratively fine-tune the proxy model for a preset time step according to a preset face image set and its description text, so as to obtain a proxy personalized generation model; A first attack module, used to attack the agent personalized generation model using a preset adversarial sample set and the description text to obtain a disturbance value, wherein the preset adversarial sample set is obtained by preprocessing the preset face image set, and the disturbance step length when calculating the disturbance value is calculated according to the time step; A second attack module, used for adding the disturbance value to the preset adversarial sample set, and continuing to execute the step of attacking the agent personalized generation model until the number of attacks reaches a preset number threshold, thereby obtaining a target adversarial sample; The second fine-tuning module is used to use the target adversarial sample to iteratively fine-tune the proxy model for a preset time step again, and obtain the final proxy personalized generation model when the current number of fine-tuning times reaches a preset disturbance threshold.

10. A model application device, characterized in that: The device comprises: A receiving module, used for receiving an image processing request, wherein the image processing request carries a face image of a user; An input module, used to input the face image into a final proxy personalized generation model, and output a target image with added privacy protection information, wherein the privacy protection information is used to prevent the diffusion model from generating images related to the subject of interest; the final proxy personalized generation model is obtained by fine-tuning the pre-trained model fine-tuning method for data protection according to any one of claims 1 to 7; A display module is used to display the target image.

Images