A text-driven face editing adversarial attack method, device and medium

Abstract

The present invention relates to a method, device, and medium for countering text-driven face editing attacks. The method comprises obtaining image samples and text samples, wherein the image samples include original images and target images; determining a fixed semantic mapping model based on the text samples; inputting the original image samples into an image inverse model to obtain original image inverse features; then inputting the fixed semantic mapping model, performing a weighted combination of the output of the fixed semantic mapping model and the original image inverse features with the original image inverse features to obtain edited image inverse features, wherein the weighting factor is an optimizable hyperparameter; generating an edited image based on the edited image inverse features; determining whether convergence has occurred, and iteratively optimizing the weighting factor to obtain the final edited image inverse features, thereby generating the final edited image. Compared with the prior art, the present invention has the advantages of ensuring the attack effectiveness of adversarial samples, enhancing the invisibility of adversarial interference, and improving the image quality of adversarial samples.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and medium for countering attacks against text-driven face editing. Background Art

[0002] Image editing models can generate edited images based on input text and an original image, ensuring that the edited image matches the semantics of the input text. For example, if the original image is a smiling face and the input text is "angry," the image editing model can generate a face that matches the original image but displays an angry expression. This technology can be applied to virtual and augmented reality scenarios, particularly in real-world scenarios such as face retouching, portrait creation, and interactive animation.

[0003] Thanks to the rapid development of artificial intelligence (AI), nearly all image editing models are based on deep neural networks. However, the highly concealed nature of generative adversarial attacks significantly reduces the robustness of deep neural networks. Specifically, attackers construct adversarial examples with malicious perturbations and input them into the network, causing legitimate users to receive irrational outputs, significantly reducing the network's usability.

[0004] Adversarial attacks are categorized into white-box and black-box attacks. White-box attacks require prior knowledge of the attack model's internal structure, which is incompatible with practical application scenarios. Black-box attacks, on the other hand, do not require knowledge of the attack model's internal structure and instead attack the attack model through query attacks and / or transfer attacks.

[0005] In order to test the stability of the image editing model, a text-driven face editing adversarial attack method is needed to improve the image quality of adversarial samples and enhance the invisibility of adversarial sample perturbations. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a text-driven face editing adversarial attack method, device and medium that improves the image quality of adversarial samples and enhances the invisibility of adversarial sample perturbations.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to a first aspect of the present invention, a method for countering attacks against text-driven face editing is provided, comprising the following steps:

[0009] Sample acquisition step: acquiring an image sample, which includes an original image sample and a target image sample, and determining whether the image sample meets the standard. If not, reacquire the image sample; acquiring a text sample, and determining whether the text sample complies with the rules. If not, reacquire the text sample;

[0010] Sample processing step: determining a fixed semantic mapping model based on the text sample, wherein the semantics of the text sample corresponds one-to-one with the fixed semantic mapping model; inputting the original image sample into a pre-established and trained image inverse model to obtain the original image inverse features;

[0011] Image editing step: inputting the original image reverse features into a fixed semantic mapping model, performing a weighted combination of the output of the fixed semantic mapping model and the original image reverse features to obtain edited image reverse features; the weighting factor in the weighted combination is an optimized hyperparameter; inputting the edited image reverse features into an image generator to obtain an edited generated image;

[0012] Iterative optimization step: determine whether the edited generated image converges; if not, re-execute the image editing step and iteratively optimize the weighting factor, the optimization process of the weighting factor including: inputting the edited generated image and the target image sample into a preset generalized face verification model, calculating the similarity of the output results as the face attack loss function value, and optimizing the weighting factor as feedback; if converged, obtaining the final edited image reverse feature, and inputting the final edited image reverse feature into the image generator to obtain the final edited generated image.

[0013] Furthermore, the feature size output by the image inverse model is the same as the feature size input by the fixed semantic mapping model.

[0014] Furthermore, the generalized face verification model is a pre-trained model focused on the face verification task and trained using heterogeneous face training data, wherein the heterogeneous face training data includes data constructed based on at least two face datasets.

[0015] Furthermore, the loss function used as feedback to optimize the weighting factor in the method also includes but is not limited to the following loss function:

[0016] Image quality loss function, which uses a pre-trained deep neural network for image classification to extract semantic features of the original image sample and the edited image respectively, thereby calculating the image quality;

[0017] The face identity loss function uses a pre-trained deep neural network for face recognition to extract the identity features of the original image sample and the edited image respectively, thereby calculating the face identity similarity;

[0018] The face parsing loss function uses a pre-trained deep neural network for facial feature parsing to extract the facial feature segmentation features of the original image sample and the edited image, thereby calculating the similarity of the facial feature segmentation features.

[0019] Furthermore, the method for determining whether the generated image after editing is convergent includes but is not limited to any one of the following determination methods or a combination of multiple determination methods:

[0020] A convergence judgment method based on an optimizable hyperparameter is used to judge whether a slope of a curve of the optimizable hyperparameter during the editing-generated image generation process is less than a preset first slope threshold according to a curve trend of the optimizable hyperparameter. If the slope of the curve is less than the first slope threshold, it is judged that the generated image after editing has reached convergence;

[0021] A convergence judgment method based on an overall loss function is used to judge whether a slope of the overall loss function during the editing-generated image generation process is less than a preset second slope threshold according to a curve trend of the overall loss function. If the slope of the curve is less than the second slope threshold, it is judged that the edited generated image has reached convergence;

[0022] The hyperparameter-based convergence judgment method is used to set the corresponding hyperparameter threshold according to the training-related hyperparameters. When the hyperparameter threshold is reached, it is judged that the edited generated image has reached convergence.

[0023] Furthermore, during the optimization process of the iterative optimization step, the weighting factors gradually converge, and the converged weighting factors are used as final weighting factors, thereby calculating the final inverse features of the edited image.

[0024] Furthermore, the process of determining whether an image sample meets the standards includes: evaluating the quality of the image sample in terms of image size, image resolution, and image quality indicators to determine whether they meet the standards; and determining whether the content of the image sample is a complete face and whether the facial features are clear.

[0025] Furthermore, the fixed semantic mapping model is a pre-trained multi-layer perceptron network.

[0026] According to a second aspect of the present invention, there is provided a device for countering attacks against text-driven face editing, comprising:

[0027] an acquisition module configured to acquire image samples and text samples, wherein the image samples include original image samples and target image samples, and determine a fixed semantic mapping model according to the text samples;

[0028] The optimization module is configured to perform the following steps: inputting the original image sample into a pre-established and trained image inverse model to obtain original image inverse features; inputting the original image inverse features into a fixed semantic mapping model, performing a weighted combination of the output of the fixed semantic mapping model and the original image inverse features to obtain edited image inverse features; the weighting factor in the weighted combination is an optimization hyperparameter; inputting the edited image inverse features into an image generator to obtain an edited generated image;

[0029] Determining whether the edited generated image converges; if not, iteratively optimizing the weighting factor, wherein the optimization process of the weighting factor includes: inputting the edited generated image and the target image sample into a preset generalized face verification model, calculating the similarity of the output results as the value of the face attack loss function, and thereby optimizing the weighting factor as feedback;

[0030] The generation module is configured to obtain the final edited image reverse features based on the converged weighting factors, and input the final edited image reverse features into the image generator to obtain the final edited generated image.

[0031] Furthermore, the loss function for optimizing the weighting factor also includes but is not limited to the following loss function:

[0032] Image quality loss function, which uses a pre-trained deep neural network for image classification to extract semantic features of the original image sample and the edited image respectively, thereby calculating the image quality;

[0033] The face identity loss function uses a pre-trained deep neural network for face recognition to extract the identity features of the original image sample and the edited image respectively, thereby calculating the face identity similarity;

[0034] The face parsing loss function uses a pre-trained deep neural network for facial feature parsing to extract the facial feature segmentation features of the original image sample and the edited image, thereby calculating the similarity of the facial feature segmentation features.

[0035] Furthermore, the method for determining whether the generated image after editing is convergent includes but is not limited to any one of the following determination methods or a combination of multiple determination methods:

[0036] A convergence judgment method based on an optimizable hyperparameter is used to judge whether a slope of a curve of the optimizable hyperparameter during the editing-generated image generation process is less than a preset first slope threshold according to a curve trend of the optimizable hyperparameter. If the slope of the curve is less than the first slope threshold, it is judged that the generated image after editing has reached convergence;

[0037] A convergence judgment method based on an overall loss function is used to judge whether a slope of the overall loss function during the editing-generated image generation process is less than a preset second slope threshold according to a curve trend of the overall loss function. If the slope of the curve is less than the second slope threshold, it is judged that the edited generated image has reached convergence;

[0038] The hyperparameter-based convergence judgment method is used to set the corresponding hyperparameter threshold according to the training-related hyperparameters. When the hyperparameter threshold is reached, it is judged that the edited generated image has reached convergence.

[0039] Furthermore, during the iterative optimization process of the weighting factors, the weighting factors gradually converge, and the converged weighting factors are used as the final weighting factors, thereby calculating the final edited image reverse features.

[0040] According to a third aspect of the present invention, an electronic device is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned text-driven face editing adversarial attack method.

[0041] Furthermore, the electronic device further includes an input device and an output device, and the memory, processor, input device and output device are all connected to a bus.

[0042] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, characterized in that a program is stored thereon, and when the program is executed by a processor, the text-driven face editing adversarial attack method as described above is implemented.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] (1) This invention provides an adversarial attack method for text-driven face editing. Compared with other methods, it can directly optimize and iteratively attack face editing tasks in the inference stage without training, thus reducing training costs, lowering the user threshold, and improving the practicality of the method.

[0045] (2) The optimization target of the present invention is only the weighting factor, rather than at least one deep neural network (such as the fixed semantic mapping model, image inverse model, generator, etc.). Because the feature dimension of the weighting factor is much smaller than the training parameters of the deep neural network, the computational overhead is greatly reduced, the carbon footprint is reduced, and the environmental friendliness of the method is improved;

[0046] (3) Compared with other black-box attack transfer attack methods, this method improves the threshold limit on image quality, face consistency, and text-image semantic consistency, and enhances the optimization process. While ensuring the effectiveness of the attack, it also enhances the invisibility of adversarial sample interference and improves the image quality of adversarial samples.

[0047] (4) The method of the present invention can conduct adversarial sample attacks more covertly, making up for the shortage of attack samples in actual scenarios, and can improve the security and robustness of the target model through adversarial training. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of an embodiment of an image editing method to which the method of the present invention can be applied;

[0049] Figure 2 This is a flow chart of an embodiment of a method for countering attacks against text-driven face editing provided by the present invention;

[0050] Figure 3 It is a face parsing loss function interpretation diagram based on an adversarial attack method for text-driven face editing provided by the present invention;

[0051] Figure 4 1 is a schematic structural diagram of an embodiment of a device for countering attacks against text-driven face editing provided by the present invention;

[0052] Figure 5 This is a block diagram of an embodiment of an electronic device for countering attacks against text-driven face editing provided by the present invention. Implementation Method

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0055] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0056] Example 1

[0057] This solution provides a text-driven face editing adversarial attack method, including the following steps:

[0058] Sample acquisition step: acquiring an image sample, which includes an original image sample and a target image sample, and determining whether the image sample meets the standard. If not, reacquire the image sample; acquiring a text sample, and determining whether the text sample complies with the rules. If not, reacquire the text sample;

[0059] Sample processing step: determining a fixed semantic mapping model based on the text sample, wherein the semantics of the text sample corresponds one-to-one with the fixed semantic mapping model; inputting the original image sample into a pre-established and trained image inverse model to obtain the original image inverse features;

[0060] Image editing step: inputting the original image reverse features into a fixed semantic mapping model, performing a weighted combination of the output of the fixed semantic mapping model and the original image reverse features to obtain edited image reverse features; the weighting factor in the weighted combination is an optimized hyperparameter; inputting the edited image reverse features into an image generator to obtain an edited generated image;

[0061] Iterative optimization step: determine whether the edited generated image converges; if not, re-execute the image editing step and iteratively optimize the weighting factor, the optimization process of the weighting factor including: inputting the edited generated image and the target image sample into a preset generalized face verification model, calculating the similarity of the output results as the face attack loss function value, and optimizing the weighting factor as feedback; if converged, obtaining the final edited image reverse feature, and inputting the final edited image reverse feature into the image generator to obtain the final edited generated image.

[0062] In the transfer attack of black-box attack, this scheme uses the transferability of adversarial samples generated by white-box models to improve the generalization and effectiveness of the attack method. By controlling the weighting factors of the original image inverse features and the output of the fixed semantic mapping model, the generalized face verification model calculates the similarity between the edited generated image of the original image sample and the target image sample, and then iteratively optimizes the weighting factors to ensure that the edited generated image and the target image sample reach a certain similarity. It is possible to generate an edited face with high fidelity to the target image sample, obtain the adversarial sample, and enhance the invisibility of the adversarial sample perturbation.

[0063] In addition, image quality loss function, face identity loss function and face parsing loss function can be constructed. By optimizing the face images before and after editing, the identity is maintained consistent and the image quality of the adversarial sample is improved.

[0064] Figure 1 A flowchart of an embodiment of an image editing method that can be applied thereto is shown, the image editing method comprising:

[0065] Step 101, obtaining an image sample;

[0066] Specifically, for the face editing scenario, the image sample refers to a high-quality face image, represented as ,in Representative is samples.

[0067] Step 102: determine whether the image sample quality and content meet the standards; if not, repeat step 101;

[0068] Specifically, to ensure the quality and content of the generated images after editing, it is necessary to perform dual quality and content testing on the image samples. The quality testing can consider but is not limited to the following set conditions: 1) image size, 2) image resolution, and 3) image quality index evaluation.

[0069] Optionally, image quality assessment includes but is not limited to being based on: 1) image pixel statistics basis, 2) information theory basis, 3) structural information basis, 4) subjective evaluation, 5) objective evaluation, etc.

[0070] Content detection can be considered to set but not limited to the following conditions: 1) image content, that is, it needs to be a complete face, 2) facial features need to be clear and unobstructed, etc.

[0071] Specifically, third-party open source tools can be used to implement image quality and content detection conditions. Image processing tools such as OpenCV and DLib can be used to perform quality detection and face detection on image samples. Alternatively, open source pre-trained deep neural networks can be used to perform quality scoring and face detection on image samples. These methods can also be used simultaneously to set strict judgment conditions for image samples. This disclosure does not limit the specific implementation methods.

[0072] It is worth noting that the content detection conditions can be adjusted in detail based on the specific scenario. In some optional implementations of this embodiment, if it is necessary to perform partial organ occlusion or exposure editing on the face image, such as "wearing a hat", "taking off a hat", "wearing glasses", "taking off glasses", etc., the content detection conditions can be appropriately relaxed. The specific adjustment conditions are not limited in this disclosure.

[0073] Step 103, obtaining a text sample;

[0074] Specifically, the text sample is used to describe the semantic content of the edited image, which is expressed as In particular, for face editing scenarios, text samples can be text used to describe the facial organ features of the edited face image, such as: "heavy makeup", "wearing glasses", "wearing lipstick", "bald", "open mouth", etc.; or they can be text describing the internal emotional features of the edited face image, such as: "sad", "happy", "surprised", "calm", etc.

[0075] Step 104: determine whether the text sample is compliant; if not, repeat step 103;

[0076] Specifically, the judgment of whether a text sample is compliant or not mainly involves checking the correctness of the text sample.

[0077] Step 105: determining a fixed semantic mapping model based on the text sample;

[0078] Specifically, the semantics of text samples correspond one-to-one to fixed semantic mapping models. The fixed semantic mapping model is an open-source pre-trained model, and its structure is usually a multi-layer perceptron (MLP) network. The fixed semantic mapping model is represented as .

[0079] Step 106: input the image sample into the image inverse model to obtain the original image inverse features;

[0080] Specifically, the image inversion model can be any deep neural network that focuses on image inversion tasks. Open source pre-trained e4e (encoder4editing) and pSp (pixel2Style2pixel) networks can be considered. The image inversion model is represented as , the original image inverse feature is expressed as .

[0081] It's worth noting that the image inversion model and the fixed semantic mapping model don't have a one-to-one correspondence. That is, the pre-training processes for the image inversion model and the fixed semantic mapping model are independent of each other. The training doesn't require the same training dataset or the same deep neural network used in the collaborative training. It only requires that the feature size of the image inversion model output be the same as the feature size of the fixed semantic mapping model input.

[0082] Step 107: input the original image reverse features into a fixed semantic mapping model, and perform a weighted combination of the fixed semantic mapping model output and the original image reverse features to obtain the edited image reverse features;

[0083] Specifically, the fixed semantic mapping model output is expressed as , the weighted combination output edited image inverse feature representation is , in is a fixed constant determined according to the specific image inverse model and fixed semantic mapping model used in different scenarios.

[0084] Step 108: Input the inverse features of the edited image into the image generator to obtain the edited generated image.

[0085] Specifically, the image generator is expressed as , the generated image after editing is represented as Furthermore, the image generator can be an open-source pre-trained style-based generative adversarial network StyleGAN series.

[0086] Continue to refer Figure 2 , which shows a flow chart of an embodiment of a method for countering attacks against text-driven face editing, including:

[0087] Step 201: Acquire an image sample, where the image sample includes an original image and a target image;

[0088] It is worth noting that in the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0089] Specifically, for face editing scenarios, the image sample content is required to be a high-quality face image, where the original image is represented as , the target image is represented as , Representative is samples.

[0090] Step 202: determine whether the image sample quality and content meet the standards; if not, repeat step 201;

[0091] In this embodiment, the specific operation of step 202 is already in Figure 1 Step 102 is described in detail in the illustrated embodiment and will not be repeated here.

[0092] Step 203, obtaining a text sample;

[0093] Specifically, the text sample is used to describe the semantic content of the edited image, which is expressed as .

[0094] Step 204: determine whether the counted text samples are compliant; if not, repeat step 203;

[0095] Specifically, the judgment of whether a text sample is compliant or not mainly involves checking the correctness of the text sample.

[0096] Step 205: determining a fixed semantic mapping model based on the text sample;

[0097] Specifically, the fixed semantic mapping model is expressed as .

[0098] Step 206: input the original image sample into the image inverse model to obtain the original image inverse feature;

[0099] Specifically, the image inverse model is expressed as , the original image inverse feature is expressed as .

[0100] Step 207: Input the original image reverse features into a fixed semantic mapping model, and perform a weighted combination of the output of the fixed semantic mapping model and the original image reverse features to obtain the edited image reverse features, wherein the weighting factor is an optimizable hyperparameter.

[0101] Specifically, the inverse feature of the edited image is expressed as:

[0102]

[0103] in is an optimizable hyperparameter.

[0104] Step 208: inputting the inverse features of the edited image into an image generator to obtain an edited generated image;

[0105] Specifically, the image generator is expressed as , the generated image after editing is represented as .

[0106] Step 209: Determine whether the generated image after editing has converged; if not, return to step 207 and iteratively optimize the weighting factors;

[0107] In this embodiment, in order to make the generated image after editing converge, it is necessary to use a generalized face verification model to constrain the generated image after editing and the objective function. Furthermore, the generalized face verification model refers to a pre-trained model that is trained using heterogeneous face training data and focuses on the face verification task, and is expressed as , its structure is usually convolutional neural network CNN.

[0108] Specifically, heterogeneous face training data differs from traditional homogeneous face training data. The final training dataset is constructed using at least two face datasets. The differences between the at least two face datasets lie in the fact that different facial features, such as facial appearance, gender, environment, region, and geographic location, can significantly distinguish different facial features. Specifically, different face datasets have varying facial features: some are thin, some are overweight, some wear hats, some wear glasses, and some wear headphones. The gender ratios vary between different face datasets. The richness of the environments varies between different face datasets: some primarily use selfie angles, some primarily use outdoor natural light imaging, and some primarily use images captured under extreme conditions. The characteristics of portraits collected in different regions vary: facial data collected in areas with strong ultraviolet rays tends to have darker skin tones, while those collected in areas with strong ultraviolet rays tend to have fairer skin tones. Facial data collected in high-altitude areas often have redder cheeks and more wrinkles. The characteristics of portraits collected in different regions also vary: Europeans and Americans have three-dimensional, hard features, while Asians have more flat faces and rounder, fuller faces. The face verification task involves determining whether two face images represent the same person based on input.

[0109] It is worth noting that in order to make the generated image converge after editing, it is necessary to adopt the face attack loss function constructed based on the generalized face verification model :

[0110]

[0111] in Refers to cosine similarity.

[0112] They will be edited to generate images and the target image Input Generalized Face Verification Model , extract semantic features respectively, and generate face verification features after editing and target image face verification features , generate face verification features after comparison and editing and target image face verification features The cosine similarity is used as the calculation result of the face attack loss function.

[0113] Furthermore, in order to ensure the quality and content of the generated image after editing, one or more of the following solutions may be considered, but not limited to:

[0114] Solution 1: To ensure that the quality of the edited image is consistent with the original image, an image quality loss function is constructed based on the feature level:

[0115]

[0116] in It is a pre-trained deep neural network for image classification used to extract semantic features of input images, commonly used networks such as Vgg and LPIPS. Refers to mean squared error MSE (mean squared error).

[0117] The original image and generate the image after editing Enter a deep neural network for image classification , extract semantic features respectively and obtain the semantic features of the original image and generate image semantic features after editing , compare the original image semantic features and generate image semantic features after editing The mean square error is used as the calculation result of the image quality loss function.

[0118] Solution 2: To ensure that the face identity of the edited image is consistent with that of the original image, a face identity loss function is constructed based on the feature level:

[0119]

[0120] in It is a pre-trained deep neural network for face recognition, used to extract identity features of input images, and commonly used networks such as ArcFace.

[0121] The original image and generate image input after editing Deep neural networks for face recognition In the example, identity features are extracted respectively to obtain the identity features of the original image and generate image identity features after editing , compared with the original image identity features and generate image identity features after editing The cosine similarity is used as the calculation result of the face identity loss function.

[0122] Solution 3: To strengthen the constraint that the face identity of the generated image after editing is consistent with that of the original image, a face parsing loss function is constructed based on the feature level:

[0123]

[0124] in It is a pre-trained deep neural network for facial features analysis, used to extract facial features segmentation features. For specific constraint details, please refer to Figure 3 , which shows a face parsing loss function interpretation graph for an adversarial attack method against text-driven face editing;

[0125] The original image and generate the image after editing Input deep neural network for facial feature analysis In the process, the facial features are extracted and the original facial analysis features are obtained. Generate face parsing features after editing , compared with the original face parsing features Generate face parsing features after editing The cosine similarity is used as the calculation result of the face parsing loss function.

[0126] Optionally, the method for determining whether the generated image after editing is convergent may be a combination of one or more of the following methods:

[0127] 1. Determine whether the hyperparameters can be optimized by recording the curve trend during the editing and generating process. Specifically, when the slope of the curve is less than a pre-set threshold, it can be determined that the generated image after editing has reached convergence;

[0128] 2. Analyze the overall loss function curve trend during the editing and generating process. Specifically, when the slope of the curve is less than a pre-set threshold, it can be determined that the generated image after editing has reached convergence.

[0129] 3. By setting the number of steps, duration, and other training-related hyperparameters in the iterative optimization process of the weighted factor, when a pre-set threshold is reached, it serves as a stopping / convergence condition.

[0130] In step 210 , the weighting factors converge to obtain the final edited image reverse features.

[0131] Specifically, the converged weighting factor is expressed as , is a constant, and the final edited image inverse feature is expressed as .

[0132] Step 211: input the reverse features of the final edited image into the image generator to obtain the final edited generated image.

[0133] Specifically, the final edited image is represented as .

[0134] Continue to refer Figure 4 As an implementation of the above-mentioned anti-attack method, the present disclosure provides a structural diagram of an embodiment of an anti-attack device 301 for text-driven face editing, including an acquisition module 302, an optimization module 303, and a generation module 304.

[0135] The acquisition module 302 is configured to acquire image samples and text samples uploaded by the user, wherein the image samples include original images and target images, and determine a fixed semantic mapping model according to the text samples.

[0136] The optimization module 303 is configured to perform the following steps: using the image inverse model to process the original image sample into the original image inverse features; initializing the weighting factor, weightedly combining the original image inverse features and the original image inverse features processed by the fixed semantic mapping model to obtain the edited image inverse features, and inputting the edited image inverse features into the image generator to obtain the edited generated image;

[0137] Determining whether the generated image after editing has converged; if not, iteratively optimizing the weighting factor, wherein the optimization process of the weighting factor includes: inputting the generated image after editing and the target image sample into a preset generalized face verification model, calculating the similarity of the output results as the value of the face attack loss function, and thus optimizing the weighting factor as feedback;

[0138] The generation module 304 is configured to weightedly combine the original image reverse features and the original image reverse features processed by the fixed semantic mapping model based on the converged weighting factors, and input the final edited image reverse features of the weighted combination result into the image generator to obtain the final edited generated image.

[0139] In this embodiment, the specific processing of the anti-attack device 301 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201-211 in the corresponding embodiment are not repeated here.

[0140] Figure 5 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown.

[0141] The term "device" is intended to refer to various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The term "electronic device" may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0142] like Figure 5 As shown, the electronic device includes a memory 401, a processor 402, an input device 403, and an output device 404. Each component is connected via a bus and can be installed on a common motherboard or in other ways according to actual needs. Figure 5The components shown, their connections and relationships, and their functions are merely illustrative examples and are not meant to limit the implementations described and / or claimed in this text.

[0143] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0144] For the convenience of description, the above device is described as being divided into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0145] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] In a typical configuration, such as Figure 5 As shown, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0148] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0149] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0150] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A text-driven face editing adversarial attack method, characterized by: The following steps are involved: Sample acquisition step: acquiring an image sample, which includes an original image sample and a target image sample, and determining whether the image sample meets the standard. If not, reacquire the image sample; acquiring a text sample, and determining whether the text sample complies with the rules. If not, reacquire the text sample; Sample processing step: determining a fixed semantic mapping model based on the text sample, wherein the semantics of the text sample corresponds one-to-one with the fixed semantic mapping model; inputting the original image sample into a pre-established and trained image inverse model to obtain the original image inverse features; Image editing step: inputting the original image reverse features into a fixed semantic mapping model, performing a weighted combination of the output of the fixed semantic mapping model and the original image reverse features to obtain edited image reverse features; the weighting factor in the weighted combination is an optimized hyperparameter; inputting the edited image reverse features into an image generator to obtain an edited generated image; Iterative optimization step: determining whether the generated image after editing converges; If convergence does not occur, the image editing step is re-executed and the weighting factor is iteratively optimized. The optimization process of the weighting factor includes: inputting the edited generated image and the target image sample into a preset generalized face verification model, calculating the similarity of the output results as the value of the face attack loss function, and thus optimizing the weighting factor as feedback; If converged, the final inverse features of the edited image are obtained, and the final inverse features of the edited image are input into the image generator to obtain the final edited generated image.

2. The method for countering text-driven face editing according to claim 1, characterized in that: The feature size output by the image inverse model is the same as the feature size input by the fixed semantic mapping model.

3. The method for countering attacks against text-driven face editing according to claim 1, characterized in that: The generalized face verification model is a pre-trained model focused on face verification tasks that is trained using heterogeneous face training data, wherein the heterogeneous face training data includes data constructed based on at least two face datasets.

4. The method for countering text-driven face editing according to claim 1, characterized in that: The loss function used as feedback to optimize the weighting factor in the method also includes but is not limited to the following loss functions: Image quality loss function, which uses a pre-trained deep neural network for image classification to extract semantic features of the original image sample and the edited image respectively, thereby calculating the image quality; The face identity loss function uses a pre-trained deep neural network for face recognition to extract the identity features of the original image sample and the edited image respectively, thereby calculating the face identity similarity; The face parsing loss function uses a pre-trained deep neural network for facial feature parsing to extract the facial feature segmentation features of the original image sample and the edited image, thereby calculating the similarity of the facial feature segmentation features.

5. The method for countering attacks against text-driven face editing according to claim 1, characterized in that: Methods for determining whether the generated image after editing is convergent include but are not limited to any one of the following determination methods or a combination of multiple determination methods: A convergence judgment method based on an optimizable hyperparameter is used to judge whether the slope of the curve of the optimizable hyperparameter during the editing and generating process of the generated image is less than a preset first slope threshold according to the trend of the curve of the optimizable hyperparameter. If the slope of the curve is less than the first slope threshold, it is judged that the generated image after editing has reached convergence; A convergence judgment method based on the overall loss function is used to judge whether the slope of the curve is less than a preset second slope threshold according to the curve trend of the overall loss function in the process of generating the edited generated image. If the slope of the curve is less than the second slope threshold, it is judged that the generated image after editing has reached convergence; The hyperparameter-based convergence judgment method is used to set the corresponding hyperparameter threshold according to the training-related hyperparameters. When the hyperparameter threshold is reached, it is judged that the edited generated image has reached convergence.

6. The method for countering attacks against text-driven face editing according to claim 1, characterized in that: During the optimization process of the iterative optimization step, the weighting factors gradually converge, and the converged weighting factors are used as final weighting factors, thereby calculating the final inverse features of the edited image.

7. The method for countering attacks against text-driven face editing according to claim 1, characterized in that: The process of judging whether an image sample meets the standards includes: evaluating the quality of the image sample in terms of image size, image resolution and image quality indicators to determine whether they meet the standards; and judging whether the content of the image sample is a complete human face and whether the facial features are clear.

8. The method for countering attacks against text-driven face editing according to claim 1, characterized in that: The fixed semantic mapping model is a pre-trained multi-layer perceptron network.

9. A text-driven face editing anti-attack device, characterized in that: include: an acquisition module configured to acquire image samples and text samples, wherein the image samples include original image samples and target image samples, and determine a fixed semantic mapping model according to the text samples; The optimization module is configured to perform the following steps: inputting the original image sample into a pre-established and trained image inverse model to obtain original image inverse features; inputting the original image inverse features into a fixed semantic mapping model, performing a weighted combination of the output of the fixed semantic mapping model and the original image inverse features to obtain edited image inverse features; the weighting factor in the weighted combination is an optimization hyperparameter; inputting the edited image inverse features into an image generator to obtain an edited generated image; Determining whether the edited generated image converges; If convergence does not occur, iteratively optimizing the weighting factor. The optimization process of the weighting factor includes: inputting the edited generated image and the target image sample into a preset generalized face verification model, calculating the similarity of the output results as the face attack loss function value, and optimizing the weighting factor as feedback; The generation module is configured to obtain the final edited image reverse features based on the converged weighting factors, and input the final edited image reverse features into the image generator to obtain the final edited generated image.

10. The text-driven face editing counterattack device according to claim 9, characterized in that: The loss function for optimizing the weighting factor also includes but is not limited to the following loss functions: Image quality loss function, which uses a pre-trained deep neural network for image classification to extract semantic features of the original image sample and the edited image respectively, thereby calculating the image quality; The face identity loss function uses a pre-trained deep neural network for face recognition to extract the identity features of the original image sample and the edited image respectively, thereby calculating the face identity similarity; The face parsing loss function uses a pre-trained deep neural network for facial feature parsing to extract the facial feature segmentation features of the original image sample and the edited image, thereby calculating the similarity of the facial feature segmentation features.

11. The text-driven face editing counterattack device according to claim 9, characterized in that: Methods for determining whether the generated image after editing is convergent include but are not limited to any one of the following determination methods or a combination of multiple determination methods: A convergence judgment method based on an optimizable hyperparameter is used to judge whether the slope of the curve of the optimizable hyperparameter during the editing and generating process of the generated image is less than a preset first slope threshold according to the trend of the curve of the optimizable hyperparameter. If the slope of the curve is less than the first slope threshold, it is judged that the generated image after editing has reached convergence; A convergence judgment method based on the overall loss function is used to judge whether the slope of the curve is less than a preset second slope threshold according to the curve trend of the overall loss function in the process of generating the edited generated image. If the slope of the curve is less than the second slope threshold, it is judged that the generated image after editing has reached convergence; The hyperparameter-based convergence judgment method is used to set the corresponding hyperparameter threshold according to the training-related hyperparameters. When the hyperparameter threshold is reached, it is judged that the edited generated image has reached convergence.

12. The text-driven face editing counterattack device according to claim 9, characterized in that: During the iterative optimization process of the weighting factors, the weighting factors gradually converge, and the converged weighting factors are used as the final weighting factors, thereby calculating the final inverse features of the edited image.

13. An electronic device, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the text-driven face editing counterattack method according to any one of claims 1 to 8.

14. The electronic device according to claim 13, wherein: The electronic device further includes an input device and an output device, and the memory, processor, input device and output device are all connected to a bus.

15. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the method for countering attacks of text-driven face editing described in any one of claims 1-8 is implemented.

Images