Image processing method and device and related equipment

Abstract

The invention provides an image processing method and device and related equipment, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-processed face image of a target object; according to a to-be-processed face image of the target object, semantic coding information corresponding to the face image is determined; determining confrontation semantic coding information of the face image according to the semantic coding information corresponding to the face image; generating a face noise coding image of the target object according to the semantic coding information corresponding to the face image and a to-be-processed face image of the target object; and generating a face processing image of the target object according to the adversarial semantic coding information of the face image and the face noise coding image of the target object. Visual identity features of the input image can be better reserved, a more natural and effective face processing image is generated, and face information protection of the target object is better achieved.

Description

Background Art

[0002] Face recognition is a biometric technology that identifies people based on facial feature information. It is achieved by extracting facial features through deep neural networks (CNN). In the traditional face recognition process, facial image information is rarely protected, and there is a risk of privacy leakage. Once the user's facial image information is leaked, it may be used maliciously and endanger the user's interests and security. Recently, some studies have used adversarial samples to protect photos. However, whether based on adding noise, based on Patch (referring to the patching and upgrading of existing software or systems) or based on Generative Adversarial Network (GAN), the quality of adversarial images generated is poor, there is a large gap with the original images in the real world, it is difficult to produce natural visual perception, and the attack success rate is low, it is difficult to achieve good information protection.

[0003] In the past, generative adversarial networks and variational autoencoders have always dominated the field of image generation. However, training adversarial networks is a complex and difficult process. The balance of the adversarial network between the generator and the discriminator cannot be well measured, and it can only generate similar samples already in the training set. Variational autoencoders generate constrained latent space distributions through encoders, and the quality of generated images is unstable. It is difficult to learn complex data distributions. At the same time, the adversarial images they generate are of poor quality and low accuracy. There is a large gap with the original images in the real world, the faces are not harmonious, and they are not user-friendly.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides an image processing method, apparatus and related equipment, which at least to a certain extent overcome the problem of improper protection of facial image information and leakage of facial information in related technologies.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, there is provided an image processing method, including: acquiring a facial image of a target object to be processed; determining semantic coding information corresponding to the facial image based on the facial image of the target object to be processed; determining adversarial semantic coding information of the facial image based on the semantic coding information corresponding to the facial image; generating a facial noise coded image of the target object based on the semantic coding information corresponding to the facial image and the facial image of the target object to be processed; generating a processed facial image of the target object based on the adversarial semantic coding information of the facial image and the facial noise coded image of the target object.

[0008] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, the semantic coding information corresponding to the facial image of the target object to be processed is determined according to the facial image of the target object to be processed, including: inputting the facial image of the target object to be processed into a pre-trained convolutional neural network model, and outputting the feature information in the facial image; converting the feature information in the facial image into semantic coding information through a semantic encoder.

[0009] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, the adversarial semantic coding information of the facial image is determined according to the semantic coding information corresponding to the facial image, including: iteratively calculating the semantic coding information corresponding to the facial image through a gradient descent algorithm to obtain the adversarial semantic coding information of the facial image.

[0010] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, a facial noise coded image of the target object is generated according to the semantic coding information corresponding to the facial image and the facial image of the target object to be processed, including: inputting the semantic coding information corresponding to the facial image and the facial image of the target object to be processed into a pre-trained conditional denoising diffusion implicit coding model, and outputting the noise coded image of the target object, wherein the conditional denoising diffusion implicit coding model converts the input facial image into a noise coded image by gradually adding noise in T forward steps.

[0011] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, a facial processed image of the target object is generated according to the adversarial semantic coding information of the facial image and the facial noise coded image of the target object, including: inputting the adversarial semantic coding information of the facial image and the facial noise coded image of the target object into a pre-trained conditional denoising diffusion implicit decoding model, and outputting the facial processed image of the target object, wherein the conditional denoising diffusion implicit decoding model converts the input noise coded image into a facial processed image by gradually denoising in T reverse steps.

[0012] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, a facial processing image of the target object is generated according to the adversarial semantic coding information of the facial image and the facial noise coding image of the target object, including: generating an attack image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coding image of the target object; extracting semantic graphs in the attack image and the facial image of the target object; determining a target loss function according to the semantic graph of the attack image and the semantic graph of the facial image; until the target loss function converges to a preset threshold, determining that the attack image of the target object is the facial processing image of the target object.

[0013] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, a target loss function is determined according to the semantic map of the attack image and the semantic map of the facial image, including: determining a first loss function according to the similarity between the semantic map of the attack image and the semantic map of the facial image; determining a second loss function according to the distance between the feature vectors in the attack image and the facial image; and determining a target loss function through the first loss function and the second loss function.

[0014] According to another aspect of the present disclosure, there is also provided an image processing device, comprising: an image acquisition module, configured to acquire a facial image of a target object to be processed;

[0015] A semantic coding information determination module is used to determine the semantic coding information corresponding to the facial image of the target object to be processed according to the facial image to be processed; an anti-semantic coding information determination module is used to determine the adversarial semantic coding information of the facial image according to the semantic coding information corresponding to the facial image; a facial noise coding image generation module is used to generate a facial noise coding image of the target object according to the semantic coding information corresponding to the facial image and the facial image to be processed of the target object; a facial processed image generation module is used to generate a facial processed image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coding image of the target object.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned image processing methods by executing the executable instructions.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned image processing methods is implemented.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including: a computer program or instructions, wherein when the computer program or instructions are executed by a processor, any one of the above-mentioned image processing methods is implemented.

[0019] An image processing method, apparatus and related equipment provided in the embodiments of the present disclosure process the facial image of the target object to be processed to obtain adversarial semantic coding information of the facial image, and obtain a facial processed image of the target object based on the adversarial semantic coding information and the facial noise coded image of the target object containing noise. Compared with the adversarial images generated in the related art, the quality is poor and there is a large gap with the original images in the real world, it is difficult to produce natural visual perception, and the attack success rate is low, and it is difficult to achieve better information protection. In the embodiments of the present disclosure, the facial processed image generated by the adversarial semantic coding information of the facial image and the facial noise coded image of the target object can better retain the visual identity features of the input image, is more natural, achieves a balance between information protection and visual effects, and closely combines face recognition with facial information protection.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 A schematic diagram showing an exemplary application system architecture of an image processing method in an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of an image processing method in an embodiment of the present disclosure is shown;

[0024] Figure 3 A schematic diagram of a target loss function determination process in an embodiment of the present disclosure is shown;

[0025] Figure 4 A schematic diagram of a facial information protection implementation process in an embodiment of the present disclosure is shown;

[0026] Figure 5 A schematic diagram of the overall operation flow of generating a facial information protection image in an embodiment of the present disclosure is shown;

[0027] Figure 6A schematic diagram of an image processing device according to an embodiment of the present disclosure is shown;

[0028] Figure 7 A schematic diagram of an electronic device to which an image processing method is applied in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0030] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0031] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0032] For ease of understanding, several terms involved in the present disclosure are explained below:

[0033] Conditional Denoising Diffusion Implicit Model (Conditional DDIM): It is an extended version of DDIM, which can generate data samples that meet specific conditions under given conditions and can be used to generate natural images and adversarial samples.

[0034] Denoising Diffusion Implicit Model Decoder (DDIMdec): A reverse denoising generation process that aims to restore the original noisy image to a reconstructed image by gradually denoising in T reverse steps.

[0035] Conditional Denoising Diffusion Implicit Model Encoder (DDIMenc): A forward diffusion process that transforms an input image into a noisy image by gradually adding noise in T forward steps.

[0036] Figure 1 FIG. 2 shows an exemplary application system architecture diagram to which the image processing method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0037] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0038] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0039] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0040] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0041] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 301. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0042] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0043] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0044] Under the above system architecture, an image processing method is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities.

[0045] In some embodiments, the image processing method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the image processing method provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the image processing method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0046] First of all, in response to the above-mentioned problems, the embodiments of the present disclosure provide a facial image information processing method that can be applied but is not limited to, and can be applied to any picture that requires information protection, or any frame image in a video. Specifically, the embodiments of the present disclosure obtain adversarial semantic coding information of the facial image by processing the facial image to be processed of the target object, and obtain a facial processed image of the target object based on the adversarial semantic coding information and the facial noise coded image containing noise of the target object. Compared with the adversarial images generated in the related art, the quality is poor, there is a large gap with the original images of the real world, it is difficult to produce natural visual perception, and the attack success rate is low, and it is difficult to achieve better information protection. In the embodiments of the present disclosure, the facial processed image generated by the adversarial semantic coding information of the facial image and the facial noise coded image of the target object can better retain the visual identity characteristics of the input image, is more natural, achieves a balance between information protection and visual effects, and closely combines face recognition with facial information protection.

[0047] Figure 2 A schematic diagram of an image processing method in an embodiment of the present disclosure is shown, and the method comprises the following steps:

[0048] S202, obtaining a facial image of the target object to be processed.

[0049] It should be noted that the facial image in the embodiments of the present disclosure may be any image containing facial information to be processed. In some embodiments, it may be an image containing facial information in a frame in any video, or a photo containing facial information taken by a user through a camera, for example, a picture taken when a user uses face-swiping payment.

[0050] S204: Determine semantic coding information corresponding to the facial image according to the facial image to be processed of the target object.

[0051] It should be noted that the semantics in the embodiments of the present disclosure refers to the actual meaning or content contained in the facial image, such as the category of the target object, the description of the scene in which the target object is located, etc. The encoding is the process of converting the facial image data of the target object to be processed into another form (usually a numerical vector or matrix). In other words, the semantic coding information in the embodiments of the present disclosure refers to the conversion of the semantic information in the acquired facial image of the target object into information that can be understood and processed by a computer.

[0052] S206, determining adversarial semantic coding information of the facial image according to the semantic coding information corresponding to the facial image.

[0053] It should be noted that the adversarial semantic coding information in the embodiments of the present disclosure is obtained through an adversarial training process, and is intended to control or remove certain specific semantic features, such as removing or weakening the gender, age, etc. of the target object. The adversarial semantic coding information in the embodiments of the present disclosure is generally used to ensure that certain unnecessary features can be ignored when generating or processing data while retaining other important semantic information.

[0054] S208, generating a facial noise coded image of the target object according to the semantic coding information corresponding to the facial image and the facial image of the target object to be processed.

[0055] It should be noted that the facial noise coded image in the embodiment of the present disclosure can be obtained by adding random noise to the facial image of the target object to be processed. By introducing noise, facial features can be concealed or blurred, thereby protecting personal privacy. For example, specific noise can be added to the facial image of the target object to be processed, so that the facial features become unrecognizable, but other useful information of the image is still retained.

[0056] S210, generating a facial processed image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coded image of the target object.

[0057] The image processing method provided in the embodiment of the present disclosure first obtains a facial image of a target object to be processed; secondly, determines semantic coding information corresponding to the facial image according to the facial image to be processed; then, determines adversarial semantic coding information of the facial image according to the semantic coding information corresponding to the facial image; then, generates a facial noise coding image of the target object according to the semantic coding information corresponding to the facial image and the facial image to be processed of the target object; finally, generates a facial processed image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coding image of the target object. Compared with the adversarial images generated in the related art, which have poor quality and a large gap with the original images of the real world, it is difficult to produce natural visual perception, and the attack success rate is low, making it difficult to achieve good information protection. The facial processed image generated by the embodiment of the present disclosure through the adversarial semantic coding information of the facial image and the facial noise coding image of the target object can better retain the visual identity features of the input image, is more natural, achieves a balance between information protection and visual effects, and closely combines face recognition with facial information protection.

[0058] In some embodiments, the disclosed embodiment determines the semantic coding information corresponding to the facial image according to the facial image to be processed of the target object, including: inputting the facial image to be processed of the target object into a pre-trained convolutional neural network model, outputting the feature information in the facial image; converting the feature information in the facial image into semantic coding information through a semantic encoder. Specifically, the disclosed embodiment uses a pre-trained convolutional neural network model as a feature extractor of the facial image to be processed of the target object, and the pre-trained convolutional neural network model can automatically extract high-level features from the facial image without manually designing a feature extraction algorithm, reducing the workload of manual feature engineering, and can capture complex patterns in the image; then the extracted feature information is converted into a vector in the semantic space through a semantic encoder to obtain semantic coding information, and the feature information is converted into semantic coding information through the semantic encoder, so that the semantic information in the image can be better retained. This coding information can not only reflect the basic features of the image, but also capture deeper meanings. Moreover, through the semantic encoder, certain sensitive information in the facial image, such as personal identity information, can be removed or weakened, thereby protecting privacy while retaining useful information.

[0059] In some embodiments, the disclosed embodiments determine adversarial semantic coding information of a facial image based on semantic coding information corresponding to the facial image, including: iteratively calculating the semantic coding information corresponding to the facial image through a gradient descent algorithm to obtain adversarial semantic coding information of the facial image. In more detail, the disclosed embodiments can control specific attributes in facial images through adversarial semantic coding information, better protect privacy and reduce bias in practical applications. Specifically, in the disclosed embodiments, the adversarial semantic coding information is initially the semantic coding information corresponding to the facial image, and then each iteration updates the noise based on the gradient change. By optimizing the adversarial semantic coding information instead of directly adding noise in the pixel space, the generated protected facial processed image can be more natural and inconspicuous.

[0060] In some embodiments, the disclosed embodiments generate a facial noise coded image of a target object based on semantic coding information corresponding to the facial image and the facial image of the target object to be processed, including: inputting the semantic coding information corresponding to the facial image and the facial image of the target object to be processed into a pre-trained conditional denoising diffusion implicit coding model, and outputting the noise coded image of the target object, wherein the conditional denoising diffusion implicit coding model converts the input facial image into a noise coded image by gradually adding noise in T forward steps. Specifically, the embodiment of the present disclosure gradually adds noise to the input facial image in the forward diffusion process. In more detail, the conditional denoising diffusion implicit coding model in the embodiment of the present disclosure specifically includes an image input layer, a noise injection layer, an encoder, a latent variable sampling layer and a latent variable reconstruction layer, wherein, in the noise injection layer, random noise is added to the input facial image to create a noisy image; the encoder is usually composed of a convolutional neural network, which is responsible for converting the noisy image into a representation in the latent space, and the output of the encoder is the mean and variance of the latent variable, which is used for the subsequent sampling process; the latent variables are sampled from the mean and variance of the encoder output; the latent variable reconstruction layer converts the sampled latent variables into feature representations of the reconstructed image; in particular, the specific encoder structure can be adjusted and expanded according to task requirements and data characteristics. In practical applications, deeper and more complex neural network structures may be used, and additional layers and modules may be added as needed to improve the performance of the model.

[0061] In some embodiments, the conditional denoising diffusion implicit coding model in the embodiments of the present disclosure is an encoder network model in the denoising diffusion implicit model. The denoising diffusion implicit model is a generative model based on the diffusion process. It uses a random perturbation method to gradually convert the distribution of real data into a distribution that is easy to model through a series of noise layers. The basic idea of ​​the denoising diffusion implicit model is to gradually blur the image by introducing noise, and then gradually remove the noise to restore the image to its original state. This process enables the model to gradually learn the true distribution of the data. During training, the denoising diffusion implicit model learns the parameters of the model by minimizing the difference between the generated image and the real image, so that the generated image is as close to the real image as possible. The training process of the denoising diffusion implicit model usually includes multiple stages, each stage uses different degrees of noise to gradually approach the real distribution. The denoising diffusion implicit model is a powerful generative model that can synthesize more realistic and high-resolution images, has a more stable training process, can extract common style and content features of images through a small number of images, and generate new high-quality images based on these features.

[0062] In some embodiments, the disclosed embodiments generate a facial processed image of a target object based on the adversarial semantic coding information of a facial image and the facial noise coded image of the target object, including: inputting the adversarial semantic coding information of the facial image and the facial noise coded image of the target object into a pre-trained conditional denoising diffusion implicit decoding model, and outputting the facial processed image of the target object, wherein the conditional denoising diffusion implicit decoding model converts the input noise coded image into the facial processed image by gradually denoising in T reverse steps. In more detail, the conditional denoising diffusion implicit decoding model in the disclosed embodiments is typically a mirror image of the conditional denoising diffusion implicit coding model, including: a latent variable input layer for accepting latent variables sampled from an encoder as input; a latent variable mapping layer for mapping latent variables to a high-dimensional space, implemented by a fully connected layer; a decoder, typically composed of a deconvolutional neural network, responsible for mapping latent variables to feature representations of reconstructed images; and a reconstructed image output layer for outputting the reconstructed image.

[0063] In some embodiments, the disclosed embodiments generate a facial processing image of a target object based on adversarial semantic coding information of a facial image and a facial noise coding image of the target object, including: generating an attack image of the target object based on adversarial semantic coding information of the facial image and a facial noise coding image of the target object; extracting semantic graphs in the attack image and the facial image of the target object; determining a target loss function based on the semantic graph of the attack image and the semantic graph of the facial image; until the target loss function converges to a preset threshold, determining that the attack image of the target object is the facial processing image of the target object. Specifically, the disclosed embodiments iteratively calculate the semantic coding information corresponding to the facial image through a gradient descent algorithm, thereby calculating the target loss function until it converges to a preset threshold, determining that the attack image of the target object is the facial processing image of the target object, and generating a facial processing image that better meets the expectations of the target object, for example, gradually changing the face shape of the target object from a heart shape to a square, to ensure that the attack image of the target object can deceive the facial recognition model.

[0064] In some embodiments, Figure 3 As shown, the embodiment of the present disclosure determines the target loss function according to the semantic graph of the attack image and the semantic graph of the facial image, including:

[0065] S302, determining a first loss function according to the similarity between the semantic graph of the attack image and the semantic graph of the facial image.

[0066] S304, determining a second loss function according to the distance between the feature vectors in the attack image and the facial image.

[0067] S306, determining a target loss function through the first loss function and the second loss function.

[0068] Specifically, the first loss function in the embodiment of the present disclosure is to calculate the facial similarity between the generated attack image and the facial image to be processed, gradually change the facial shape of the generated attack image, maintain the facial semantic consistency between the generated attack image and the facial image to be processed, so as to generate an image that better meets the user's expectations. In more detail, the first loss function is obtained by respectively extracting the semantic graphs of the input facial image and the generated attack image through a facial parsing network model, and calculating the cross entropy between them; the embodiment of the present disclosure uses the second loss function to close the distance between the finally generated facial processing image and the facial image to be processed. Furthermore, the embodiment of the present disclosure denotes the first loss function as loss1, denotes the second loss function as loss2, and determines the target loss function as loss through the following formula (1):

[0069] loss=mloss1+loss2 (1)

[0070] Among them, m represents the weight information, which is used to balance the facial semantic consistency loss (loss1) and the distance loss (loss2) between the generated image and the specified image in the targeted attack.

[0071] In some embodiments, the disclosed embodiments find an optimal balance point by adjusting m, so that the generated image can maintain the facial similarity with the original image and meet the demand of being close to the target image. For example, the target object expects to change the face shape from heart shape to square while maintaining the similarity of facial features with the original image: if m is set to a higher value (for example, m=0.8), the generated image will pay more attention to maintaining the similarity of facial features, but will also gradually change the face shape to make it closer to the square; if m is set to a lower value (for example, m=0.2), the generated image will pay more attention to being close to the square face shape, but may sacrifice the similarity of some facial features. By adjusting m, the generated facial processed image is promoted to maintain similar facial semantic features with the original facial image, thereby better preserving the visual identity.

[0072] In some embodiments, the conditional denoising diffusion implicit model in the embodiments of the present disclosure includes a conditional denoising diffusion implicit encoding model and a conditional denoising diffusion implicit decoding model, wherein the conditional denoising diffusion implicit encoding model converts the input facial image into a noisy image by gradually adding noise in T forward steps, and the conditional denoising diffusion implicit decoding model restores the original noisy image to obtain a reconstructed image by gradually denoising in T reverse steps. The embodiments of the present disclosure can better control the visual quality and semantic features of the generated adversarial samples by separating the encoding methods of the forward diffusion and reverse denoising generation processes.

[0073] In some embodiments, the disclosed embodiments convert an input facial image into corresponding semantic coding information and a facial noise coding image of a target object based on a semantic encoder and a conditional denoising diffusion implicit model, and obtain a facial processed image by further iteratively optimizing the adversarial semantic coding information, thereby deceiving the facial recognition model and protecting facial privacy.

[0074] In some embodiments, Figure 4 As shown, the overall operation process of generating a facial information protection image of a target object in the embodiment of the present disclosure includes:

[0075] S402, provide an original facial image of a human face and input it into a semantic encoder and a conditional denoising diffusion implicit model; the semantic encoder maps the input facial image x to semantic encoding information z that captures high-level facial information, expressed as z=Enc(x), where Enc represents a semantic encoder, which is used to convert the input data (here refers to the facial image x) into a compact representation, namely the semantic encoding information z.

[0076] S404, the conditional DDIM coding model takes the semantic coding information z and the facial image x as input, trains the noise adding network, gradually adds noise through T forward iteration steps, and finally realizes converting the input facial image x into the facial noise coding image x T , denoted by x T =DDIMenc(x, z), where DDIMenc represents the conditional denoising diffusion implicit coding model, which is used to convert the input data (here refers to the semantic coding information z and the facial image x) into the facial noise coding image x T .

[0077] S406, conditional DDIM decoding model to counter the semantic encoding information z adv and facial noise coded image x T As input, the noise in the facial noise coded image is gradually removed through T reverse iterations to achieve the original facial noise coded image x T Restore the reconstructed facial processed image x pro , denoted by x pro =DDIMdec(x T , z adv ), where DDIMdec represents the conditional denoising diffusion implicit decoding model, which is used to transform the input data (here refers to the adversarial semantic encoding information z adv and facial noise coded image x T ) is converted to a face processed image x pro .

[0078] S408, the facial processed image x proThe facial image x of the target object is input into the face recognition model, the similarity loss loss2 between the two images is calculated, and the facial processing image x is calculated. pro The facial similarity loss loss1 between the target object’s facial image x and the target object’s facial image x is used to determine the target loss function. By adjusting the weight information m in the target loss function, the generated facial processed image x is gradually changed. pro The facial shape of the generated facial processing image x pro The facial semantic consistency with the original facial image x can generate an image that is more in line with user expectations.

[0079] In some embodiments, Figure 5 As shown, the overall operation process for facial information protection image generation in the embodiment of the present disclosure includes: inputting the acquired facial image of the target object into the semantic encoder and the conditional DDIM coding model to obtain the semantic coding information and the facial noise coding image corresponding to the facial image of the target object, and then determining the adversarial semantic coding information according to the generated semantic coding information, inputting the adversarial semantic coding information and the facial noise coding image into the pre-trained conditional DDIM decoding model, gradually removing the noise in the facial noise coding image through T reverse iterations, and restoring the original facial noise coding image with added noise to obtain a reconstructed facial processing image, and then inputting the facial processing image and the facial image of the target object into the face recognition model, calculating the similarity loss between the two images, and calculating the facial semantic consistency loss between the facial processing image and the facial image of the target object, and updating and optimizing the adversarial semantic coding information according to the similarity loss between the facial processing image and the facial image of the target object, so as to make the generated facial processing image more natural and effective, and better realize the facial information protection of the target object.

[0080] Based on the same inventive concept, the present disclosure also provides an image processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0081] Figure 6 A schematic diagram of an image processing device in an embodiment of the present disclosure is shown, the device comprising:

[0082] An image acquisition module 601 is used to acquire a facial image of a target object to be processed;

[0083] The semantic coding information determination module 602 is used to determine the semantic coding information corresponding to the facial image according to the facial image to be processed of the target object;

[0084] The anti-semantic coding information determination module 603 is used to determine the anti-semantic coding information of the facial image according to the semantic coding information corresponding to the facial image;

[0085] The facial noise coded image generation module 604 is used to generate a facial noise coded image of the target object according to the semantic coding information corresponding to the facial image and the facial image to be processed of the target object;

[0086] The facial processed image generating module 605 is used to generate a facial processed image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coded image of the target object.

[0087] An image processing device provided in an embodiment of the present disclosure obtains a facial image of a target object to be processed through an image acquisition module; determines semantic coding information corresponding to the facial image according to the facial image to be processed of the target object through a semantic coding information determination module; determines adversarial semantic coding information of the facial image according to the semantic coding information corresponding to the facial image through an anti-semantic coding information determination module; generates a facial noise coding image of the target object according to the semantic coding information corresponding to the facial image and the facial image to be processed of the target object through a facial noise coding image generation module; generates a facial processing image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coding image of the target object through a facial processing image generation module. Compared with the adversarial images generated in the related art, which have poor quality and a large gap with the original images of the real world, it is difficult to produce natural visual perception, and the attack success rate is low, and it is difficult to achieve good information protection. The facial processing image generated by the adversarial semantic coding information of the facial image and the facial noise coding image of the target object in the embodiment of the present disclosure can better retain the visual identity characteristics of the input image, is more natural, achieves a balance between information protection and visual effects, and closely combines face recognition with facial information protection.

[0088] In some embodiments, the semantic coding information determination module in the embodiments of the present disclosure is also used to input the facial image of the target object to be processed into a pre-trained convolutional neural network model, and output the feature information in the facial image; and convert the feature information in the facial image into semantic coding information through a semantic encoder.

[0089] In some embodiments, the anti-semantic coding information determination module in the embodiments of the present disclosure is also used to iteratively calculate the semantic coding information corresponding to the facial image through a gradient descent algorithm to obtain the anti-semantic coding information of the facial image.

[0090] In some embodiments, the facial noise coded image generation module in the embodiments of the present disclosure is also used to input the semantic coding information corresponding to the facial image and the facial image of the target object to be processed into a pre-trained conditional denoising diffusion implicit coding model, and output the noise coded image of the target object, wherein the conditional denoising diffusion implicit coding model converts the input facial image into a noise coded image by gradually adding noise in T forward steps.

[0091] In some embodiments, the facial processed image generation module in the embodiments of the present disclosure is also used to input the adversarial semantic coding information of the facial image and the facial noise coded image of the target object into a pre-trained conditional denoising diffusion implicit decoding model, and output the facial processed image of the target object, wherein the conditional denoising diffusion implicit decoding model converts the input noise coded image into the facial processed image by gradually denoising in T reverse steps.

[0092] In some embodiments, the facial processing image generation module in the embodiments of the present disclosure is also used to generate an attack image of the target object based on the adversarial semantic coding information of the facial image and the facial noise coding image of the target object; extract semantic maps in the attack image and the facial image of the target object; determine the target loss function based on the semantic map of the attack image and the semantic map of the facial image; until the target loss function converges to a preset threshold, determine that the attack image of the target object is the facial processing image of the target object.

[0093] In some embodiments, the facial processing image generation module in the disclosed embodiments is also used to determine a first loss function based on the similarity between the semantic map of the attack image and the semantic map of the facial image; determine a second loss function based on the distance between the feature vectors in the attack image and the facial image; and determine a target loss function through the first loss function and the second loss function.

[0094] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0095] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned image processing methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above-mentioned method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0096] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0097] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including the storage unit 702 and the processing unit 701).

[0098] The storage unit stores program codes, which can be executed by the processing unit 701, so that the processing unit 701 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0099] In some embodiments, when the electronic device is used to control the image processing method disclosed above, for example, the processing unit 701 may perform the following steps of the above method embodiment:

[0100] Acquire a facial image of a target object to be processed; determine semantic coding information corresponding to the facial image based on the facial image to be processed; determine adversarial semantic coding information of the facial image based on the semantic coding information corresponding to the facial image; generate a facial noise coded image of the target object based on the semantic coding information corresponding to the facial image and the facial image of the target object to be processed; generate a processed facial image of the target object based on the adversarial semantic coding information of the facial image and the facial noise coded image of the target object.

[0101] The storage unit 702 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7021 and / or a cache storage unit 7022 , and may further include a read-only storage unit (ROM) 7023 .

[0102] The storage unit 702 may also include a program / utility 7024 having a set (at least one) of program modules 7025, such program modules 7025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0103] Bus 703 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0104] The electronic device 700 may also communicate with one or more external devices 704 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or may communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication may be performed through an input / output (I / O) interface 705. In addition, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 706. As shown, the network adapter 706 communicates with other modules of the electronic device 700 through a bus 703. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0105] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0106] Based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, any of the above-mentioned image processing methods is implemented. Since the principle of solving the problem in the embodiment of the computer-readable storage medium is similar to that in the above-mentioned method embodiment, the implementation of the embodiment of the computer-readable storage medium can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0107] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0109] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0110] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0111] Based on the same inventive concept, a computer program product is also provided in the embodiment of the present disclosure, including a computer program product, including: a computer program or an instruction, and when the computer program or the instruction is executed by a processor, the image processing method of any one of the above method embodiments is implemented. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0112] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0113] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0114] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0115] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. An image processing method, characterized in that: include: Obtaining a facial image of a target object to be processed; Determining semantic coding information corresponding to the facial image according to the facial image to be processed of the target object; Determining adversarial semantic coding information of the facial image according to the semantic coding information corresponding to the facial image; Generate a facial noise coded image of the target object according to the semantic coding information corresponding to the facial image and the facial image of the target object to be processed; A facial processed image of the target object is generated according to the adversarial semantic coding information of the facial image and the facial noise coded image of the target object.

2. The image processing method according to claim 1, characterized in that: Determining semantic coding information corresponding to the facial image according to the target object to be processed includes: Inputting the target object's to-be-processed facial image into a pre-trained convolutional neural network model, and outputting feature information in the facial image; The feature information in the facial image is converted into semantic coding information through a semantic encoder.

3. The image processing method according to claim 1, characterized in that: Determining adversarial semantic coding information of the facial image according to semantic coding information corresponding to the facial image includes: The semantic coding information corresponding to the facial image is iteratively calculated through a gradient descent algorithm to obtain adversarial semantic coding information of the facial image.

4. The image processing method according to claim 1, characterized in that: Generating a facial noise coded image of the target object according to the semantic coding information corresponding to the facial image and the facial image of the target object to be processed, comprising: The semantic coding information corresponding to the facial image and the facial image to be processed of the target object are input into a pre-trained conditional denoising diffusion implicit coding model, and a noise coded image of the target object is output, wherein the conditional denoising diffusion implicit coding model converts the input facial image into a noise coded image by gradually adding noise in T forward steps.

5. The image processing method according to claim 1, characterized in that: Generating a processed facial image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coded image of the target object, comprising: The adversarial semantic coding information of the facial image and the facial noise coded image of the target object are input into a pre-trained conditional denoising diffusion implicit decoding model, and a facial processed image of the target object is output, wherein the conditional denoising diffusion implicit decoding model converts the input noise coded image into a facial processed image by gradually denoising in T reverse steps.

6. The image processing method according to claim 1, characterized in that: Generating a processed facial image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coded image of the target object, comprising: generating an attack image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coding image of the target object; Extracting semantic graphs from the attack image and the facial image of the target object; Determining a target loss function according to the semantic map of the attack image and the semantic map of the facial image; Until the target loss function converges to a preset threshold, it is determined that the attack image of the target object is the face-processed image of the target object.

7. The image processing method according to claim 6, characterized in that: Determining a target loss function according to the semantic graph of the attack image and the semantic graph of the facial image includes: Determining a first loss function according to a similarity between a semantic map of the attack image and a semantic map of the facial image; determining a second loss function based on a distance between feature vectors in the attack image and the facial image; A target loss function is determined by using the first loss function and the second loss function.

8. An image processing device, characterized in that: include: An image acquisition module is used to acquire a facial image of a target object to be processed; A semantic coding information determination module, used to determine the semantic coding information corresponding to the facial image according to the facial image to be processed of the target object; An anti-semantic coding information determination module, used to determine the anti-semantic coding information of the facial image according to the semantic coding information corresponding to the facial image; A facial noise coded image generation module, used to generate a facial noise coded image of the target object according to the semantic coding information corresponding to the facial image and the facial image to be processed of the target object; The facial processed image generation module is used to generate a facial processed image of the target object according to the adversarial semantic coding information of the facial image and the facial noise coded image of the target object.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the image processing method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the image processing method according to any one of claims 1 to 7.

Images