Image processing method, electronic device and computer program product

Abstract

The invention discloses an image processing method, electronic equipment and a computer program product, and relates to the technical field of image processing. The method comprises the steps that a to-be-protected image is acquired, a to-be-protected object is extracted from the to-be-protected image, the to-be-protected object is input into a trained generative adversarial network, a disturbance mask corresponding to the to-be-protected object is acquired, the disturbance mask comprises modification information of the to-be-protected object, the disturbance mask is added to the to-be-protected object, and a target adversarial object is acquired; and replacing the to-be-protected object in the to-be-protected image with the target confrontation object to obtain a protected image. According to the invention, the mask protection is carried out on the to-be-protected object in the image without influencing the beauty, so that the security can be improved on the basis of ensuring the beauty.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image processing method, an electronic device, and a computer program product. Background Art

[0002] In today's big data era, with the rapid development of technology, the field of face recognition has gradually improved, and various face recognition technologies have been used in various fields of life, but the widespread use of face recognition systems has also brought a series of problems. At present, the most widely mentioned issue is the privacy protection of faces. Face data is of vital importance. This data is directly related to personal property security. However, there is currently a lack of protection for face data, resulting in poor security. Summary of the invention

[0003] The present application proposes an image processing method, an electronic device and a computer program product.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: acquiring an image to be protected; extracting an object to be protected from the image to be protected; inputting the object to be protected into a trained generative adversarial network to obtain a perturbation mask corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected; adding the perturbation mask to the object to be protected to obtain a target adversarial object; replacing the object to be protected in the image to be protected with the target adversarial object to obtain a protected image.

[0005] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0006] In a third aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program implements the above method when executed.

[0007] The image processing method, electronic device, and computer program product provided by the embodiments of the present application obtain an image to be protected, extract an object to be protected from the image to be protected, input the object to be protected into a trained generative adversarial network, and obtain a perturbation mask corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected, adds a perturbation mask to the object to be protected, obtains a target adversarial object, replaces the object to be protected in the image to be protected with the target adversarial object, and obtains a protected image, thereby improving security while ensuring aesthetics by masking the object to be protected in the image without affecting aesthetics. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0009] Figure 1 A schematic diagram of a process flow of an image processing method provided by an embodiment of the present application is shown;

[0010] Figure 2 A schematic diagram of a process flow of an image processing method provided by an embodiment of the present application is shown;

[0011] Figure 3 The overall flow chart of the image processing method provided by the embodiment of the present application is shown;

[0012] Figure 4 A schematic diagram of a process flow of an image processing method provided by an embodiment of the present application is shown;

[0013] Figure 5 A schematic diagram of the model structure of a generative adversarial network provided in an embodiment of the present application is shown;

[0014] Figure 6 A module block diagram of an image processing device provided by an embodiment of the present application is shown;

[0015] Figure 7 A block diagram of an electronic device according to an embodiment of the present application for executing an image processing method according to an embodiment of the present application is shown;

[0016] Figure 8 A storage unit for storing or carrying program codes for implementing the image processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0018] At present, in order to protect facial data, the following methods are generally used:

[0019] First, in the physical world, real objects such as special glasses and special hats are put on real users. However, the production cost of wearing special glasses and special hats in the physical world is too high and inconvenient to carry, and it cannot protect digital images.

[0020] Second, make visible modifications to the facial features of digital face photos, such as applying eye shadow, etc. However, algorithmic modifications to the facial features of digital face photos have obvious changes, which makes the photos easily distorted. On the other hand, such changes can hardly be run in real time on the mobile phone side.

[0021] In view of the above problems, the inventors have discovered after long-term research and proposed the image processing method, electronic device and computer program product provided in the embodiments of the present application. By masking the objects to be protected in the image without affecting the appearance, the security can be improved while ensuring the appearance. That is, the image processing method of the embodiments of the present application can make it possible to modify the original portrait photo, and the user's naked eye will not see any changes, but the face recognition algorithm cannot distinguish that it is the same face. Among them, the specific image processing method is described in detail in the subsequent embodiments.

[0022] The following is an explanation of the terms involved in the embodiments of the present application.

[0023] GAN: Generative Adversarial Network (GAN) is a deep learning architecture. This architecture trains two neural networks to compete against each other to generate new and more realistic data from a given training dataset. For example, new images can be generated from an existing database of images, or original music can be generated from a database of songs. GAN is called an adversarial network because the architecture trains two different networks and pits them against each other. One network generates new data by taking input data samples and modifying them as much as possible. The other network tries to predict whether the generated data output belongs to the original dataset. In other words, the prediction network decides whether the generated data is fake or real. Newer and improved versions of the fake data values ​​are generated until the prediction network can no longer distinguish between the fake data values ​​and the original data values.

[0024] Face Detection: Face detection is a computer technology that finds the location and size of a face in any digital image. It detects facial features and ignores everything else such as buildings, trees, and bodies. Sometimes, face detection is also responsible for finding subtle features of the face, such as the fine location of the eyes, nose, mouth, etc.

[0025] DCGAN: DCGAN (or Deep Convolutional GAN) is a Generative Adversarial Network architecture. It uses some guiding principles, in particular: replacing any pooling layers with strided convolutions (discriminator) and fractionally strided convolutions (generator).

[0026] WGAN: The Wasserstein Generative Adversarial Network (WGAN) is a variant of the Generative Adversarial Network (GAN) proposed in 2017 to improve the stability of learning, eliminate problems such as mode collapse, and provide meaningful learning curves that can be used for debugging and hyperparameter search.

[0027] WGAN-GP: Wasserstein GAN + Gradient Penalty (or WGAN-GP) is a generative adversarial network that uses the Wasserstein loss formula plus a gradient norm penalty to achieve Lipschitz continuity.

[0028] MS-SSIM: A more advanced form of SSIM is called Multi-Scale SSIM (MS-SSIM), which operates at multiple scales through a multi-stage subsampling process, reminiscent of multi-scale processing in early vision systems. It has been shown to perform as well as or better than SSIM on different subjective image and video databases.

[0029] Pixel intensity: Pixel intensity refers to the sum, mean, or median of pixel intensities within a region of interest (ROI) in an image. It reflects the amount of tissue in that area and can be used to analyze articulator dynamics in computer science applications.

[0030] MobileNet-arcface: A face recognition algorithm that can extract features from digital face images.

[0031] Resnet-Facenet: A face recognition algorithm that can extract features from digital face images.

[0032] Embedding: can be simply understood as the features of digital faces.

[0033] SOTA: state-of-the-art model, the best effect. In the text, it means the best algorithm or model.

[0034] Upsamping: Upsampling an image or feature map.

[0035] See also Figure 1 , Figure 1 The flowchart of the image processing method provided by an embodiment of the present application is shown. The method is used to protect the objects to be protected in the image by masking them without affecting the appearance, thereby improving security while ensuring the appearance. In a specific embodiment, the image processing method is applied to Figure 6 The image processing device 200 and the electronic device 100 ( Figure 7). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment may include a smart phone, a tablet computer, etc., which is not limited here. Figure 1 The process shown is described in detail, and the image processing method may specifically include the following steps:

[0036] Step S110: Acquire the image to be protected.

[0037] In this embodiment, an image to be protected may be acquired, wherein the image to be protected refers to an image that requires privacy protection.

[0038] In some embodiments, the image to be protected can be stored locally on the electronic device, and accordingly, the image to be protected can be obtained from the local electronic device; the image to be protected can be stored on other devices connected to the electronic device, and accordingly, the image to be protected can be obtained from the other devices through network communication; the image to be protected can be stored on a server connected to the electronic device, and accordingly, the image to be protected can be obtained from the server through network communication, etc., which are not limited here.

[0039] In some embodiments, if the image to be protected is saved in the form of a single image, the image to be protected can be read directly; if the image to be protected is saved in the form of an image frame in a video, the image frame can be extracted from the video, and the extracted image frame can be obtained as the image to be protected.

[0040] Step S120: extracting the object to be protected from the image to be protected.

[0041] In this embodiment, when an image to be protected is acquired, it is possible to detect whether the image to be protected includes the object to be protected. If it is detected that the image to be protected includes the object to be protected, the object to be protected can be extracted from the image to be protected; if it is detected that the image to be protected does not include the object to be protected, the image to be protected may not be protected.

[0042] Optionally, the object to be protected may include: face, portrait, text, etc., which is not limited here.

[0043] In some embodiments, when an image to be protected is acquired, the image to be protected may be preprocessed, feature acquired, feature classified, detected and positioned, and image cropped in sequence to extract the object to be protected from the image to be protected.

[0044] The preprocessing of the image to be protected may include: performing operations such as denoising, image enhancement, and color space conversion on the image to be protected, so as to improve the image quality and reduce the difficulty of subsequent processing.

[0045] The feature acquisition of the preprocessed image to be protected may include: extracting features from the preprocessed image to be protected using a specific algorithm or technology, and the extracted features may include color, shape, texture, etc.

[0046] Among them, feature classification of the extracted features may include: using a classifier (such as a support vector machine, a neural network, etc.) to classify the extracted features to determine whether the image to be protected includes the object to be protected, wherein the training of the classifier requires the use of an image data set including the object to be protected and the object not to be protected.

[0047] The detection and positioning of the object to be protected may include: if it is detected that the image to be protected includes the object to be protected, the classifier will output the position information (such as the coordinates of the bounding box) of the object to be protected.

[0048] The image clipping of the object to be protected may include: clipping a region including the object to be protected from the image to be protected according to the position information of the object to be protected output by the classifier, so as to extract the object to be protected from the image to be protected.

[0049] As an implementable manner, the object to be protected includes a face. Then, if it is detected that the image to be protected includes the object to be protected, extracting the object to be protected from the image to be protected may include: performing face detection on the image to be protected, and if it is detected that the image to be protected includes a face, the face can be extracted from the image to be protected. Optionally, the purpose of face detection is to detect whether there is a face in the image to be protected, and when there is a face, the specific location of the face, and the SOTA face detection model can be used for face detection.

[0050] Step S130: inputting the object to be protected into the trained generative adversarial network to obtain a perturbation mask corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected.

[0051] In this embodiment, a generative adversarial network may be pre-trained to obtain a trained generative adversarial network. When the object to be protected is obtained, the object to be protected may be input into the trained generative adversarial network to obtain a perturbation mask (or noise map) corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected.

[0052] As an implementable manner, the trained generative adversarial network includes a generator, wherein the generator is usually a convolutional neural network (CNN), which can capture the features of the input image (the object to be protected in this embodiment) through learning, and generate new data (the perturbation mask in this embodiment) based on these features. The design of the generator may include multiple convolutional layers, activation layers (such as ReLU, LeakyReLU), and possible deconvolution layers (or transposed convolution layers) to maintain the size of the image or perform upsampling when output. Based on this, when the object to be protected is obtained, the object to be protected can be input into the generator of the trained generative adversarial network, and the perturbation mask corresponding to the object to be protected is generated by the generator of the trained generative adversarial network, and the perturbation mask corresponding to the object to be protected is output.

[0053] Among them, in a specific implementation, the object to be protected can be provided as input to the generator of the trained generative adversarial network. The generator of the trained generative adversarial network generates a corresponding perturbation mask based on the input object to be protected and the distribution and transformation rules of the image data it has learned. The perturbation mask can be an image with the same shape as the object to be protected, but its pixel value represents the modification information that needs to be applied to the object to be protected. Optionally, the modification information may include pixel-level noise, indication of blurred areas, or other forms of transformation, etc., which are not limited here.

[0054] In some embodiments, inputting the object to be protected into a trained generative adversarial network to obtain a perturbation mask corresponding to the object to be protected may include: normalizing the object to be protected to obtain the normalized object to be protected, inputting the normalized object to be protected into the trained generative adversarial network to obtain a perturbation mask corresponding to the normalized object to be protected.

[0055] Among them, when the object to be protected is obtained, the digital matrix of the object to be protected can be obtained, and the digital matrix of the object to be protected can be normalized to the interval of [-1, 1]. For example, ((x-125) / 125) can be used to implement it, and then it is input into the trained generative adversarial network as the input data of the trained generative adversarial network to obtain the trained generative adversarial network output perturbation mask. It can be understood that the original pixel value of the object to be protected is usually an integer of 0-255, which can be converted to a numerical range suitable for network processing. Specifically, it can be normalized by ((x-125) / 125) to scale the pixel value from [0, 255] to the interval of [-1, 1] to meet the data processing requirements of the neural network.

[0056] Step S140: adding the disturbance mask to the object to be protected to obtain a target adversarial object.

[0057] In this embodiment, when a perturbation mask corresponding to the object to be protected is obtained, the perturbation mask can be added to the object to be protected to obtain a target adversarial object. The generated perturbation mask is applied to the object to be protected, which usually involves modifying the pixel value of the object to be protected pixel by pixel or region by region to achieve a protection effect. For example, the final target adversarial object, i.e., the protected image, can be generated by fusing the perturbation mask with the object to be protected in some form (such as weighted sum, element-by-element multiplication, element-by-element addition, etc.).

[0058] As an example, when a perturbation mask corresponding to the object to be protected is obtained, the perturbation mask and the object to be protected may be added pixel by pixel to obtain a target adversarial object.

[0059] Step S150: Replace the to-be-protected object in the to-be-protected image with the target confrontation object to obtain a protected image.

[0060] In this embodiment, when the target confrontation object is obtained, the object to be protected in the image to be protected can be replaced with the target confrontation object (i.e., the protected object) to obtain a protection image. At this time, the image to be protected and the protection image do not change to the naked eye of the user, that is, the user will think that the image to be protected and the protection image are consistent, but the face recognition algorithm cannot distinguish that they are the same face, that is, the face recognition algorithm will think that the image to be protected and the protection image are inconsistent, thereby improving security while ensuring aesthetics.

[0061] In some embodiments, when the target antagonistic object is obtained, the target antagonistic object can be copied and pasted into the image to be protected, that is, the pixel intensity (pixelintensity) of the position of the object to be protected in the image to be protected is set to zero, and then the target antagonistic object is placed in the area of ​​the object to be protected in the image to be protected. Specifically, in the image to be protected, the pixel intensity (i.e., pixel value) of the detected object to be protected is set to zero, for example, it can be achieved by setting all pixel values ​​in the face boundary box to black (i.e., RGB value is (0, 0, 0)), and if the background of the image to be protected is not pure black, the background can be sampled first, and then the area of ​​the object to be protected can be filled with the average color of the background or some interpolation method to avoid obvious color differences. Afterwards, the target antagonistic object is pasted into the area of ​​the object to be protected in the image to be protected. Of course, in order to make the pasted target antagonistic object more integrated with the image to be protected, some post-processing can also be performed, such as edge smoothing, color adjustment, etc., which are not limited here.

[0062] An image processing method provided by an embodiment of the present application obtains an image to be protected, extracts an object to be protected from the image to be protected, inputs the object to be protected into a trained generative adversarial network, obtains a perturbation mask corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected, adds a perturbation mask to the object to be protected, obtains a target adversarial object, replaces the object to be protected in the image to be protected with the target adversarial object, and obtains a protected image, thereby improving security while ensuring aesthetics by masking the object to be protected in the image without affecting aesthetics.

[0063] See also Figure 2 , Figure 2 The flowchart of the image processing method provided by an embodiment of the present application is shown in FIG. Figure 2 The process shown is described in detail, and the image processing method may specifically include the following steps:

[0064] Step S210: Acquire the image to be protected.

[0065] Step S220: extracting the object to be protected from the image to be protected.

[0066] For the detailed description of step S210 to step S220, please refer to step S110 to step S120, which will not be repeated here.

[0067] Step S230: aligning the object to be protected to obtain the object to be protected after posture correction.

[0068] It is understandable that after the object to be protected is extracted from the image to be protected, the object to be protected is likely to have a certain posture. Therefore, the object to be protected can be aligned, the purpose of which is to correct the posture of the object to be protected and obtain the object to be protected after posture correction. This makes it convenient to send it to the trained generative adversarial model to generate a perturbation mask for the object to be protected after posture correction.

[0069] As an example, assuming that the object to be protected is a human face, and after the face is extracted from the image to be protected, the face has a tilted head posture, then the face can be aligned to obtain a face with posture correction. At this time, the line connecting the two eyes in the face can be clearly horizontal.

[0070] In some implementations, after the object to be protected is extracted from the image to be protected, the object to be protected may be aligned by similarity transformation to obtain the object to be protected after posture correction. The formula of similarity transformation may be as follows:

[0071] In some embodiments, after the object to be protected is extracted from the image to be protected, the object to be protected can be aligned by affine transformation to obtain the object to be protected after posture correction. Affine transformation can be understood as an extension of similarity transformation, which allows the object to be protected to be translated, rotated, scaled, and tilted (i.e., non-uniform scaling) to more flexibly handle the tilt problem of the face at different angles in face alignment.

[0072] In some embodiments, after the object to be protected is extracted from the image to be protected, the object to be protected can be aligned by perspective transformation to obtain the object to be protected after posture correction. Perspective transformation is a further extension of affine transformation, which introduces more degrees of freedom and allows the object to be protected to undergo more complex deformation, so as to be used in face alignment to handle the deformation problem of the face at different viewing angles (such as looking up, looking down).

[0073] Of course, in this embodiment, the alignment of the object to be protected can be performed in many other ways, for example, a method based on feature points, a method based on three-dimensional modeling and alignment, a method based on deep learning, etc., which will not be repeated here.

[0074] Step S240: inputting the posture-corrected object to be protected into the trained generative adversarial network to obtain a perturbation mask corresponding to the posture-corrected object to be protected, wherein the perturbation mask includes modification information of the posture-corrected object to be protected.

[0075] In this embodiment, when the object to be protected after posture correction is obtained, the object to be protected after posture correction can be input into a trained generative adversarial network, and the trained generative adversarial network generates and outputs the object to be protected after posture correction. Accordingly, a perturbation mask corresponding to the object to be protected after posture correction can be obtained. It can be understood that the perturbation mask corresponding to the object to be protected after posture correction can include modification information of the object to be protected after posture correction.

[0076] In some embodiments, when the object to be protected after posture correction is obtained, the object to be protected after posture correction can be input into the generator of a trained generative adversarial network, and the trained generative adversarial network generator generates and outputs the object to be protected after posture correction. Accordingly, a perturbation mask corresponding to the object to be protected after posture correction output by the generator of the trained generative adversarial network can be obtained.

[0077] Step S250: adding the disturbance mask to the object to be protected to obtain a target adversarial object.

[0078] For the detailed description of step S250, please refer to step S140, which will not be repeated here.

[0079] Step S260: restoring the posture of the target confrontation object to obtain the target confrontation object after the posture restoration.

[0080] Among them, since the target antagonistic object corresponds to the object to be protected after posture correction, the target antagonistic object can also be considered to have undergone posture correction, that is, it is inconsistent with the original posture of the object to be protected. Therefore, in this embodiment, when the target antagonistic object is obtained, the posture of the target antagonistic object can be restored to obtain the target antagonistic object after posture restoration. It can be understood that the target antagonistic object after posture restoration has the same posture as the object to be protected.

[0081] As an example, suppose the object to be protected is a human face, and after the face is extracted from the image to be protected, the face has a tilted head posture, and the line connecting the two eyes in the face after posture correction is obviously horizontal. Then, the line connecting the two eyes in the target adversarial object (target adversarial face) is also obviously horizontal, so it is necessary to restore the posture of the target adversarial object to obtain the target adversarial object after posture restoration. At this time, the target adversarial object after posture restoration has a tilted head posture.

[0082] In some embodiments, it is assumed that the alignment of the objects to be protected is performed by the similarity transformation If it is realized, then we can The inverse matrix is ​​calculated, and then the pixel matrix formed by the target adversarial object is multiplied by the inverse matrix. This can be considered as the inverse operation of alignment. The aligned object to be protected is reversed to restore the original posture of the object to be protected in the image to be protected.

[0083] Step S270: replacing the object to be protected in the image to be protected with the target confrontation object after the posture restoration, to obtain the protection image.

[0084] In this embodiment, when the target antagonistic object after posture restoration is obtained, the image to be protected in the image to be protected may be replaced with the target antagonistic object after posture restoration to obtain the image to be protected.

[0085] An image processing method provided by an embodiment of the present application obtains an image to be protected, extracts an object to be protected from the image to be protected, aligns the object to be protected, obtains the object to be protected after posture correction, inputs the object to be protected after posture correction into a trained generative adversarial network, obtains a perturbation mask corresponding to the object to be protected after posture correction, wherein the perturbation mask includes modification information of the object to be protected, adds a perturbation mask to the object to be protected, obtains a target adversarial object, restores the posture of the target adversarial object, obtains the target adversarial object after posture restoration, replaces the object to be protected in the image to be protected with the target adversarial object after posture restoration, and obtains a protected image. Compared to Figure 1 The image processing method shown in this embodiment also performs posture correction on the object to be protected by alignment, and subsequently restores the posture of the target antagonistic object to ensure smooth generation and addition of the disturbance mask, so as to improve the image processing effect.

[0086] See also Figure 3 , Figure 3 The overall flow chart of the image processing method provided by the embodiment of the present application is shown. Figure 3 As shown, when the object to be protected is a human face, an image can be read in and face detection can be performed on the read image to determine whether there is a face in the read image. Of course, if there are multiple faces that need to be disturbed, the process can be repeated multiple times. If it is determined that there is a face in the read image, the face can be aligned, and adversarial noise can be added to the aligned face photo. After that, the face posture is restored and the face is copied and pasted into the original image, thereby achieving face protection.

[0087] See also Figure 4 , Figure 4 The flowchart of the image processing method provided by an embodiment of the present application is shown in FIG. Figure 4 The process shown is described in detail, and the image processing method may specifically include the following steps:

[0088] Step S310: Iteratively train the generator, discriminator and feature extractor of the generative adversarial network to be trained based on the training sample set, wherein the training samples of the generator are the training objects, the training samples of the discriminator are the training objects and the training adversarial objects, the training samples of the feature extractor are the training objects and the training adversarial objects, and the training adversarial objects are obtained by adding a perturbation mask corresponding to the training object generated by the generator to the training object.

[0089] See also Figure 5 , Figure 5 The schematic diagram of the model structure of the generative adversarial network provided in the embodiment of the present application is shown. Figure 5As shown, the generative adversarial network includes a generator G, a discriminator D and a feature extractor F.

[0090] Optionally, the generator G adopts an Encoder-Decoder structure and adopts the design of an FCN fully convolutional network. The encoding part is constructed using a convolutional layer with reduced dimensionality, and the decoding part adopts deconvolution / Upsampling based on padding+convolution. The encoding part extracts the features of the object to be protected X (such as extracting facial features), including identity features and image appearance color and texture features. The decoding part is used to generate perturbations. Between the encoder and the decoder, a partial shortconnect structure is used to connect the feature maps to realize information transmission. A reasonable short-connect structure can significantly improve the image quality of the generated adversarial map. It can be understood that the input of the generator G can be the object to be protected X, and the output of the generator G can be the perturbation mask G(X) corresponding to the object to be protected.

[0091] Optionally, the discriminator D is used to distinguish between the object to be protected X and the generated adversarial object X+G(X), thereby forcing the adversarial object X+G(X) generated by the generator G to be more realistic and natural in image quality. The structure of the discriminator D can be designed using the design of DCGAN (Deep Convolutional GAN), or the idea of ​​WGAN (Wasserstein GAN) or WGAN-GP, where the WGAN series discriminator can make the loss curve of the learning process more stable.

[0092] Optionally, the feature extractor F is used to extract the feature vector of the object to be protected (such as a face). Its function is to extract the feature vector of the object to be protected (ie: X) before the disturbance is added and the adversarial object (ie: X+G(X)) after the disturbance is added, so as to force the object to be protected and the adversarial object to be offset as much as possible in the feature space defined by the feature extractor F before and after the disturbance. In implementation, when the object to be protected is a face, a feature extractor for the face can be constructed by adopting a mainstream face recognition model, and only the classification layer needs to be removed. This is essentially a black box migration attack, and the feature extractor F is a substitute model, so the choice of feature extractor F has a great influence on the effect of the final generator G. The construction methods of feature extractor F can include the following:

[0093] First, it is extracted from a single face recognition model. The closer the structure of the selected stand-in model is to the structure of the attacked black box model, the better the attack effect. The closer the data set used for training the stand-in model is to the black box model, the better the attack effect. The more fully the stand-in model is trained and the more robust the model is, the better the attack effect.

[0094] Second, use a group of models with different network structures, different data sets and training methods to form an integrated model. The idea is: when the generalization ability of a model is limited and the detailed information of the black box model to be attacked is uncertain, then use the idea of ​​ensemble learning to improve the expressive power of the substitute model by using a group of models for integration. However, in actual practice, the number and structure of sub-models in the integrated model require experiments to find a better combination, and the number of sub-models is not necessarily the more the better.

[0095] Third, use adversarial training or defensive distillation to obtain a more robust model, or directly distill and learn the knowledge of multiple models into a substitute model.

[0096] Moreover, experiments have shown that different feature extractors F have different effects on the quality of the generated adversarial objects. Under the same training method and hyper-parameter conditions, the quality and pattern of the generated adversarial objects by the feature extractor based on MobileNet-arcface and the feature extractor based on Resnet-Facenet are significantly different. Therefore, this embodiment can perform targeted parameter adjustment.

[0097] Therefore, for the above-mentioned generative adversarial network, this embodiment iteratively trains the generator G, discriminator D and feature extractor F of the generative adversarial network to be trained based on the training sample set. It can be understood that the training samples of the generator G are training objects (such as original faces used for training), and the training samples of the discriminator are training objects and training adversarial objects, wherein the training adversarial objects are obtained by adding a perturbation mask corresponding to the training object generated by the generator G to the training object (such as adversarial faces used for training (adding the perturbation mask generated by the generator G to the original faces used for training)), and the training samples of the feature extractor F are training objects and training adversarial objects.

[0098] Optionally, a perturbation limiting loss L_pertu is set in the generative adversarial network, and the perturbation limiting loss L_pertu is used to limit the norm of the perturbation mask G(X) generated by the generator G so that it is neither too large nor completely equal to 0, as shown in the following formula: L_pertubation = E x [max(||G(x)|| 2 -ρ, 0)], after being constrained, the perturbation mask G(X) generated by the generator G presents a Laplace distribution with a mean close to 0, indicating that in the perturbation mask G(X), the perturbations of most positions are very close to 0, and the perturbations of a few positions (key positions) have a large value > 0. This shows that the generator G actually only perturbs and modifies a very small number of key areas in the protected object.

[0099] Corresponding to this embodiment, when training the generative adversarial network to be trained, the training object can be input into the generator to obtain the perturbation mask corresponding to the training object output by the generator, the norm size of the perturbation mask corresponding to the training object is constrained based on the perturbation limit loss, and the parameters of the generator are updated through back propagation.

[0100] Optionally, an adversarial loss L_adv is set in the generative adversarial network to constrain the authenticity and naturalness of the adversarial object after the perturbation is added by the discriminator D. Among them, the scheme of using the discriminator D of DCGAN and WGAN can achieve better results, wherein WGAN is implemented using a weight-clip scheme.

[0101] Corresponding to this embodiment, when training the generative adversarial network to be trained, the training object and the training adversarial object can be input into the discriminator to obtain the authenticity discrimination result output by the discriminator, the authenticity discrimination result is constrained based on the adversarial loss, and the parameters of the generator and the discriminator are updated through back propagation.

[0102] Optionally, an identity loss L_ident is set in the generative adversarial network, and the cosine similarity between the object to be protected X and the perturbation mask G(X) is calculated. The calculation method is as follows: L_ident = max(1-distance-ε,0), where distance is the cosine distance of the embedding of the object to be protected X and the perturbation mask G(X), 1-distance is the similarity between the object to be protected X and the perturbation mask G(X), and ε is a constraint on the lower limit of the similarity, because too small similarity may make the rest difficult to optimize and the image quality out of control. Generally, ε is about 0.15 to 0.2, which can have a better effect.

[0103] Corresponding to this embodiment, when training the generative adversarial network to be trained, the training object can be input into the feature extractor, and the feature extractor performs feature extraction on the training object to obtain a first feature vector, the training adversarial object can be input into the feature extractor, and the feature extractor performs feature extraction on the training object to obtain a second feature vector, the cosine similarity between the first feature vector and the second feature vector is calculated, the lower limit of the cosine similarity is constrained based on the identity loss, and the parameters of the generator and the discriminator are updated through back propagation.

[0104] Step S320: If it is determined that the generative adversarial network to be trained meets the training end condition, the trained generative adversarial network is obtained.

[0105] In some embodiments, a training end condition may be pre-set and stored, and the training end condition is used as a basis for judging whether the training of the generative adversarial network to be trained has ended. Optionally, the training end condition may include that the training duration reaches a duration threshold, the training reaches a number threshold at this time, the training parameters converge, etc., which are not limited here. Therefore, in this embodiment, in the process of iteratively training the generative adversarial network to be trained, it is possible to detect whether the generative adversarial network to be trained meets the training end condition, wherein, if it is detected that the generative adversarial network to be trained meets the training end condition, the training can be stopped, and the trained generative adversarial network is obtained; if it is detected that the generative adversarial network to be trained does not meet the training end condition, the training can be continued until it is detected that the generative adversarial network to be trained meets the training end condition, and the training is stopped, and the trained generative adversarial network is obtained.

[0106] Step S330: Acquire the image to be protected.

[0107] Step S340: extracting the object to be protected from the image to be protected.

[0108] Step S350: inputting the object to be protected into the trained generative adversarial network to obtain a perturbation mask corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected.

[0109] Step S360: adding the disturbance mask to the object to be protected to obtain a target adversarial object.

[0110] Step S370: Replace the object to be protected in the image to be protected with the target confrontation object to obtain a protection image.

[0111] For the detailed description of step S330 to step S370 , please refer to step S110 to step S150 , which will not be repeated here.

[0112] An image processing method provided by an embodiment of the present application iteratively trains a generator, a discriminator, and a feature extractor of a generative adversarial network to be trained based on a training sample set, wherein the training samples of the generator are training objects, the training samples of the discriminator are training objects and training adversarial objects, and the training samples of the feature extractor are training objects and training adversarial objects. The training adversarial objects are obtained by adding a perturbation mask corresponding to the training objects generated by the generator to the training objects. If it is determined that the generative adversarial network to be trained meets the training end condition, the trained generative adversarial network is obtained, the image to be protected is obtained, the object to be protected is extracted from the image to be protected, the object to be protected is input into the trained generative adversarial network, and a perturbation mask corresponding to the object to be protected is obtained, wherein the perturbation mask includes modification information of the object to be protected, a perturbation mask is added to the object to be protected, a target adversarial object is obtained, and the object to be protected in the image to be protected is replaced with the target adversarial object to obtain a protected image. Compared to Figure 1 The image processing method shown in the figure, this embodiment also trains a generative adversarial network composed of a generator, a discriminator and a feature extractor to improve the accuracy of the generated perturbation mask.

[0113] Based on the above embodiments, it can be known that this solution can use an algorithm to mask the protected object (such as a face) in a digital photo without affecting the appearance. Mask protection means that after adding a disturbance invisible to the naked eye to the protected object, the identity of the protected object in the photo cannot be recognized by AI (face recognition algorithm). This solution can successfully deceive the face recognition algorithm to identify "Zhang San" as "Li Si" with a success rate of more than 60%, and the disturbance parameters can be adjusted. The success rate can be increased to nearly 100% under the premise of slightly sacrificing the aesthetics of the picture. In addition, the algorithm can be implemented and run on electronic devices and has certain advantages in memory usage and power consumption. After measurement, the algorithm running memory peak is no more than 238M, and the speed is 508 milliseconds.

[0114] See also Figure 6 , Figure 6 The module block diagram of the image processing device provided by an embodiment of the present application is shown below. Figure 6 The block diagram shown in the figure is used to illustrate that the image processing device 200 includes: a to-be-protected image acquisition module 210, a to-be-protected object extraction module 220, a disturbance mask acquisition module 230, a target adversarial object acquisition module 240 and a protected image acquisition module 250, wherein:

[0115] The to-be-protected image acquisition module 210 is used to acquire the to-be-protected image.

[0116] The object to be protected extraction module 220 is used to extract the object to be protected from the image to be protected.

[0117] Furthermore, the object to be protected includes a face, and the object to be protected extraction module 220 includes: a face detection submodule and a object to be protected extraction submodule, wherein:

[0118] The face detection submodule is used to perform face detection on the image to be protected.

[0119] The to-be-protected object extraction submodule is used to extract the human face from the to-be-protected image if it is detected that the to-be-protected image includes the human face.

[0120] The perturbation mask acquisition module 230 is used to input the object to be protected into the trained generative adversarial network to obtain a perturbation mask corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected.

[0121] Further, the perturbation mask acquisition module 230 includes: an alignment submodule and a first perturbation mask acquisition submodule, wherein:

[0122] The alignment submodule is used to align the object to be protected to obtain the object to be protected after posture correction.

[0123] The first perturbation mask acquisition submodule is used to input the posture-corrected object to be protected into the trained generative adversarial network to obtain a perturbation mask corresponding to the posture-corrected object to be protected.

[0124] Furthermore, the disturbance mask acquisition module 230 includes: a normalization processing submodule and a second disturbance mask acquisition submodule, wherein:

[0125] The normalization processing submodule is used to perform normalization processing on the object to be protected to obtain the normalized object to be protected.

[0126] The second perturbation mask obtaining submodule is used to input the normalized object to be protected into the trained generative adversarial network to obtain a perturbation mask corresponding to the normalized object to be protected.

[0127] The target adversarial object obtaining module 240 is used to add the disturbance mask to the object to be protected to obtain the target adversarial object.

[0128] The protection image acquisition module 250 is used to replace the to-be-protected object in the to-be-protected image with the target confrontation object to obtain a protection image.

[0129] Furthermore, the protection image acquisition module 250 includes: a posture restoration submodule and a protection image acquisition submodule, wherein:

[0130] The posture restoration submodule is used to restore the posture of the target confrontation object to obtain the target confrontation object after posture restoration.

[0131] The protection image acquisition submodule is used to replace the object to be protected in the image to be protected with the target confrontation object after the posture restoration, so as to obtain the protection image.

[0132] Furthermore, the image processing device 200 further includes: a generative adversarial network training module and a generative adversarial network acquisition module, wherein:

[0133] A generative adversarial network training module is used to iteratively train the generator, discriminator and feature extractor of the generative adversarial network to be trained based on a training sample set, wherein the training sample of the generator is the training object, the training sample of the discriminator is the training object and the training adversarial object, the training sample of the feature extractor is the training object and the training adversarial object, and the training adversarial object is obtained by adding a perturbation mask corresponding to the training object generated by the generator to the training object.

[0134] Furthermore, the generative adversarial network training module includes: a first feature vector acquisition submodule, a second feature vector acquisition submodule and a first parameter update submodule, wherein:

[0135] The first feature vector obtaining submodule is used to input the training object into the feature extractor, and extract features of the training object through the feature extractor to obtain a first feature vector.

[0136] The second feature vector obtaining submodule is used to input the training adversarial object into the feature extractor, and extract features from the training object through the feature extractor to obtain a second feature vector.

[0137] The first parameter updating submodule is used to calculate the cosine similarity between the first feature vector and the second feature vector, constrain the lower limit of the cosine similarity based on the identity loss, and update the parameters of the generator and the discriminator through back propagation.

[0138] Furthermore, the generative adversarial network training module includes: a third perturbation mask acquisition submodule and a third parameter updating submodule, wherein:

[0139] The third perturbation mask acquisition submodule is used to input the training object into the generator to obtain the perturbation mask corresponding to the training object output by the generator.

[0140] The second parameter updating submodule is used to constrain the norm size of the perturbation mask corresponding to the training object based on the perturbation limit loss, and update the parameters of the generator through back propagation.

[0141] Furthermore, the generative adversarial network training module includes: an authenticity discrimination result acquisition submodule and a third parameter updating submodule, wherein:

[0142] The authenticity discrimination result obtaining submodule is used to input the training object and the training adversarial object into the discriminator to obtain the authenticity discrimination result output by the discriminator.

[0143] The third parameter updating submodule is used to constrain the authenticity discrimination result based on the adversarial loss and update the parameters of the generator and the discriminator through back propagation.

[0144] The generative adversarial network acquisition module is used to obtain the trained generative adversarial network if it is determined that the generative adversarial network to be trained meets the training end condition.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0146] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0147] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0148] See also Figure 7 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0149] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0150] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0151] See also Figure 8 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0152] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0153] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0154] In summary, the image processing method, electronic device, and computer program product provided by the embodiments of the present application obtain an image to be protected, extract an object to be protected from the image to be protected, input the object to be protected into a trained generative adversarial network, and obtain a perturbation mask corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected, adds a perturbation mask to the object to be protected, obtains a target adversarial object, replaces the object to be protected in the image to be protected with the target adversarial object, and obtains a protected image, thereby improving security while ensuring aesthetics by masking the object to be protected in the image without affecting aesthetics.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire the image to be protected; extracting the object to be protected from the image to be protected; Inputting the object to be protected into a trained generative adversarial network to obtain a perturbation mask corresponding to the object to be protected, wherein the perturbation mask includes modification information of the object to be protected; Adding the disturbance mask to the object to be protected to obtain a target adversarial object; The object to be protected in the image to be protected is replaced by the target confrontation object to obtain a protection image.

2. The method according to claim 1, characterized in that The step of inputting the object to be protected into a trained generative adversarial network to obtain a perturbation mask corresponding to the object to be protected includes: Aligning the object to be protected to obtain the object to be protected after posture correction; The object to be protected after posture correction is input into the trained generative adversarial network to obtain a disturbance mask corresponding to the object to be protected after posture correction.

3. The method according to claim 2, characterized in that The step of replacing the to-be-protected object in the to-be-protected image with the target confrontation object to obtain a protected image includes: Restoring the posture of the target confrontation object to obtain the target confrontation object after posture restoration; The protected object in the protected image is replaced by the target confrontation object after the posture restoration to obtain the protected image.

4. The method according to claim 1, characterized in that The step of inputting the object to be protected into a trained generative adversarial network to obtain a perturbation mask corresponding to the object to be protected includes: Normalizing the object to be protected to obtain a normalized object to be protected; The normalized object to be protected is input into a trained generative adversarial network to obtain a perturbation mask corresponding to the normalized object to be protected.

5. The method according to claim 1, characterized in that The object to be protected includes a human face, and extracting the object to be protected from the image to be protected includes: Performing face detection on the image to be protected; If it is detected that the image to be protected includes a human face, the human face is extracted from the image to be protected.

6. The method according to any one of claims 1 to 5, characterized in that: Before acquiring the image to be protected, the method further includes: Iteratively training a generator, a discriminator, and a feature extractor of a generative adversarial network to be trained based on a training sample set, wherein the training sample of the generator is a training object, the training sample of the discriminator is the training object and a training adversarial object, the training sample of the feature extractor is the training object and the training adversarial object, and the training adversarial object is obtained by adding a perturbation mask corresponding to the training object generated by the generator to the training object; If it is determined that the generative adversarial network to be trained meets the training end condition, a trained generative adversarial network is obtained.

7. The method according to claim 6, characterized in that The iterative training of the generator, discriminator and feature extractor of the generative adversarial network to be trained based on the training sample set includes: Inputting the training object into the feature extractor, and extracting features of the training object through the feature extractor to obtain a first feature vector; Inputting the training adversarial object into the feature extractor, and extracting features from the training object by the feature extractor to obtain a second feature vector; The cosine similarity between the first feature vector and the second feature vector is calculated, a lower limit of the cosine similarity is constrained based on the identity loss, and parameters of the generator and the discriminator are updated through back propagation.

8. The method according to claim 6, characterized in that The iterative training of the generator, discriminator and feature extractor of the generative adversarial network to be trained based on the training sample set includes: Inputting the training object into the generator, and obtaining a perturbation mask corresponding to the training object output by the generator; The norm size of the perturbation mask corresponding to the training object is constrained based on the perturbation limit loss, and the parameters of the generator are updated through back propagation.

9. The method according to claim 6, characterized in that The iterative training of the generator, discriminator and feature extractor of the generative adversarial network to be trained based on the training sample set includes: Inputting the training object and the training adversarial object into the discriminator to obtain the authenticity discrimination result output by the discriminator; The authenticity discrimination result is constrained based on the adversarial loss, and the parameters of the generator and the discriminator are updated through back propagation.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 9.

11. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images