Image processing method and related equipment

Abstract

The invention discloses an image processing method and related equipment, and the method comprises the steps: determining the attribute feature of a first face image, and determining the identity feature of a second face image; a face area in the first face image is masked to obtain a first image mask, and a face area in the second face image is masked to obtain a second image mask; determining a third image mask based on an overlapping area between the first image mask and the second image mask; and generating a third face image based on the attribute feature, the identity feature and the third image mask.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, device, electronic device, storage medium and computer program product. Background Art

[0002] Face-swapping technology is used to replace one face with another in an image or video.

[0003] When swapping faces, it is usually necessary to maintain the identity ID features of the source face. However, when extracting the identity ID features, not only the identity ID features of the source face are extracted, but also the source face's attribute features unrelated to the face, such as lighting, expression, and posture. As a result, the generated third face image will also carry these attribute features unrelated to the source face, thus affecting the face swapping effect. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method and related equipment.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Determining attribute features of the first facial image and determining identity features of the second facial image;

[0007] Masking the face area in the first face image to obtain a first image mask, and masking the face area in the second face image to obtain a second image mask;

[0008] determining a third image mask based on an overlapping area between the first image mask and the second image mask;

[0009] A third facial image is generated based on the attribute features, the identity features, and the third image mask.

[0010] In a second aspect, the present application provides an image processing method, comprising:

[0011] Inputting the first sample face image and the second sample face image into the initial image processing model for first processing, the first processing comprising: generating a third sample face image based on the first sample face image and the second sample face image; the first sample face image comprising a sample attribute feature, and the second sample face image comprising a sample identity feature;

[0012] Inputting the third sample facial image into the initial image processing model for second processing, the second processing comprising: extracting reference attribute features of the third sample facial image and extracting reference identity features of the third sample facial image;

[0013] Based on the reference attribute feature, the sample attribute feature, the sample identity feature and the reference identity feature, a training loss is determined, and based on the training loss, the parameters of the initial image processing model are adjusted to obtain an image processing model.

[0014] In a third aspect, the present application provides an image processing device, comprising:

[0015] a first determining module, configured to determine attribute features of the first facial image and identity features of the second facial image;

[0016] a masking module, configured to mask the face region in the first face image to obtain a first image mask, and mask the face region in the second face image to obtain a second image mask;

[0017] a second determining module, configured to determine a third image mask based on an overlapping area between the first image mask and the second image mask;

[0018] A generating module is used to generate a third face image based on the attribute feature, the identity feature, and the third image mask.

[0019] In a fourth aspect, the present application further provides an image processing device, comprising:

[0020] a processing module, configured to input the first sample facial image and the second sample facial image into an initial image processing model for first processing, wherein the first processing comprises: generating a third sample facial image based on the first sample facial image and the second sample facial image; the first sample facial image comprises a sample attribute feature, and the second sample facial image comprises a sample identity feature;

[0021] The processing module is further configured to input the third sample facial image into the initial image processing model for second processing, wherein the second processing includes: extracting reference attribute features of the third sample facial image and extracting reference identity features of the third sample facial image;

[0022] An adjustment module is used to determine a training loss based on the reference attribute feature, the sample attribute feature, the sample identity feature and the reference identity feature, and to adjust the parameters of the initial image processing model based on the training loss to obtain an image processing model.

[0023] In a fifth aspect, the present application provides an electronic device, including:

[0024] one or more processors;

[0025] Memory;

[0026] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs are configured to perform the steps in the method described in the first aspect or the second aspect.

[0027] In a sixth aspect, the present application provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the method described in the first aspect or the second aspect above are implemented.

[0028] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps in the method described in the first or second aspect above.

[0029] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: when generating a third facial image based on the second facial image and the first facial image, the attribute features in the first facial image used to characterize the attribute information of the first facial image and the identity features in the second facial image used to characterize the identity information of the second facial image can be obtained. Therefore, the third facial image generated based on the attribute features, identity features and the third image mask will not carry the attribute features of the second facial image, and the generated third facial image also integrates the attribute features of the first facial image, which can make the attribute features of the generated third facial image consistent with those of the first facial image, and the identity features of the third facial image consistent with the identity features of the second facial image, thereby achieving the effect that the generated new facial image does not retain the attribute features in the second facial image that are not related to the face. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0031] Figure 1 A schematic diagram of a specific image processing application scenario is provided for the embodiment of the present application;

[0032] Figure 2 A schematic diagram of an application environment of the image processing method provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of an implementation flow of an image processing method is provided for an embodiment of the present application;

[0034] Figure 4 Schematic diagram for performing face extraction on the second image and the first image to obtain corresponding second face image and first face image;

[0035] Figure 5 A schematic diagram of the network structure of an image processing model provided in an embodiment of the present application;

[0036] Figure 6 Schematic diagram of the structure of AEI-Net;

[0037] Figure 7 This is a schematic diagram of the structure of the AAD ResBlk module;

[0038] Figure 8 This is a schematic diagram of the specific structure of the AAD module included in faceshifter;

[0039] Figure 9 A schematic diagram of a process for obtaining a third image mask according to an embodiment of the present application;

[0040] Figure 10 A schematic diagram of the implementation flow of another image processing method provided in an embodiment of the present application;

[0041] Figure 11 A schematic diagram of the specific structure of an image processing device is provided for an embodiment of the present application;

[0042] Figure 12 A schematic diagram of the specific structure of another image processing device provided in an embodiment of the present application;

[0043] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0045] The following is an explanation of the nouns involved in this application:

[0046] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0047] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0048] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, and smart transportation. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0049] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0050] Generative Adversarial Network (GAN): A method of unsupervised learning that learns by pitting two neural networks against each other. It consists of a generative network and a discriminative network. The generative network randomly samples from the latent space as input, and its output is required to closely mimic the real samples in the training set. The discriminative network takes real samples (i.e., true values) or the output of the generative network as input, with the goal of distinguishing the output of the generative network from real samples as much as possible. The generative network, in turn, is required to deceive the discriminative network as much as possible. The two networks compete with each other, continuously updating their parameters, ultimately producing images that are indistinguishable from the real ones.

[0051] Face swapping: swapping the input face source image to the template image, and making the output third face image keep the expression, angle, background and other information of the template image. Figure 1 As shown, the input face source image is face A, the face in the template image is another face B, and the output result is a photo in which face B in the template image is replaced with face A.

[0052] In order to solve the problem that the third facial image obtained according to the relevant technology carries attribute features unrelated to the source face, thereby affecting the face-changing effect, the inventor provides an image processing method. When generating a third facial image based on the second facial image and the first facial image, the inventor can obtain the attribute features in the first facial image used to characterize the attribute information of the first facial image, and the identity features in the second facial image used to characterize the identity information of the second facial image. Therefore, the third facial image generated based on the attribute features, identity features and the third image mask will not carry the attribute features of the second facial image, and the generated third facial image also integrates the attribute features of the first facial image, which can make the attribute features of the generated third facial image consistent with those of the first facial image, and the identity features of the third facial image consistent with the identity features of the second facial image, thereby achieving the effect of the generated new facial image not retaining the attribute features in the second facial image that are unrelated to the face.

[0053] The image processing method provided by the embodiment of the present application can be applied to any scenario that requires face generation, and the method can be applied to products in these scenarios, such as film and television portrait production, game character design, virtual images, privacy protection, etc. In film and television production, when an actor is unable to complete professional actions, professionals can complete them first, and then face-changing technology can be used to automatically replace the human face with the actor. When an actor needs to be replaced, a new face can be replaced by face-changing technology to avoid reshooting, which can save a lot of costs. In virtual image design, such as in a live broadcast scene, users can use virtual characters to change faces, which increases the fun of the live broadcast and protects personal privacy. The results of video face-changing can also provide anti-attack training materials for services such as face authentication.

[0054] Taking the video editing application used in film and television portrait production as an example, the video editing application is used to replace the first face in the first facial image in a specified video clip with the second face in the second facial image, and then display the specified video clip after the face-swap processing. This video editing application can be applied to a variety of scenarios. For example, in the film and television portrait production scenario, the second facial image can be the image of an actor who did not actually participate in the filming of the movie, and the first facial image can be the image of an actor who actually participated in the filming of the movie. After inputting the second facial image and the first facial image, the video editing application can replace the first face in the first facial image in the specified video clip with the second face in the second facial image, completing the face-swap operation.

[0055] For example, Figure 1 As shown, a specific image processing application scenario is provided. User 101 sends a face swap request 102. Terminal device 103 can collect a second face image 1041 of user 101 and a first face image 1042 of the person whose face is to be swapped. Identity features are extracted from the second face image 1041 and attribute features are extracted from the first face image 1042 through video editing application 105, and a mask image of the second face within the face area of ​​the first face is obtained. Finally, video editing application 105 fuses these extracted features and mask images to generate a third face image 106, and displays corresponding face swap content.

[0056] In order to better understand the image processing method and related equipment provided by the embodiment of the present application, the application environment applicable to the embodiment of the present application is described below.

[0057] See also Figure 2 , Figure 2 The following is a schematic diagram showing an application environment of the image processing method provided by an embodiment of the present application. For example, the image processing method, image processing device, electronic device, storage medium and computer program product provided by the embodiment of the present application can be applied to electronic devices, wherein the electronic device can be such as Figure 2 The server 210 shown in FIG. 2 can be connected to the terminal device 220 via a network. The network is used to provide a medium for a communication link between the server 210 and the terminal device 220. The network can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in the embodiments of the present application. Alternatively, in other embodiments, the electronic device can also be a smartphone, a laptop computer, etc.

[0058] It should be understood that Figure 2 The server 210, network and terminal device 220 are merely illustrative. Depending on the implementation requirements, there may be any number of servers, networks and terminal devices. For example, the server 210 may be a physical server or a server cluster consisting of multiple servers. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. It is understandable that the embodiments of the present application may also allow multiple terminal devices 220 to access the server 210 at the same time.

[0059] In some embodiments, the terminal device 220 may input the second facial image and the first facial image to the server 210 for image processing. In other embodiments, the terminal device 220 may capture the second face to obtain the second facial image, and then extract the second face from the second facial image and obtain the first face from the first facial image on the terminal device 220 or the server 210. Then, the first face in the first facial image may be replaced with the second face on the server 210 to generate a third facial image.

[0060] The above application environment is only an example for ease of understanding. It should be understood that the embodiments of the present application are not limited to the above application environment.

[0061] The image processing method and related equipment provided in the embodiments of the present application will be described in detail below through specific examples.

[0062] See also Figure 3 , which shows a flow chart of an image processing method provided by an embodiment of the present application. Figure 3 The image processing method may include the following steps:

[0063] Step 310: Determine attribute features of the first facial image and determine identity features of the second facial image.

[0064] In some embodiments, the second image and the first image can be respectively subjected to face extraction to obtain the second face image and the first face image respectively; for example, Retinaface (a face detector) can be used to extract the face of the second image and the first image respectively. Figure 4 The second image and the first image shown in FIG are subjected to face extraction, thereby obtaining the following Figure 4 The second face image and the first face image are shown.

[0065] In some embodiments, the aforementioned attribute features are features unrelated to the identity of the face in the facial image, that is, other information in the facial image besides the identity features. For example, attribute features may include the background of the face, the lighting in the facial image, the facial expression in the facial image, and the facial posture in the facial image. The aforementioned identity features are features used to uniquely identify the identity of the face in the facial image. For example, identity features may include the distance between the eyes, the curvature of the mouth, the shape of the eyes, etc.

[0066] It should be understood that the lighting, expression and posture of the first face in the first face image are one of the important factors of the face image and also have an important impact on the face-changing effect. The appearance of the face in the image is usually determined by the texture and lighting of the face itself. If the lighting attribute characteristics of the first face in the first face image cannot be extracted separately during the face-changing process, it will be difficult to transfer the real texture of the second face under the lighting attributes of the first face image to the first face image, and then there may be problems such as incoordination between the face and the face background after the face-changing, resulting in poor face-changing effect. At the same time, if the facial posture and facial expression cannot be effectively decoupled in the face-changing, the face after the face-changing will not match the image context in which the face is located (especially in the video face-changing scene), thereby affecting the face-changing effect. Therefore, in the method of the embodiment of the present invention, the attribute characteristics such as lighting attributes, expression attributes and posture attributes are decoupled, that is, the lighting parameters and posture parameters corresponding to the first face image are obtained.

[0067] In some embodiments, obtaining attribute features of the first face in the first facial image may include the following sub-steps 31011 and 31012:

[0068] Sub-step 31011: Input the first face image into the attribute feature extraction network in the image processing model to extract the attribute features of the first face image; wherein the attribute features include lighting attribute features, expression attribute features and posture attribute features.

[0069] In some exemplary embodiments, the network structure diagram of the image processing model is as follows: Figure 4As shown, the system includes an attribute feature extraction network, a mask module, an identity feature extraction network, and a face fusion network. The attribute feature extraction network is used to extract the attribute features of a first face in a first face image; the mask module is used to obtain a mask image (Mask) of a second face within the facial region of the first face; the identity feature extraction network is used to extract the identity ID (i.e., identity features) of the second face from the second face image; and the face fusion network is used to fuse the attribute features extracted by the attribute feature extraction network, the mask image, and the identity ID extracted by the identity feature extraction network to obtain a fused image as a third face image.

[0070] In an optional embodiment, an implementation principle similar to Unet (Unity Networking) can be used to extract attribute features of the first face from the first facial image. Specifically, in the embodiment of the present application, an attribute feature extraction network can be used to downsample and encode the first facial image, and then decode it to obtain the attribute features of the first face. Compared with Unet, the decoding scheme provided in the embodiment of the present application is different from that of Unet.

[0071] A Unet, consists of an encoder (downsampler) and decoder (upsampler), similarly, as Figure 5 As shown, the attribute feature extraction network used in the embodiment of the present application may also include an encoder and a decoder. Among them, the encoder of the attribute feature extraction network is composed of a series of convolutional networks. The convolutional network is mainly composed of convolutional layers and pooling layers.

[0072] In the convolutional layer, the square "boxes" filled with data are called convolution kernels. In a CNN network, after passing through the input layer, the original image is converted into a grayscale or RGB data matrix. The convolution kernel is aligned with the data matrix, and the corresponding data in each are multiplied and then added. The convolution kernel is then moved across the image at a certain distance (step size). The resulting new matrix reflects the local features of the original image and is called a feature map. The feature map serves as both the output of this layer and the input to the next layer (the pooling layer).

[0073] The pooling layer is used to select the main features of the image. Pooling methods include max pooling (taking the maximum value of a region in the feature map to preserve the image's texture information), min pooling (taking the minimum value of a region in the feature map), and average pooling (taking the average value of a region in the feature map to preserve more background information). The main features of the image selected by the pooling layer are also output as feature maps.

[0074] The decoder of the attribute feature extraction network upsamples the feature map input to the decoder and then performs convolution on the upsampled feature map. This refines the geometric shapes of objects in the feature map and compensates for the loss of detail caused by the encoder's pooling layer, which reduces the size of objects. Similar to the encoder, the decoder also contains convolutional layers, but these layers are used during the decoding phase to process the upsampled image features extracted from the encoder's feature map. The upsampled 2x2 region contains only one 1x1 feature point from the previous layer, while the rest of the region is empty. Therefore, these empty values ​​need to be padded with appropriate feature values ​​to make the region complete and smooth. This "padding" is performed by the decoder's convolutional layers. The convolution operation performed by the decoder's convolutional layers is also called "deconvolution."

[0075] In an optional embodiment, the left half of the attribute feature extraction network (i.e. Figure 5 The convolution of the encoder part of the attribute feature extraction network in the right half (i.e. Figure 5 The convolution in the decoder portion of the attribute feature extraction network (Figure 2) represents a 3x3 convolution with a stride of 1 / 2. A stride of 2 means each movement is a step of 2, which means the resolution is reduced, indicating downsampling. A stride of 1 / 2 simply represents a convolutional layer for feature extraction, further extracting the attribute or identity features extracted by the previous layer. The convolution kernel size is 64, and the convolution is followed by a Prelu layer, followed by a deconvolution layer for upsampling. The convolution layers on the left represent downsampling layers, while the convolution layers on the right represent upsampling layers.

[0076] Sub-step 31012: Obtain the attribute features output by each upsampling layer of the attribute feature extraction network of the first face image.

[0077] It should be noted that the attribute features of the first face in the above-mentioned first face image may include: the attribute features output by each upsampling layer of the attribute feature extraction network of the first face image. Figure 5For example, the upsampling layers of the attribute feature extraction network include upsampling layer 1, upsampling layer 2, upsampling layer 3, upsampling layer 4, and upsampling layer 5. Then, the attribute features of the first face image output by each upsampling layer of the attribute feature extraction network include attribute feature xt1 output by upsampling layer 1, attribute feature xt2 output by upsampling layer 2, attribute feature xt3 output by upsampling layer 3, attribute feature xt4 ​​output by upsampling layer 4, and attribute feature xt5 output by upsampling layer 5. Compared with attribute features that are not upsampled or only using the attribute features output by the final upsampling layer, attribute features in different spatial domains can be obtained after multiple upsamplings. The attribute features that each spatial domain focuses on are different. These attribute features are higher-dimensional features and the attribute features they contain are relatively more comprehensive.

[0078] In some embodiments, the above-mentioned identity features (also referred to as identity ID in the embodiments of the present application) refer to features that can characterize the facial characteristics of a face. For example, including but not limited to: one or more of the following features: the shape of the eyes, the distance between the eyes and the mouth, the curvature of the mouth, etc. For the second face, the identity features of the second face refer to features that can characterize the characteristics of the second face. In order to more comprehensively express the characteristics of the face, the identity features can be spliced ​​features obtained by splicing these features. Specifically, obtaining the identity features of the second face in the second facial image can include the following sub-steps 31021 to 31022:

[0079] Sub-step 31021: Input the second facial image into the identity feature extraction network in the image processing model to extract the identity features of the second facial image through the identity feature extraction network.

[0080] In an optional embodiment, the identity feature extraction network is similar to the attribute feature extraction network, and the left half of the identity feature extraction network (i.e. Figure 5 The convolution of the encoder part of the identity feature extraction network in the figure represents a 3*3 convolution with a step size of 2, a convolution kernel size of 64, and a Prelu layer after the convolution. Figure 5 The convolution in the decoder portion of the identity feature extraction network (Figure 2) represents a 3x3 convolution with a stride of 1 / 2. A stride of 2 means each movement is a step of 2, which means the resolution is reduced, indicating downsampling. A stride of 1 / 2 simply represents a convolutional layer for feature extraction, further extracting the attributes or identity features extracted by the previous layer. The convolution kernel size is 64, and the convolution is followed by a Prelu layer, followed by a deconvolution layer for upsampling. The convolution layers on the left represent downsampling layers, while the convolution layers on the right represent upsampling layers.

[0081] like Figure 5As shown in the figure, the downsampling layer of the identity feature extraction network includes five convolutional layers, and the upsampling layer of the identity feature extraction network includes five DenseBlocks, which can be labeled DenseBlock1 to DenseBlock5. Among them, DenseBlock contains many interconnected DenseLayer layers. The output feature maps of each layer are of the same size, so that they can be connected on the channel. Specifically, the input of each layer comes from the features of all the layers before it, and the output of each layer is directly connected to the input of all the layers after it. The layers are densely connected.

[0082] Optionally, the identity feature extraction network in the embodiment of the present application may adopt an existing pre-trained identity feature extraction network for extracting identity IDs. That is, in order to save computing resources, the embodiment of the present application does not train the identity feature extraction network.

[0083] Sub-step 31022: Obtain the identity features of the second facial image output by each upsampling layer of the identity feature extraction network.

[0084] It should be noted that the identity features of the first face in the second face image include: the identity features of the second face image output at each upsampling layer of the identity feature extraction network. Figure 5 For example, the upsampling layers of the attribute feature extraction network include DenseBlock1, DenseBlock2, DenseBlock3, DenseBlock4, and DenseBlock5. Then, the identity features output by each upsampling layer of the identity feature extraction network for the second facial image include attribute feature Id1 output by upsampling layer DenseBlock1, attribute feature Id2 output by upsampling layer DenseBlock2, attribute feature Id3 output by upsampling layer DenseBlock3, attribute feature Id4 output by upsampling layer DenseBlock4, and attribute feature Id5 output by upsampling layer DenseBlock5. Compared to un-upsampling attribute features or using only the attribute features output by the final upsampling layer, multiple upsampling steps yield identity features in different spatial domains. Each spatial domain focuses on different aspects of the identity features, and the later identity IDs are more representative of the characteristics of the source image, such as facial features. These identity features are higher-dimensional features and contain relatively more comprehensive identity features.

[0085] By upsampling the identity features at different times, we can obtain identity features in different spatial domains. Compared with the un-upsampled identity features, the upsampled identity features are higher-dimensional features and contain more comprehensive features.

[0086] Therefore, by adopting the above-mentioned specific implementation method of sub-steps 31021 to sub-steps 31022, the upsampled identity ID is used to participate in the generation of the third facial image, which can improve the quality of the third facial image finally generated, so that the generated third facial image can express the facial features in the second facial image more comprehensively and accurately.

[0087] Step 320 : Masking the face region in the first face image to obtain a first image mask, and masking the face region in the second face image to obtain a second image mask.

[0088] Optionally, a target mask image of the first face and a source mask image of the second face can be obtained through SGHM (Semantic Segmentation Guided Human Masking). Specifically, the first face image (Xt) and the second face image (Xs) can be input into SGHM to obtain a binary face mask image of the first face image and a mask image of the second face image.

[0089] SGHM builds on the semantic human segmentation network and introduces a lightweight matting module that requires minimal computational cost. Given a color image I, the matting task can be formulated as follows: I = αF + (1-α)B, α∈[0,1]. Here, F and B are the foreground and background, and α is the alpha matte representing the location of the foreground. For image matting, we should predict the alpha matte from the input color image, which is a difficult and ill-posed task. As mentioned earlier, existing methods rely on additional auxiliary inputs such as trimap or pre-captured background. Automatic methods such as RVM are not robust to semantic errors. Based on this, a framework can be designed to better utilize the semantic priors from segmentation while producing fine details and transparent matting values. A simple approach relies on semantic segmentation masks and uses a novel matting network to generate matting results. This setup is developed and demonstrated in Mask-Guided (MG) matting. The two-step setup treats segmentation and matting as two separate tasks, which has several drawbacks. First, the matting network only uses the predicted segmentation map, ignoring rich semantic features. Second, using a separate matting network will extract features from the image again and introduce additional computation, which will significantly slow down the process at high resolutions.

[0090] Based on the above analysis, a new method for human matting, called semantically guided human matting (SGHM), was proposed. This method uses a segmentation network to guide human matting. Specifically, the encoder is shared between the segmentation and matting tasks. As a result, the matting task can learn accurate semantic understanding by reusing the rich semantic features in the encoder, while focusing on predicting alpha details in the matting decoder.

[0091] SGHM consists of a shared encoder for extracting image features, a segmentation decoder for predicting image segmentation masks, and a matting decoder with a progressive refinement module (PRM) for predicting high-resolution matting results. An Attentive Shortcut Module (ASM) is used to combine features from the encoder and masks from the segmentation decoder for use in the matting decoder.

[0092] Step 330 : Determine a third image mask based on the overlapping area between the first image mask and the second image mask.

[0093] In some exemplary embodiments, determining a third image mask based on an overlapping area between the first image mask and the second image mask includes:

[0094] Determine the overlapping area between the second image mask and the first image mask, and use the overlapping area as the third image mask; or,

[0095] The overlapping area between the second image mask and the first image mask is mapped to the second face image, the face occlusion area of ​​the second face image within the overlapping area is determined, and the face occlusion area in the overlapping area is removed to obtain a third image mask.

[0096] The overlapping area between the second image mask and the first image mask is determined, and the overlapping area is used as the third image mask. Specifically, the mask image of the first face and the mask image of the second face can be overlapped within the facial area of ​​the first face at the same resolution and in the same coordinate system. Figure 9 As shown in (1), it is a schematic diagram of obtaining a third image mask provided in an embodiment of the present application.

[0097] The five-pointed star portion represents the second face image, and the rectangular portion represents the first face image. Considering that the mask image of the face is relatively complex in practice, the diagram in the embodiment of the present application is only an abstract representation. In order to keep the shape of the second face in the second face image as much as possible and make the generated third face image more realistic, the embodiment of the present application can delete the part where the first face is larger than the second face in the first face image (i.e. Figure 9 while preserving the shape of the second face within the range of the first face image.

[0098] Since occlusion is also a major influencing factor in face swapping, considering the occlusion problem of the hair part of the upper and lower parts of the second face in the second facial image, the embodiment of the present application removes the mask image of the second face that exceeds the range of the first facial image. As for the skin of the forehead area of ​​the second face, the skin part of the first face can be used to adapt to the second face, and the image processing model can learn the skin of the forehead area by itself. Specifically, the overlapping area between the second image mask and the first image mask can be mapped to the second facial image, and the facial occlusion area of ​​the second facial image within the overlapping area is determined, and the facial occlusion area in the overlapping area is removed to obtain the third image mask. As shown Figure 9 As shown in (2), it is another schematic diagram of obtaining the third image mask provided by the embodiment of the present application. Among them, the five-pointed star part represents the second face image, the rectangular part represents the first face image, and the shadow part in the five-pointed star is the face occlusion area. Considering that many factors such as hair occlusion in practice make the mask image of the face more complicated, the figure drawn in the embodiment of the present application is only an abstract representation. In order to keep the second face shape in the second face image as much as possible, so that the generated third face image is more realistic, the embodiment of the present application can be Figure 9 (1) Delete the part where the first face is larger than the second face in the first face image (i.e. Figure 9 After deleting the two sides of the five-pointed star, the shadow part of the five-pointed star representing the facial occlusion area in the second face image is removed to obtain the third image mask.

[0099] Using the above method, the face area in the first face image is masked to obtain a first image mask, and the face area in the second face image is masked to obtain a second image mask. Based on the overlapping area between the second image mask and the first image mask, a third image mask that retains the shape of the first face area and the texture distribution of the second face area can be obtained, so that the third face image finally generated based on the attribute features, identity features and the third image mask can retain both the shape of the first face area and the texture features of the second face area, thereby further improving the face-changing effect.

[0100] Step 340 : Generate a third facial image based on the attribute features, the identity features, and the third image mask.

[0101] As an example, the attribute features, identity features, and the third image mask may be fused to generate a third facial image. Specifically, the attribute features, identity features, and the third image mask are fused to generate the third facial image, including:

[0102] Performing fusion processing based on the attribute features, the identity features, and the third image mask to obtain a third face image;

[0103] The fusion processing includes performing a specified matrix operation based on the attribute features, the identity features and the third image mask.

[0104] The specified matrix operations based on the attribute features, identity features and the third image mask include:

[0105] The attribute activation results obtained by convolution processing, vector operation and denormalization processing based on the attribute features, the processing results obtained by mapping the identity features to the sample label space and vector operation, and the third image mask are subjected to the specified matrix operation.

[0106] The specified matrix operation may include: performing a vector dot multiplication operation between a difference between a specified value and the third image mask and an attribute activation result to obtain a first vector dot multiplication operation result; and performing a vector dot multiplication operation based on a processing result obtained by mapping the third image mask and the identity feature to a sample label space and the vector operation to obtain a second vector dot multiplication operation result;

[0107] A third facial image is generated by performing a dot product operation on the first vector and a dot product operation on the second vector.

[0108] The result of the first vector dot product operation and the result of the second vector dot product operation can be seen in detail below. The calculation formula of .

[0109] In some exemplary embodiments, the attribute features, identity features, and mask image may be input into a face fusion network of an image processing model for fusion to generate a third face image. The face fusion network is configured to fuse the mask image, the attribute features extracted by the attribute feature extraction network, and the identity ID generated by the identity feature extraction network to generate a fused image as the third face image.

[0110] In some exemplary embodiments, generating the third face image based on the attribute features, the identity features, and the mask image may include the following sub-steps 331 and 332:

[0111] Sub-step 331: Input the attribute features, identity features, and mask image into each fusion module in the face fusion network of the image processing model.

[0112] As an example, Figure 5 The face fusion network in

[15] includes 5 fusion modules (AAD ResBlk in the figure).

[0113] In some exemplary embodiments, inputting the attribute features, identity features, and mask image into the fusion modules in the face fusion network of the image processing model may include the following sub-steps 3311 and 3312:

[0114] Sub-step 3311: Use the attribute features output by the first upsampling layer of the attribute feature extraction network of the first face image, the identity features output by the last layer of the identity feature extraction network of the second face image, and the mask image as the input of the first fusion module in the face fusion network to obtain the output of the first fusion module.

[0115] Combine Figure 5 Specifically, the attribute feature xt1 output by the first upsampling layer of the attribute feature extraction network for the first facial image can be input into the first fusion module AAD ResBlk from the left. The identity feature Id1 output by the last layer of the identity feature extraction network for the second facial image can be input into the first fusion module AAD ResBlk as two branches (i.e., as two inputs). The mask image can also be input into the first fusion module AAD ResBlk for fusion to obtain the first and third facial images. Because the residual module in the ADD network needs to perform residual calculations on the input and output, an additional input Id1 is used here.

[0116] The structure of the AAD ResBlk module in faceshifter can be referenced. Faceshifter is a two-stage model that can be used to swap any two facial images. It consists of two components: AEI-Net and HEAR-Net. With proper training, AEI-Net itself can achieve good face-swapping results. On this basis, HEAR-Net can be trained to address facial occlusion issues in the first image and further optimize the face-swapping effect.

[0117] AEI-Net is an attribute encoder, which consists of three parts: identity ID encoding module (Identity Encoder), attribute encoding module (Multi-level Attributes Encoder) and AAD generator (Adaptive Attentional Denormalization Generator). The structure of AEI-Net is as follows Figure 6 shown. Figure 6 Middle: Xt represents the second face image, Xs represents the first face image; Represents the attribute features extracted by the first layer of the attribute encoding module, represents the attribute features extracted by the second layer of the attribute encoding module, ..., Figure 6 The meanings of other similar parameters in are similar; id Indicates the identity ID extracted by the identity ID encoding module; Indicates the integrated processing result obtained by the AAD generator by integrating the attribute features input to the AAD generator by the attribute encoding module and the identity ID input to the AAD generator by the identity ID encoding module. In the face-changing scenario, Represents the third face image.

[0118] The identity encoding module is a pre-trained face recognition module, also known as the Archface module. Throughout the network, this module performs only forward propagation and does not perform loss backpropagation training. In face-swapping scenarios, the identity encoding module extracts the identity ID from the second face image and inputs it into the AAD module contained in the AAD generator. The attribute encoding module, also known as a multi-level attribute encoding module, extracts attribute features representing facial posture, expression, scene brightness, and background information from the first face image and inputs them into the AAD module.

[0119] The AAD generator is composed of multiple AAD ResBlk modules, and each AAD ResBlk module is composed of an AAD module, Relu and Conv stack. Figure 7 The structure diagram of the AAD ResBlk module is shown. For each AAD module, it has two inputs: the identity ID extracted from the second face image by the identity ID encoding module (i.e. Figure 6 and Figure 7 The z shown in id ), and the attribute features extracted from the first face image by the attribute encoding module (i.e. The specific structural diagram of the AAD module included in faceshifter can be found in the appendix of the manual. Figure 8 .

[0120] Figure 8 middle: Equivalent to the above That is, the attribute features extracted by the attribute encoding module from the first face image; and Both expressed Attribute features obtained after convolution processing; Represents the result graph after inputting the activation of the AAD layer; Perform batch gradient normalization (Batch Norm) to obtain z id represents the identity ID extracted by the identity ID encoding module; M is the third image mask determined in step 330 based on the overlapping area between the first image mask and the second image mask.

[0121] Since the M in the existing ADD layer k Yes Perform convolution and sigmoid (Conv+sigmoid) operations to generate an attention mask M k , that is, M k The input of is the feature output by the previous network, which only emphasizes the features of the second face, but cannot guarantee the shape of the second face. In order to better maintain the shape of the second face, the embodiment of the present application replaces the M in the original network with k The second face is replaced by a mask image M within the facial area of ​​the first face. Since the mask image M retains the unobstructed facial area of ​​the second face, it can be ensured that the face in the finally generated third facial image better maintains the shape of the second face.

[0122] is the output of the AAD module, and the specific calculation formula is:

[0123] Among them, A k For Denormalize to get the attribute activation result; I k It is calculated in the following way: and All use the fully connected FC layer to z id The processing result obtained after processing. is the result of the first vector dot multiplication operation, 1 is the specified value, M is the third image mask, A k is the attribute activation result obtained by denormalization processing, is the result of the second vector dot multiplication operation, I k Map identity features to sample label space.

[0124] Since the third face image generated by AEI-Net cannot handle the occlusion problem, the HEAR-Net described above is introduced in the related art to solve this problem. Since the embodiment of this application is not closely related to HEAR-Net, its detailed introduction is omitted here.

[0125] The above is an introduction to the AAD ResBlk module.

[0126] Sub-step 3312: Use the attribute features output by the first face image in the i-th upsampling layer of the attribute feature extraction network, the identity features output by the second face image in the i-th layer of the identity feature extraction network, and the output of the i-1-th fusion module as the input of the i-th fusion module in the face fusion network to obtain the output of the i-th fusion module; where i is a positive integer greater than 1 and not greater than the number of fusion modules in the face fusion network.

[0127] by Figure 5For example, the attribute feature xt2 output by the second upsampling layer of the attribute feature extraction network of the first face image, the identity feature Id2 output by the second-to-last layer of the identity feature extraction network of the second face image, and the output of the first fusion module (the first and third face images) are used as the input of the second fusion module in the face fusion network to obtain the output of the second fusion module (i.e., the second and third face images);

[0128] ...And so on. Finally, the attribute feature xt5 output by the 5th upsampling layer of the attribute feature extraction network of the first face image, the identity feature Id5 output by the 5th to last layer of the identity feature extraction network of the second face image, and the output of the 4th fusion module (the fourth and third face images) are used as the input of the 5th fusion module in the face fusion network to obtain the output of the 5th fusion module (i.e., the fifth and third face images) as the final third face image.

[0129] Sub-step 332: Obtain the fused image output by the last fusion module in the face fusion network as the third face image.

[0130] By using the image processing method provided in the embodiment of the present application, when generating a third facial image based on the second facial image and the first facial image, the attribute features in the first facial image used to characterize the attribute information of the first facial image and the identity features in the second facial image used to characterize the identity information of the second facial image can be obtained. Therefore, the third facial image generated based on the attribute features, identity features and the third image mask will not carry the attribute features of the second facial image, and the generated third facial image also integrates the attribute features of the first facial image, which can make the attribute features of the generated third facial image consistent with those of the first facial image, and the identity features of the third facial image consistent with the identity features of the second facial image, thereby achieving the effect that the generated new facial image does not retain the attribute features in the second facial image that are not related to the face.

[0131] See also Figure 10 , which shows a flow chart of another image processing method provided by an embodiment of the present application. The image processing method may include the following steps:

[0132] Step 1010: Input the first sample face image and the second sample face image into an initial image processing model for first processing.

[0133] The first processing includes: generating a third sample face image based on the first sample face image and the second sample face image; the first sample face image includes a sample attribute feature, and the second sample face image includes a sample identity feature.

[0134] Step 1020: Input the third sample face image into the initial image processing model for second processing.

[0135] The second processing includes: extracting reference attribute features of the third sample face image and extracting reference identity features of the third sample face image;

[0136] Step 1030 , based on the reference attribute features, the sample attribute features, the sample identity features and the reference identity features, a training loss is determined, and the parameters of the initial image processing model are adjusted based on the training loss to obtain an image processing model.

[0137] Optionally, determining the training loss based on the reference attribute feature, the sample attribute feature, the sample identity feature, and the reference identity feature includes:

[0138] Obtaining a first loss based on the sample identity feature and the reference identity feature;

[0139] Obtaining a second loss based on the sample attribute features and the reference attribute features;

[0140] Calling the generative adversarial network to determine a third loss based on the third sample face image;

[0141] Based on the first loss, the second loss, and the third loss, the training loss is determined.

[0142] As an example, the generative adversarial network of an embodiment of the present application uses a pre-trained deepfake detection network Face X-ray to input the generated final third face image into the generative adversarial network to identify authenticity.

[0143] As an example, the training data for a pre-trained identity feature extraction network used for ID extraction can be the FFHQ dataset. Face-swapped data can be generated using multiple face-swapped algorithms, such as Faceshifer, FSGAN, and E4S. This data is then used to train the identity feature extraction. Once this network is trained, its parameters can be fixed, and the entire image processing model can be trained using datasets such as FFHQ and CelebA. The following describes the loss functions used in training each network module.

[0144] The first loss is the loss function L of the identity feature extraction network id : In order to verify the identity features of the third face image finally generated by the image processing model to determine that the identity features of the third face image are consistent with the identity features in the second face image, the embodiment of the present application uses the loss function L during the training phase of the image processing model. idThe third face image generated by the image processing model is input into the identity feature extraction network to obtain the output identity feature. The cosine distance is calculated with the identity feature output after the second face image is input to ensure that the two identity features are similar. X s is the second face image, This is the third face image finally generated by the image processing model.

[0145] The second loss is the loss function L of the attribute feature extraction network att : That is, the L2 norm, which is used to input the third face image finally generated by the image processing model into the third face image finally generated by the image processing model, obtain the output attribute features, and calculate the L2 distance between the output attribute features and the original input first face image. X t is the first face image.

[0146] The third loss is the loss function L of the discriminator of the generated adversarial network adv : It is used to determine whether the third face image finally generated by the image processing model is a real image or a forged image, in order to maintain the authenticity of the generated face-swapped image.

[0147] The training loss is the final loss function L of the image processing model AEI-Net =L id +L att +L rec +L adv .

[0148] Optionally, determining a training loss based on the first loss, the second loss, and the third loss includes:

[0149] When the first sample facial image and the second sample facial image are the same, determining a fourth loss based on the third sample facial image and the first sample facial image;

[0150] Based on the first loss, the second loss, the third loss, and the fourth loss, the training loss is determined.

[0151] Among them, the fourth loss is the loss function L of the recognition rec :This loss will only be generated when the second face image input is exactly the same as the first face image. This loss actually measures the similarity between the third face image finally generated by the image processing model and the first face image. Therefore, only when the second face image is exactly the same as the first face image will the third face image finally generated by the image processing model be similar to the first face image. At other times, the loss defaults to 0.

[0152]

[0153] By using the image processing method provided in the embodiment of the present application, when training the image processing model, not only can the attribute features used to characterize the attribute information of the first sample face image and the identity features used to characterize the identity information of the second sample face image be obtained in the first sample face image, but also the reference attribute features of the third sample face image and the reference identity features of the third sample face image can be extracted, and the training loss is determined based on the reference attribute features, sample attribute features, sample identity features and reference identity features. Finally, the parameters of the initial image processing model are adjusted based on the training loss to obtain the image processing model. In this process, since the image processing model can be trained to decouple the attribute features and identity features from the face image respectively, and the training loss is determined based on the reference attribute features and reference identity features, the trained image processing model can perform operations such as decoupling the attribute features and identity features and fusing the attribute features and identity features on the input face image when performing face image processing, thereby effectively improving the face generation effect of the image processing model in the face-changing scenario.

[0154] Figure 11 FIG1 is a structural diagram of an image processing device 1100 provided by an exemplary embodiment of the present application. Figure 11 As shown, the apparatus 1100 includes: a first determination module 1110, a mask module 1120, a second determination module 1130, and a generation module 1140, wherein:

[0155] A first determining module 1110 is configured to determine attribute features of the first facial image and identity features of the second facial image;

[0156] a masking module 1120 configured to mask the facial region in the first facial image to obtain a first image mask, and to mask the facial region in the second facial image to obtain a second image mask;

[0157] A second determining module 1130 is configured to determine a third image mask based on an overlapping area between the first image mask and the second image mask;

[0158] The generating module 1140 is configured to generate a third facial image based on the attribute features, the identity features, and the third image mask.

[0159] Optionally, the second determining module 1130 is specifically configured to:

[0160] determining an overlapping area between the second image mask and the first image mask, and using the overlapping area as the third image mask; or,

[0161] The overlapping area between the second image mask and the first image mask is mapped to the second facial image, the facial occlusion area of ​​the second facial image within the overlapping area is determined, and the facial occlusion area existing in the overlapping area is removed to obtain the third image mask.

[0162] Optionally, the generating module 1140 is specifically configured to:

[0163] Performing fusion processing based on the attribute feature, the identity feature, and the third image mask to obtain the third face image;

[0164] The fusion processing includes: performing a specified matrix operation based on the attribute feature, the identity feature and the third image mask.

[0165] Optionally, when performing a specified matrix operation based on the attribute feature, the identity feature, and the third image mask, the generating module 1140 is specifically configured to:

[0166] The attribute activation result obtained by convolution processing, vector operation and denormalization processing based on the attribute features, the processing result obtained by mapping the identity features to the sample label space and vector operation, and the third image mask are subjected to specified matrix operation.

[0167] Optionally, the specified matrix operation includes:

[0168] performing a vector dot product operation on the attribute activation result and a difference between the specified value and the third image mask to obtain a first vector dot product operation result, and performing a vector dot product operation on the third image mask and a processing result obtained by mapping the identity feature to the sample label space and the vector operation to obtain a second vector dot product operation result;

[0169] The third facial image is generated by performing a dot product operation on the first vector and a dot product operation on the second vector.

[0170] The image processing device 1100 can realize Figures 1 to 5 For details, please refer to the method of the embodiment of the method. Figures 1 to 5 The image processing method of the illustrated embodiment will not be described in detail.

[0171] Figure 12 FIG1 is a structural diagram of another image processing device 1200 provided by an exemplary embodiment of the present application. Figure 12 As shown, the apparatus 1200 includes: a processing module 1210 and an adjustment module 1220, wherein:

[0172] The processing module 1210 is configured to input the first sample facial image and the second sample facial image into an initial image processing model for first processing, wherein the first processing includes: generating a third sample facial image based on the first sample facial image and the second sample facial image; the first sample facial image includes a sample attribute feature, and the second sample facial image includes a sample identity feature;

[0173] The processing module 1210 is further configured to input the third sample facial image into the initial image processing model for second processing, wherein the second processing includes: extracting reference attribute features of the third sample facial image and extracting reference identity features of the third sample facial image;

[0174] The adjustment module 1220 is used to determine the training loss based on the reference attribute feature, the sample attribute feature, the sample identity feature and the reference identity feature, and adjust the parameters of the initial image processing model based on the training loss to obtain the image processing model.

[0175] Optionally, when determining the training loss based on the reference attribute feature, the sample attribute feature, the sample identity feature, and the reference identity feature, the adjustment module 1220 is specifically configured to:

[0176] Based on the sample identity feature and the reference identity feature, a first loss is obtained; based on the sample attribute feature and the reference attribute feature, a second loss is obtained;

[0177] Calling a generative adversarial network to determine a third loss based on the third sample face image;

[0178] The training loss is determined based on the first loss, the second loss, and the third loss.

[0179] Optionally, when the adjustment module 1220 determines the training loss based on the first loss, the second loss, and the third loss, it is specifically configured to:

[0180] When the first sample facial image and the second sample facial image are the same, determining a fourth loss based on the third sample facial image and the first sample facial image;

[0181] The training loss is determined based on the first loss, the second loss, the third loss, and the fourth loss.

[0182] The image processing device 1200 can realize Figures 6 to 10 For details, please refer to the method of the embodiment of the method. Figures 1 to 10 The image processing method of the illustrated embodiment will not be described in detail.

[0183] Figure 13 This is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Figure 13 As shown, the device includes: a memory 131 and a processor 132.

[0184] The memory 131 is used to store computer programs and may be configured to store various other data to support operations on the computing device. Examples of such data include instructions for any application or method operating on the computing device, contact data, phone book data, messages, images, videos, etc.

[0185] The processor 132 is coupled to the memory 131 and configured to execute a computer program in the memory 131 to: determine attribute features of a first facial image and determine identity features of a second facial image; mask a facial region in the first facial image to obtain a first image mask, and mask a facial region in the second facial image to obtain a second image mask; determine a third image mask based on an overlapping area between the first image mask and the second image mask; and generate a third facial image based on the attribute features, the identity features, and the third image mask.

[0186] Alternatively, the first sample face image and the second sample face image are input into the initial image processing model for a first processing, wherein the first processing includes: generating a third sample face image based on the first sample face image and the second sample face image; the first sample face image includes a sample attribute feature, and the second sample face image includes a sample identity feature; the third sample face image is input into the initial image processing model for a second processing, wherein the second processing includes: extracting a reference attribute feature of the third sample face image and extracting a reference identity feature of the third sample face image; determining a training loss based on the reference attribute feature, the sample attribute feature, the sample identity feature and the reference identity feature, and adjusting the parameters of the initial image processing model based on the training loss to obtain an image processing model.

[0187] Further, if Figure 13 As shown, the electronic device also includes: a communication component 133, a display 134, a power component 135, an audio component 136 and other components. Figure 13 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 13 In addition, depending on the implementation form of the traffic playback device, Figure 13 The components in the dotted box are optional components, not mandatory components. For example, when the electronic device is implemented as a terminal device such as a smartphone, tablet computer or desktop computer, it may include Figure 13Components in the dotted box; when the electronic device is implemented as a conventional server, cloud server, data center or server array and other server-side devices, it may not include Figure 13 Components within the dotted box.

[0188] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0189] Accordingly, an embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed, can implement the steps that can be performed by the electronic device in the above method embodiment. Optionally, the computer program product, in addition to performing the steps in the above method embodiment.

[0190] above Figure 13 The communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may also include a near field communication (NFC) module, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, etc.

[0191] above Figure 13 The memory in the MCU may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0192] above Figure 13 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0193] above Figure 13The power supply component in a device provides power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0194] above Figure 13 The audio component in the device may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0195] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0197] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0200] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0201] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0202] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0203] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0204] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: include: Determining attribute features of the first facial image and determining identity features of the second facial image; Masking the face area in the first face image to obtain a first image mask, and masking the face area in the second face image to obtain a second image mask; determining a third image mask based on an overlapping area between the first image mask and the second image mask; A third facial image is generated based on the attribute features, the identity features, and the third image mask.

2. The method according to claim 1, wherein The determining a third image mask based on an overlapping area between the first image mask and the second image mask includes: determining an overlapping area between the second image mask and the first image mask, and using the overlapping area as the third image mask; or, The overlapping area between the second image mask and the first image mask is mapped to the second facial image, the facial occlusion area of ​​the second facial image within the overlapping area is determined, and the facial occlusion area existing in the overlapping area is removed to obtain the third image mask.

3. The method according to claim 1, wherein Generating a third face image based on the attribute feature, the identity feature, and the third image mask includes: Performing fusion processing based on the attribute feature, the identity feature, and the third image mask to obtain the third face image; The fusion processing includes: performing a specified matrix operation based on the attribute feature, the identity feature and the third image mask.

4. The method according to claim 3, wherein The performing a specified matrix operation based on the attribute feature, the identity feature, and the third image mask includes: The attribute activation result obtained by convolution processing, vector operation and denormalization processing based on the attribute features, the processing result obtained by mapping the identity features to the sample label space and vector operation, and the third image mask are subjected to specified matrix operation.

5. The method according to claim 4, wherein The specified matrix operations include: performing a vector dot product operation on the attribute activation result and a difference between the specified value and the third image mask to obtain a first vector dot product operation result, and performing a vector dot product operation on the third image mask and a processing result obtained by mapping the identity feature to the sample label space and the vector operation to obtain a second vector dot product operation result; The third facial image is generated by performing a dot product operation on the first vector and a dot product operation on the second vector.

6. An image processing method, characterized in that: include: Inputting the first sample face image and the second sample face image into the initial image processing model for first processing, the first processing comprising: generating a third sample face image based on the first sample face image and the second sample face image; the first sample face image comprising a sample attribute feature, and the second sample face image comprising a sample identity feature; Inputting the third sample facial image into the initial image processing model for second processing, the second processing comprising: extracting reference attribute features of the third sample facial image and extracting reference identity features of the third sample facial image; Based on the reference attribute feature, the sample attribute feature, the sample identity feature and the reference identity feature, a training loss is determined, and based on the training loss, the parameters of the initial image processing model are adjusted to obtain an image processing model.

7. The method according to claim 6, wherein The determining of the training loss based on the reference attribute feature, the sample attribute feature, the sample identity feature, and the reference identity feature includes: Based on the sample identity feature and the reference identity feature, a first loss is obtained; based on the sample attribute feature and the reference attribute feature, a second loss is obtained; Calling a generative adversarial network to determine a third loss based on the third sample face image; The training loss is determined based on the first loss, the second loss, and the third loss.

8. The method according to claim 7, wherein The determining the training loss based on the first loss, the second loss, and the third loss includes: When the first sample facial image and the second sample facial image are the same, determining a fourth loss based on the third sample facial image and the first sample facial image; The training loss is determined based on the first loss, the second loss, the third loss, and the fourth loss.

9. An image processing device, characterized in that: include: a first determining module, configured to determine attribute features of the first facial image and identity features of the second facial image; a masking module, configured to mask the face region in the first face image to obtain a first image mask, and mask the face region in the second face image to obtain a second image mask; a second determining module, configured to determine a third image mask based on an overlapping area between the first image mask and the second image mask; A generating module is used to generate a third face image based on the attribute feature, the identity feature, and the third image mask.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs being configured to perform the steps in the method according to any one of claims 1 to 8.

Images