Image processing method and device, computer device, storage medium, and computer program

Abstract

The present application provides an image processing method and apparatus, a computer device, a storage medium and a program product, which relate to technical fields such as artificial intelligence, machine learning, smart transportation, etc. The image processing method includes the steps of: inputting identity features of a source image and initial attribute features of at least one scale of a target image into a face swap model in response to a face swap request, the face swap request being used to request replacing a target face in the target image with a source face in the source image; iteratively performing feature fusion on the identity features and the initial attribute features of at least one scale by the face swap model to obtain fusion features; generating a target face swap image by the face swap model based on the fusion features, and outputting the target face swap image, where a face in the target face swap image is a fusion of the identity features of the source face and the target attribute features of the target face.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is filed based on a Chinese patent application having application number 202210626467.1 and filing date of June 2, 2022, and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to technical fields such as artificial intelligence, machine learning, and smart transportation, and in particular to an image processing method and apparatus, a computer device, a storage medium, and a program product. [Background technology]

[0003] Face swapping is an important technique in the field of computer vision, and is widely used in content generation, film and television portrait production, entertainment video production, avatar or privacy protection, etc. Face swapping refers to replacing a target face in an image with another face.

[0004] In the related art, face swapping is usually realized using a neural network model, for example, an image is input to a neural network model for face swapping, and the neural network model performs face swapping on the image to output the obtained image. However, there is a large difference between the image obtained by the face swapping technology in the related art and the ideal image after face swapping, which causes a problem that the effect of face swapping is low. Summary of the Invention [Means for solving the problem]

[0005] The embodiments of the present application provide an image processing method and apparatus, a computer device, a computer-readable storage medium, and a computer program product, which can improve the quality of an image after face swapping.

[0006] An embodiment of the present application provides an image processing method, the image processing method comprising: obtaining identity features of a source image and initial attribute features of at least one scale of a target image in response to a received face swap request, the face swap request being used to request replacement of a target face in the target image with a source face in the source image, the identity features representing an object to which the source face belongs and the initial attribute features representing three-dimensional attributes of the target face; inputting the identity features and the at least one scale initial attribute features into a face swap model; Iteratively performing feature fusion on the identity features and the at least one scale initial attribute features through the face swap model to obtain a fusion feature; and generating a target face-swap image using the face-swap model based on the fusion features, and outputting the target face-swap image, wherein a face in the target face-swap image is a fusion of identity features of the source face and target attribute features of the target face.

[0007] An embodiment of the present application further provides an image processing device, comprising: a feature acquisition module configured to acquire identity features of a source image and initial attribute features of at least one scale of a target image in response to a received face swap request, the face swap request being used to request replacing a target face in the target image with a source face in the source image, the identity features representing an object to which the source face belongs and the initial attribute features representing three-dimensional attributes of the target face; inputting the identity features and the at least one scale initial attribute features into a face swap model in a face swap module; Iteratively performing feature fusion on the identity features and the at least one scale initial attribute features through the face swap model to obtain a fusion feature; generating a target face-swap image by the face-swap model based on the fusion features, and outputting the target face-swap image, wherein a face in the target face-swap image is a fusion of identity features of the source face and target attribute features of the target face.

[0008] An embodiment of the present application further provides a computer device, the computer device including a memory and a processor; The memory stores a computer program; The processor executes a computer program stored in the memory to implement the image processing method described in the embodiments of the present application.

[0009] An embodiment of the present application further provides a computer readable storage medium storing a computer program for causing a processor to execute the image processing method according to the embodiment of the present application.

[0010] An embodiment of the present application further provides a computer program product including a computer program for causing a processor to perform the image processing method according to the embodiment of the present application.

[0011] The beneficial effects brought by the technical solutions provided by the embodiments of the present application are as follows:

[0012] In the image processing method of the embodiment of the present application, the identity features of the source image and the initial attribute features of the target image are input into a face swap model, and the face swap model performs feature fusion iteratively on the identity features and the initial attribute features of at least one scale to obtain a fusion feature, that is, the input end of the face swap model performs view decoupling on the identity features and the attribute features, so that the obtained fusion feature fuses the identity features of the object in the source image and the three-dimensional attributes of the face of the object in the target image.

[0013] Based on the fusion features, a target face-swap image is generated by a face-swap model, and the target face-swap image is output, and the face in the target face-swap image is a fusion of the identity features of the source face and the target attribute features of the target face. In this way, the target face-swap image is generated based on the fusion features obtained by feature fusion, so that the attributes and detailed features of the target face in the target face-swap image are effectively preserved while ensuring the identity consistency between the face in the target face-swap image and the face in the source image, and the clarity, accuracy and authenticity of the face in the face-swap image are greatly improved, and high-resolution face swapping is realized. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of an implementation environment of an image processing method according to an embodiment of the present application; [Diagram 2] 1 is a schematic flow chart of an image processing method according to an embodiment of the present application. [Diagram 3] FIG. 2 is a structural schematic diagram of a face swap model according to an embodiment of the present application; [Figure 4] FIG. 2 is a structural schematic diagram of blocks within a generator according to an embodiment of the present application. [Diagram 5] 1 is a schematic flow chart of a method for training a face swap model according to an embodiment of the present application. [Figure 6] FIG. 2 is a schematic diagram of a control mask of at least one scale according to an embodiment of the present application. [Figure 7] 1A and 1B are schematic diagrams illustrating comparison of face swapping results according to an embodiment of the present application; [Figure 8] 1 is a structural schematic diagram of an image processing device according to an embodiment of the present application; [Figure 9] FIG. 1 is a structural schematic diagram of a computer device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present application will be described with reference to the drawings in the present application. It should be understood that the following embodiments described with reference to the drawings are exemplary explanations for interpreting the technical solutions of the embodiments of the present application, and do not limit the technical solutions of the embodiments of the present application.

[0016] In the following description, references are made to "some embodiments," which describe a subset of all possible embodiments, but it can be understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict.

[0017] Those skilled in the art can understand that the singular forms "a", "one", "the", and "the" used herein can also include the plural forms unless otherwise stated. The terms "including" and "comprising" used in the embodiments of this application mean that the corresponding features can be realized as the presented features, information, data, steps, and operations, but do not exclude them from being realized as other features, information, data, steps, and operations supported in the art.

[0018] It can be understood that in a specific embodiment of the present application, any data related to a subject, such as related source images, target images, source faces, target faces, and at least one set of samples in a sample dataset used during model training, and any data related to a subject, such as face-swap target images, facial features of target faces, and attribute parameters used when performing face swapping using a face swap model, are all obtained after obtaining the consent or permission of the relevant subject. When the following embodiments of the present application are applied to a specific product or technology, the permission or consent of the subject needs to be obtained, and the collection, use, and processing of the relevant data needs to comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, any face swapping process performed on a face image of any subject using the image processing method of the present application is a face swapping process performed after obtaining the permission or consent of the relevant subject based on a face swap service or face swap request triggered by the relevant subject.

[0019] The image processing method provided in the embodiment of the present application involves the following technologies such as artificial intelligence and computer vision, for example, using technologies such as cloud computing and big data processing in artificial intelligence technology to realize the process of training a face swap model, extracting multi-scale attribute features in an image, etc. For example, computer vision technology is used to perform face recognition on an image to obtain identity features corresponding to a face in an image.

[0020] It should be understood that artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use the knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that seeks to understand the nature of intelligence and produce new intelligent machines that respond in a manner similar to human intelligence. In other words, artificial intelligence studies the design principles and realization methods of various intelligent machines, and gives machines the functions of sensing, reasoning, and decision-making.

[0021] Artificial intelligence technology is a comprehensive discipline with a wide range of fields, including hardware technology and software technology. AI basic technologies generally include technologies such as sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, electro-mechanical integration, etc. AI software technology mainly includes computer vision technology, voice processing technology, natural language processing technology, machine learning / deep learning, autonomous driving, smart transportation and other themes.

[0022] As a matter of understanding, Computer Vision Technology (CV) is the science that studies how to make machines "see", using cameras and computers instead of human eyes to identify and measure targets, and further graphic processing to make the computer-processed image more suitable for human eye observation or transfer to equipment detection. As a scientific discipline, Computer Vision studies related theories and technologies to build artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer Vision Technology usually includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning and map building, autonomous driving, smart transportation, and also includes biometric recognition technologies such as face recognition and fingerprint recognition in general.

[0023] 1 is a schematic diagram of an implementation environment of the image processing method according to the present application. As shown in FIG. 1, the implementation environment includes a server 11 and a terminal 12.

[0024] The server 11 is configured with a trained face swap model, and the server 11 can provide a face swap function to the terminal 12 based on the face swap model. The face swap function may be used to generate a face swap image based on a source image and a target image, where the generated face swap image has identity features of a source face in a source image and attribute features of a target face in a template image, where the identity features represent an object to which the source face belongs, and the initial attribute features represent three-dimensional attributes of the target face.

[0025] In some embodiments, the terminal 12 is installed with an application program, and the application program may have a face-swapping function pre-configured, and the server 11 may be a background server of the application program. The terminal 12 and the server 11 may perform data interaction through the application program to realize a face-swapping process. Exemplarily, the terminal 12 may send a face-swapping request to the server 11, and the face-swapping request is used to request replacing a target face in the target image with a source face in the source image. The server 11 may execute the image processing method of the present application based on the face-swapping request to generate a target face-swapping image, and return the target face-swapping image to the terminal 12. For example, the application program may be any one application that supports a face-swapping function, for example, the application program may include, but is not limited to, a video editing application, an image processing tool, a video application, a live broadcast application, a social application, a content interaction platform, a game application, and the like.

[0026] The server may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The network may include, but is not limited to, wired networks and wireless networks, where the wired networks include local area networks, metropolitan area networks, and wide area networks, and the wireless networks include networks that realize Bluetooth, Wi-Fi, and other wireless communications. The terminal may be, but is not limited to, a smartphone (Android mobile phone, iOS mobile phone, etc.), a tablet computer, a notebook computer, a digital broadcast receiver, a mobile Internet device (MID), a personal digital assistant (PDA), a desktop computer, an in-vehicle terminal (in-vehicle navigation terminal, in-vehicle computer, etc.), a smart home appliance, an airplane, a smart speaker, a smart watch, etc., and the terminal and the server may be directly or indirectly connected by wired or wireless communications, but is not limited to these. Specifically, the terminal may be determined based on the requirements of the actual application scenario, and is not limited here.

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, the following describes the embodiments of the present application in detail with reference to the drawings.

[0028] In the following, we first explain technical terms related to this application.

[0029] Face swapping: Replacing a face in an image with another face. For example, if we take a source image X s and target image Xt , the image processing method of the present application can be used to generate a face-swapped image Y s,t Generate the face-swapped image Y s,t is the source image X s The identity feature of the target image X t It retains non-identity related attribute features in the

[0030] Face Swap Model: Used to replace a target face in a target image with a source face in the source image.

[0031] Source Image: An image that provides identity features, and the faces in the generated face-swap image have the identity features of the faces in the source image.

[0032] Target image: an image that provides attribute features, and the face in the generated face-swap image has the attribute features of the face in the target image. For example, when the source image is an image of a target A, and the target image is an image of a target B, and the face of the target B in the target image is replaced with the face of the target A to obtain a face-swap image, the identity of the face in the face-swap image is the face of the target A, and the face in the face-swap image has the same identity features as the eye shape, eye spacing, nose size, etc. of the target A, and the face in the face-swap image has attribute features such as facial expression, hair, light exposure, wrinkles, posture, face occlusion, etc. of the target B.

[0033] 2 is a schematic flowchart of an image processing method according to an embodiment of the present application. The execution subject of the method may be a computer device (which may be a terminal or a server). As shown in FIG. 2, the method includes the following steps 201 to 203.

[0034] In step 201, the computing device obtains identity features of a source image and initial attribute features of at least one scale of a target image in response to a received face swap request.

[0035] The face swap request is used to request replacing a target face in the target image with a source face in the source image. In practical application, the face swap request includes a source image and a target image, and the computer device obtains the source image and the target image by analyzing the face swap request, or the face swap request includes an identifier of a source image and an identifier of a target image, and the computer device obtains the identifier of a source image and the identifier of a target image by analyzing the face swap request, and then searches for the source image and the target image in an image library according to the identifiers.

[0036] The computing device can obtain a face-swap image using a trained face-swap model, thereby providing a face-swap function. Here, the identity feature represents an object to which the source face belongs. Exemplarily, the identity feature may be a feature that identifies the identity of the object, and the identity feature can include at least one of the five senses features of the target face of the object or the contour feature of the target face. The five senses features of the target face refer to features corresponding to the five senses, and the contour feature of the target face refers to features corresponding to the contour of the target face. For example, the identity feature can include, but is not limited to, eye shape, distance between the eyes, nose size, eyebrow shape, facial contour, etc. The initial attribute feature represents a three-dimensional attribute of the target face, for example, the initial attribute feature can represent attributes such as the posture, spatial environment, etc. of the target face in a three-dimensional space. For example, the initial attribute feature can include, but is not limited to, background, lighting, wrinkles, posture, facial expression, hair, facial occlusion, etc.

[0037] In some embodiments, the face swap model may include an identity recognition network, and the computer device may input a source image into the face swap model, and perform face recognition on the source image through the identity recognition network in the face swap model to obtain identity features of the source image. Exemplarily, the identity recognition network is used to recognize an identity to which a face in an image belongs based on the input image. For example, the identity recognition network may be a fixed face recognition network (Fixed FR Net) in the face swap model. For example, when the source image is a face image, the identity recognition network may be a trained face recognition model, and the face recognition model is used to recognize an object to which a face in the source image belongs and obtain identity features for identifying the object, and the identity features may include at least one of an eye shape feature, an eye spacing feature, a nose size feature, an eyebrow shape feature, and a face contour feature. The identity features may be a fixed-dimensional feature vector output by the face recognition model, for example, a 512-dimensional feature vector. The 512-dimensional feature vector may represent features such as eye shape, eye spacing, nose size, eyebrow shape, and face contour.

[0038] In some embodiments, the face swap model further includes an attribute feature extraction network, and the attribute feature extraction network can include an encoder and a decoder, the encoder includes at least one encoding network layer (e.g., includes at least two encoding network layers), and the decoder includes at least one decoding network layer (e.g., includes at least two decoding network layers). For example, the attribute feature extraction network is a U-type deep network including an encoder and a decoder. In practical application, the computer device can obtain the initial attribute feature of at least one scale of the target image in the following manner:

[0039] The computer device performs layer-by-layer downsampling on a target image by at least one encoding network layer of an encoder to obtain encoded features, performs layer-by-layer upsampling on the encoded features by at least one decoding network layer of a decoder to output decoded features of different scales, and takes the decoded features of different scales produced by the at least one decoding network layer as initial attribute features, where each decoding network layer corresponds to one of the scales.

[0040] Exemplarily, each of the encoding network layers is used to perform an encoding operation on a target image to obtain an encoded feature, and each of the decoding network layers is used to perform a decoding operation on the encoded feature to obtain an initial attribute feature. The decoder performs a reverse operation according to the operation principle of the encoder when executed, for example, the encoder can perform downsampling on the target image, and the decoder can perform upsampling on the downsampled encoded feature. For example, the encoder can be an autoencoder (AE), and the decoder can be a decoder corresponding to the autoencoder.

[0041] In some embodiments, each encoding network layer is used to downsample the encoding features output by the previous encoding network layer to obtain encoding features of at least one scale, and each encoding network layer corresponds to one scale. Each decoding network layer is used to upsample the decoded features output by the previous decoding network layer to obtain initial attribute features of at least one scale, and each decoding network layer corresponds to one scale. The scales of the encoding network layer and the decoding network layer located in the same layer may be the same. Here, each decoding network layer can combine the encoding features of the encoding network layer of the corresponding scale and perform upsampling on the initial attribute features output by the previous decoding network layer. As shown in FIG. 3, in FIG. 3, a U-type deep network is used to decode a target image Xt For example, a target image is input to an encoder, which includes multiple (i.e., at least two) encoding network layers, each of which corresponds to the resolution (i.e., scale) of one feature map, and the multiple encoding network layers of the encoder extract features from the target image X t The resolutions of the feature maps of the encoded features of are respectively 1024×1024, 512×512, 256×256, 128×128, and 64×64. The 64×64 feature map is input to the first decoding network layer of the decoder for upsampling to obtain a 128×128 decoded feature map. The 128×128 decoded feature map and the 128×128 encoded feature map are concatenated, and the concatenated feature map is upsampling to obtain a 256×256 decoded feature map. By analogy, the feature maps of various resolutions obtained by decoding according to the network structure of the U-type deep network are used as the initial attribute features. In the initial attribute features, the initial attribute features of each scale are used to represent the attribute features at the corresponding scale of the target image, and the attribute features corresponding to the initial attribute features of different scales may be different, and the initial attribute features of a relatively small scale can represent the information such as the global position, posture, etc. of the target face in the target image, and the initial attribute features of a relatively large scale can represent the local details of the target face in the target image, so that the initial attribute features of the at least one scale can cover the attribute features at multiple levels of the object. For example, the initial attribute features of the at least one scale may be multiple feature maps with resolutions from small to large, and the feature map of resolution R1 can represent the facial position of the target face in the target image, the feature map of resolution R2 can represent the posture expression of the target face in the target image, and the feature map of resolution R3 can represent the facial details of the facial position of the target face in the target image. Here, the resolution R1 is smaller than R2, and R2 is smaller than R3.

[0042] In step 202, the computing device iteratively performs feature fusion on the identity features and the initial attribute features of at least one scale through a face swap model to obtain a fusion feature.

[0043] In step 203, the computing device generates a target face-swapped image by a face-swap model based on the fusion features, and outputs the target face-swapped image.

[0044] Here, the face in the target face-swap image is a fusion of the identity features of the source face and the target attribute features of the target face.

[0045] In some embodiments, the face swap model includes a generator, the generator includes at least one convolution layer (e.g., at least two convolution layers), the at least one convolution layer is connected in series, and each convolution layer corresponds to one scale. The computer device can iteratively perform feature fusion on the identity features and the initial attribute features of at least one scale through the face swap model in the following manner to obtain fusion features.

[0046] The computer device performs the following process on the identity features and the initial attribute features of the corresponding scales through each convolution layer of the face swap model: obtain a first feature map output by a convolution layer before the current convolution layer, generate a second feature map according to the identity features and the first feature map, select a target attribute feature from the initial attribute features of at least one scale, the target attribute feature being a feature other than the identity feature of the target face, and generate a third feature map according to the target attribute feature and the second feature map, the third feature map being the first feature map of the convolution layer next to the current convolution layer, and determine the third feature map output by the last convolution layer of the at least one convolution layer as a fusion feature.

[0047] In practical application, the initial attribute features and the number of convolution layers are both target numbers, the target number of convolution layers are connected in series, different initial attribute features correspond to different scales, each convolution layer corresponds to the initial attribute features of one scale, and the target number is 2 or more. When the current convolution layer is the first convolution layer of the target number of convolution layers, obtain an initial feature map, and use the initial feature map as the first feature map input to the current convolution layer. Here, in practical application, the initial feature map may be an all-zero feature vector with a fixed dimension.

[0048] In some embodiments, the computing device can select target attribute features from the initial attribute features of at least one scale in the following manner: determine a control mask at a corresponding scale of the image based on the feature map and the attribute features, the control mask being used to represent pixel points carrying features other than identity features of the target face; and select the initial attribute features of at least one scale based on the control mask to obtain target attribute features.

[0049] Exemplarily, the computer device can input the identity feature to each convolutional layer of the generator. The computer device inputs the initial attribute feature of at least one scale to a convolutional layer that matches the scale of the initial attribute feature in the generator, where the scale of the feature map output by each convolutional layer of the generator is different, and the scale of the feature map output by the convolutional layer that matches the scale of the initial attribute feature is the same as the scale of the initial attribute feature. For example, if a convolutional layer in the generator is used to process a 64x64 feature map from a previous convolutional layer and output a 128x128 feature map, the initial attribute feature of 128x128 can be input to the convolutional layer.

[0050] In some embodiments, in the generator, the computer device can determine a control mask of at least one scale of the target image based on the identity features and the initial attribute features of at least one scale, and obtain a target face-replaced image based on the identity features, the control mask of at least one scale, and the initial attribute features. Illustratively, the control mask represents pixel points carrying features other than the identity features of the target face, and the computer device can determine a target attribute feature of at least one scale based on the control mask of the at least one scale and the initial attribute features, and generate the target face-replaced image based on the identity features and the target attribute feature of the at least one scale.

[0051] The computer device can obtain the target face-swapped image by layer-by-layer processing of each convolutional layer of the generator. In one possible example, the computer device performs the following steps S1 to S4 on the input identity features and corresponding scale initial attribute features by each convolutional layer of the generator.

[0052] In step S1, the computing device obtains a first feature map output by a convolutional layer before a current convolutional layer.

[0053] In the generator, each convolution layer can process the feature map output by the previous convolution layer and output to the next convolution layer. Here, for the first convolution layer, the computer device can input an initial feature map to the first convolution layer, for example, the initial feature map can be a 4×4×512 all-zero feature vector. For the last convolution layer, the computer device can generate a final target face-swapped image based on the feature map output by the last convolution layer.

[0054] In step S2, the computing device generates a second feature map based on the identity features and the first feature map, and determines a control mask at a corresponding scale of the target image based on the second feature map and the initial attribute features.

[0055] The control mask represents pixel points that carry features other than the identity features of the target face.

[0056] In some embodiments, the computer device adjusts weights of convolution kernels of the current convolution layer based on the identity feature, and obtains the second feature map based on the first feature map and the adjusted convolution kernel. Illustratively, the step of the computer device generating the second feature map may include the steps of: the computer device performs an affine transformation on the identity feature to obtain a first control vector; and the computer device maps the first convolution kernel of the current convolution layer to a second convolution kernel based on the first control vector, and performs a convolution operation on the first feature map based on the second convolution kernel to generate the second feature map. Illustratively, the identity feature may be expressed in the form of an identity feature vector, and the affine transformation refers to an operation of performing a linear transformation and a translation on the identity feature vector to obtain a first control vector. The affine transformation operation includes, but is not limited to, translation, zoom, rotation, and inversion transformation, and each convolution layer of the generator includes a trained affine parameter matrix, and the computer device can perform transformation such as translation, zoom, rotation, and inversion on the identity feature vector based on the affine parameter matrix. Exemplarily, the computer device can perform a modulation operation (Mod) and a demodulation operation (Demod) on the first convolution layer of the current convolution layer according to a first control vector to obtain a second convolution kernel. Here, the modulation operation can be a zoom operation on the weight of the convolution kernel of the current convolution layer, and the demodulation operation can be a normalization operation on the weight of the convolution kernel after the zoom operation. For example, the computer device can perform a zoom operation on the weight of the convolution kernel according to the zoom ratio corresponding to the first feature map input to the current convolution layer and the first control vector.

[0057] In some embodiments, the computer device obtains a control mask of a corresponding scale according to the second feature map and the initial attribute feature of the corresponding scale input to the current convolution layer. The process may include: the computer device performs feature connection on the second feature map and the initial attribute feature to obtain a connected feature map; and the computer device maps the connected feature map to the control mask according to a pre-set mapping convolution kernel and activation function. Exemplarily, the control mask is a binarized image, in which pixel points carrying features other than the identity feature of the target face, such as pixel points in the hair region and pixel points in the background region, take 1, and pixel points carrying the identity feature take 0. Exemplarily, the mapping convolution kernel may be a 1×1 convolution kernel, and the activation function may be a Sigmoid function. For example, the second feature map and the initial attribute features may be expressed in the form of a feature vector, and the computing device may perform a merging operation on the feature vector corresponding to the second feature map and the feature vector corresponding to the initial attribute features to obtain the concatenated vector, and perform a convolution operation and an activation operation on the concatenated vector to obtain the control mask.

[0058] Exemplarily, the generator may include multiple blocks, each block including multiple layers, and the computer device inputs the identity features and the initial attribute features of each scale into the block of the corresponding scale, and the block may perform layer-by-layer processing on the identity features and the initial attribute features input by at least one layer. Exemplarily, FIG. 4 shows the network structure of the i-th block (i-th GAN block, i-th adversarial network block) in the generator, where N represents the attribute injection module (AttrInjection), and the dashed box on the right side shows the internal structure of the attribute injection module in an enlarged manner. As shown in FIG. 4, the i-th block includes two layers, and the first layer is taken as an example. In FIG. 4, w on the left side represents the identity feature f of the source image. id4, A represents an affine transform operation, and a first control vector is obtained by performing an affine transform operation on the identity feature vector. Mod and Demod in FIG. 4 represent modulation and demodulation operations on the convolution kernel Conv3×3, and the computer device performs an upsample operation on the first feature map input to the current layer of the current block, and then performs a convolution operation on the upsampled first feature map by the convolution kernel Conv3×3 after the Mod and Demod operations to obtain a second feature map. Then, the computer device performs a convolution operation on the first feature map after upsample by using the convolution kernel Conv3×3 after the Mod and Demod operations to obtain a second feature map. Then, the computer device performs a convolution operation on the first feature map after upsample by using the second feature map and the initial attribute feature f input to the current block. i att Then, using the convolution kernel Conv1×1 and the Sigmoid function, we apply the concatenated feature vector to the control mask M i,j att to.

[0059] In step S3, the computing device filters the initial attribute features based on the control mask to obtain target attribute features.

[0060] The computing device performs dot multiplication on the feature vector corresponding to the control mask and the feature vector corresponding to the initial attribute feature to select the target attribute feature among the initial attribute features.

[0061] As shown in FIG. 4, the computer device includes a control mask M i,j att and the initial attribute feature f id , and adding the feature vector obtained by the dot multiplication to the feature vector corresponding to the second feature map to obtain the target attribute feature.

[0062] In step S4, the computing device generates a third feature map based on the target attribute feature and the second feature map, and outputs the third feature map to the next convolution layer of the current convolution layer as the first feature map of the next convolution layer.

[0063] The computing device may add a feature vector corresponding to the second feature map and a feature vector corresponding to the target attribute feature to obtain the third feature map.

[0064] It should be noted that, for each convolutional layer included in the generator, the computing device can repeatedly perform the above steps S1 to S4 until repeatedly performing the above steps S1 to S4 for the last convolutional layer of the generator, obtain a third feature map output by the last convolutional layer, and generate a target face-swapped image based on the third feature map output by the last convolutional layer.

[0065] As shown in Figure 4, if the i-th block includes two layers, the third feature map can be input to the second layer of the i-th block, and the operation of the first layer is repeated, and the feature map obtained by the second layer is output to the next block, and so on until the last block. As shown in Figure 3, in Figure 3, N represents an attribute injection module, and the dashed box represents a generator that adopts the StyleGAN2 model. For the N blocks included in the generator, s Identity characteristics of f id The attribute injection module inputs the corresponding initial attribute features f1 att , f2 att , ..., f i att , ..., f N-1 att , f N attare input into N blocks correspondingly, and the above steps S1 to S4 are carried out in each block until the feature map output by the last block is obtained. Then, the final target face-swapped image Y s,t , thereby completing the face swap.

[0066] FIG. 5 is a schematic flowchart of a method for training a face swap model according to an embodiment of the present application, the method may be performed by a computer device. As shown in FIG. 5, the method includes: In step 501, a computing device obtains sample identity features of a sample source image in a sample image pair and sample initial attribute features of at least one scale of a sample target image in the sample image pair.

[0067] In practical application, the computer device obtains a sample dataset, the sample dataset includes at least one sample image pair, and the computer device trains a face swap model by the sample dataset. Wherein, each sample image pair includes one sample source image and one sample target image. In some embodiments, the sample image pair can include a first sample image pair and a second sample image pair, the first sample image pair includes a sample source image and a sample target image belonging to the same object, and the second sample image pair includes a sample source image and a sample target image belonging to different objects. For example, the sample image pair includes one source image X of object A, and one sample target image X of object B. s and one target image X t and one source image X of object A. s and one target image X of subject B t and a second sample image pair consisting of: The first sample image pair and the second sample image pair are both marked with a truth label, the truth label indicating whether the corresponding source and target images are of the same object.

[0068] Here, obtaining the sample identity feature of the sample source image and the sample initial attribute feature of the sample target image can be realized by an initial face exchange model. In some embodiments, the initial face exchange model can include an initial identity recognition network and an attribute feature map extraction network, and the computer device can respectively extract the sample identity feature of the sample source image and the sample initial attribute feature of at least one scale of the sample target image by the initial identity recognition network and the attribute feature map extraction network. It should be explained that here, the embodiment of obtaining the sample identity feature and the sample initial attribute feature is the same process as the method of obtaining the identity feature and the initial attribute feature in step 201 above, and will not be described again here.

[0069] In step 502, the computing device iteratively performs feature fusion on the sample identity features and the sample initial attribute features at at least one scale by the generator of the initial face swap model to obtain sample fusion features, and generates a sample generated image by the generator of the initial face swap model based on the sample fusion features.

[0070] In some embodiments, the generator of the initial face swap model determines a sample mask for at least one scale based on sample identity features of the sample source image and sample initial attribute features for at least one scale of the sample target image, and generates a sample generated image corresponding to the sample image pair based on the sample identity features, the sample mask for the at least one scale, and the sample initial attribute features.

[0071] The generator includes a plurality of convolutional layers, and for each sample image pair, the computing device inputs sample identity features to each convolutional layer, inputs sample initial attribute features of at least one scale to a convolutional layer matching the scale of the sample initial attribute features, and obtains the sample generated image by layer-by-layer processing of each convolutional layer.

[0072] For example, the computer device may perform the following steps for the input sample identity features and sample initial attribute features of corresponding scales by each convolutional layer of the generator: The computer device obtains a first sample feature map output by an initial convolutional layer before the current initial convolutional layer, generates a second sample feature map according to the sample identity features and the first sample feature map, determines a sample mask at the corresponding scale of the sample target image according to the second sample feature map and the sample initial attribute features, and filters the sample initial attribute features according to the sample mask to obtain sample target attribute features; The computer device generates a third sample feature map according to the sample target attribute features and the second sample feature map, and outputs the third sample feature map to the next convolutional layer of the current convolutional layer as the first sample feature map of the next convolutional layer. The above steps are repeated in this manner until the last convolutional layer of the generator is subjected to the cycle to obtain a third feature map output by the last convolutional layer, and a sample generated image is obtained based on the third feature map output by the last convolutional layer.

[0073] It should be noted that in the model training stage, the steps performed by each convolutional layer are similar to the steps performed by each convolutional layer in the generator of the trained face swap model (i.e., steps S1-S4 above), and will not be repeated here.

[0074] In step S503, the computer device uses a classifier of the initial face transformation model to classify the sample generated image and the sample source image, and obtains a classification result.

[0075] Here, a sample source image and a sample generated image in a sample image pair are input to a classifier of the initial face transformation model, and a classification result of each of the sample source image and the sample generated image by the classifier is obtained.

[0076] The initial face transformation model may further include a classifier, and for each sample image pair, the computer device inputs the sample source image and the sample generated image into the classifier, and the classifier outputs a first classification result for the sample source image and a second classification result for the sample generated image, where the first classification result may represent a probability that the sample source image is a real image, and the second classification result may represent a probability that the sample generated image is a real image.

[0077] In some embodiments, the classifier includes at least one convolution layer. Each convolution layer can be used to process the discriminant feature map output by the previous convolution layer of the classifier and output to the next convolution layer of the classifier. Each convolution layer outputs a discriminant feature map for performing feature extraction on the sample source image and a discriminant feature map for performing feature extraction on the sample generated image until the last convolution layer of the classifier, and obtains a first discrimination result based on the discriminant feature map of the sample source image output by the last convolution layer, and obtains a second discrimination result based on the discriminant feature map of the sample generated image output by the last convolution layer.

[0078] In step S504, the computer device determines a loss of the initial face transformation model according to the discrimination result, and trains the initial face transformation model according to the loss to obtain a face transformation model.

[0079] For each sample image pair, the computing device determines a first loss value based on a sample mask of at least one scale of the sample target image in the sample image pair, determines a second loss value based on each discrimination result (i.e., the first discrimination result and the second discrimination result) of the sample source image and the sample generated image by the discriminator, then obtains a training total loss based on the first loss value and the second loss value, trains an initial face transformation model until a target condition is met based on the training total loss, and when the target condition is met, stops the training and obtains a face transformation model.

[0080] In practical application, the computer device can accumulate sample masks of at least one scale, and the accumulated value corresponding to the sample mask of at least one scale is the first loss value.For example, the sample mask can be a binarized image, and the computer device can accumulate the value of each pixel point in the binarized image to obtain the first sum value corresponding to each sample mask, and accumulate the first sum value corresponding to the sample mask of at least one scale to obtain the first loss value.

[0081] As an example, assuming that the generator includes at least one initial block, and each initial block includes at least one layer, for each sample image pair, the computing device can determine a first loss value based on a sample mask of at least one scale of a sample target image in each sample image pair, according to the following Equation 1: Formula 1:L mask =Σ i,j |M i,j |1

[0082] Here, L mask denotes the first loss value, i denotes the ith block of the generator, j denotes the jth layer of the ith block, and M i,j represents the sample mask of the jth layer of the ith block. The computer device accumulates the sample mask of at least one layer of at least one block according to the above formula 1, and in the training stage, the first loss value L maskBy minimizing, the generator can be trained, and the obtained control mask can effectively represent the pixel points of key attribute features other than identity features. Then, the control mask can be used to select key attribute features in the initial attribute features, filter redundant features in the initial attribute features, and retain key features and necessary features in the initial attribute features, thereby avoiding redundant attributes and improving the accuracy of the finally generated face-swapped image.

[0083] It should be noted that the pixel points carrying features other than the identity features of the target face represented by the binarized images of different scales have different degrees of subdivision. FIG. 6 shows sample masks of different scales corresponding to three target images, and each row of sample masks is a sample mask of each scale corresponding to one of the target images. As shown in FIG. 6, for any target image, the resolution of each sample mask from left to right increases sequentially. Take the sample mask change of each scale in the first row as an example, from 4×4, 8×8, 16×16, 32×32, the position of the face in the target image is gradually clearly located, where the pixel points corresponding to the face area take 0, and the pixel points corresponding to the background area other than the face area take 1. From 64×64, 128×128, 256×256, 512×512, 1024×1024, the facial posture and expression in the target image are gradually clearer, and the facial details in the target image are gradually embodied.

[0084] Exemplarily, the computer device may determine a second loss value based on the discrimination results of the sample source image and the sample generated image by the discriminator, respectively, according to the following Equation 2: Formula 2:L GAN =min G max D E[log(D(X s ))]+E[log(1-D(Y s,t ))

[0085] Here, L GAN represents the second loss value, and D(X s) represents a first discrimination result of the sample source image by the discriminator, and the first discrimination result is a sample source image X s is the probability that the image is real, and D(Y s,t ) is the sample generated image Y s,t represents a second discrimination result, which may be a probability that the sample generated image is a real image, and E[log(D(X s ))] is log(D(X s )) and can represent the loss value of the discriminator, E[log(1-D(Y s,t ))] is log(1-D(Y s,t )) and can represent the loss value of the generator, and min G represents the minimized loss function value that the generator expects, and max D represents the maximization loss function value of the discriminator. It should be noted that the initial face transformation model includes a generator and a discriminator, and may be an adversarial network, which learns by playing the generator and the discriminator against each other to obtain an expected machine learning model, which is an unsupervised learning method. The training goal of the generator is to obtain an expected output based on the input. The training goal of the discriminator is to distinguish the image generated by the generator from the real image as much as possible. The input of the discriminator includes a sample source image and a sample generated image generated by the generator. The two network models learn against each other, constantly adjusting the parameters, and the final goal is that the generator can fool the discriminator as much as possible, so that the discriminator cannot determine whether the image generated by the generator is true or not.

[0086] In some embodiments, the computing device may determine the total training loss as a sum of the first loss value and the second loss value.

[0087] In some embodiments, the computer device may further perform training based on sample images of the same subject, and before the computer device determines the training total loss, the computer device obtains a third loss value corresponding to the first sample image pair based on a sample generated image and a sample target image in the first sample image pair. The step of the computer device determining the training total loss may include the step of the computer device obtaining the training total loss based on the third loss value corresponding to the first sample image pair, the first loss value and the second loss value corresponding to the sample image pair.

[0088] Exemplarily, the computing device may obtain a third loss value based on the sample generated image and the sample target image in the first sample image pair according to the following Equation 3: Formula 3:L rec =|Y s,t -X t |1

[0089] Here, L rec represents the third loss value, and Y s,t represents the sample generated image corresponding to the first sample image pair, and X t represents a sample target image in the first sample image pair. It should be noted that, when the sample source image and the sample target image belong to the same object, the face swap result is constrained to be the same as the sample target image, so that when the trained face swap model performs face swap on images of the same object, the generated face swap image is close to the target image, and the accuracy of the model training can be improved.

[0090] In some embodiments, the classifier includes at least one convolutional layer. The computer device can perform loss calculation based on the output result of each convolutional layer of the classifier, and before determining the total training loss, for each sample image pair, the computer device determines a first similarity between the non-face region of the first discriminant feature map and the non-face region of the second discriminant feature map, the first discriminant feature map being a feature map corresponding to the sample target image output by the first partial convolutional layer of the at least one convolutional layer, and the second discriminant feature map being a feature map corresponding to the sample generated image output by the first partial convolutional layer. The computer device determines a second similarity between the third discriminant feature map and the fourth discriminant feature map, the third discriminant feature map being a feature map of the sample target image output by the second partial convolutional layer of the convolutional layer, and the fourth discriminant feature map being a feature map of the sample generated image output by the second partial convolutional layer. The computing device determines a fourth loss value based on the first similarity measure and the second similarity measure corresponding to each sample image pair. The step of determining the training total loss may include the computing device obtaining the training total loss based on the first loss value, the second loss value, and the fourth loss value.

[0091] Exemplarily, the computer device can determine the first similarity through a trained segmentation model. For example, the computer device can obtain a segmentation mask of the first discriminant feature map or the second discriminant feature map through the segmentation model, and determine a first similarity between the non-face region of the first discriminant feature map and the non-face region of the second discriminant feature map based on the segmentation mask. Here, the segmentation mask can be a binarized image of the first discriminant feature map or the second discriminant feature map, in which the pixel points corresponding to the non-face region in the binarized image have a value of 1, and the pixel points corresponding to the region other than the non-face region have a value of 0, thereby effectively extracting the background region other than the face.

[0092] Exemplarily, the computing device may determine a third loss value corresponding to the sample image pair according to the following Equation 4:

number

[0093] Here, L FM represents the fourth loss value, M bg represents the division mask, the classifier includes M convolution layers, the 1st to mth convolution layers are the first partial convolution layers, and the mth to Mth convolution layers are the second partial convolution layers. D i (X t ) represents the feature map of the sample target image output by the i-th convolutional layer in the first partial convolutional layer, and D i (Y s,t ) represents the feature map of the sample generated image output by the i-th convolutional layer in the first partial convolutional layer, and D j (X t ) represents the feature map of the sample target image output by the jth convolutional layer in the second partial convolutional layer, and D j (Y s,t ) represents the feature map of the sample generated image output by the j-th convolution layer in the second partial convolution layer. It should be noted that the value of m is a positive integer from 0 to M, and the value of m may be set as necessary, and the present application is not limited thereto.

[0094] In some embodiments, the computer device can further obtain a similarity situation between identity features based on each image, respectively, and perform loss calculation. Illustratively, before determining the training total loss, for each sample image pair, the computer device can respectively extract a first identity feature of a sample source image, a second identity feature of a sample target image, and a third identity feature of a sample generated image, and determine a first identity similarity between the sample source image and the sample generated image based on the first identity feature and the third identity feature. The computer device determines a first identity distance between the sample generated image and the sample target image based on the second identity feature and the third identity feature, and determines a second identity distance between the sample source image and the sample target image based on the first identity feature and the second identity feature, and the computer device determines a distance difference based on the first identity distance and the second identity distance. The computer device determines a fifth loss value corresponding to the sample image pair based on the first identity similarity and the distance difference corresponding to each sample image pair. The computing device determining a training total loss may include the computing device deriving the training total loss based on a first loss value, a second loss value, and a fifth loss value.

[0095] Illustratively, the computing device may determine the fifth loss value according to Equation 5: Formula 5: L ICL =1-cos(z id (Y s,t ),z id (X s ))+(cos(z id (Y s,t ),z id (X t ))-cos(z id (X s ),z id (X t ))) 2

[0096] Here, L ICL represents the fifth loss value, z id (X s ) represents the first identity feature of the sample source image, and z id (X t ) represents the second identity feature of the sample target image, and z id (Y s,t ) represents the third identity feature of the sample generated image, and 1-cos(z id (Y s,t ),z id (X s )) represents the first identity similarity between the sample source image and the sample generated image, and cos(z id (Y s,t ),z id (X t )) represents the first identity distance between the sample generated image and the sample target image, and cos(z id (X s ),z id (X t )) represents the second identity distance between the sample source image and the sample target image, and (cos(z id (Y s,t ),z id (X t ))-cos(z id (X s ),z id (X t ))) 2 represents the distance difference.

[0097] It should be noted that the distance difference is determined by the first identity distance and the second identity distance, and the distance between the sample source image and the sample target image is measured by the second identity distance, so that the distance difference is minimized to make the first identity distance, i.e., the sample generated image and the sample target image have a certain distance, which corresponds to the distance between the sample source image and the sample target image. The first identity similarity ensures that the generated image has the identity features of the target image, thereby improving the accuracy of model training and improving the accuracy of face swapping.

[0098] For example, the training total loss includes the above five loss values, and the computer device can determine the training total loss according to the following Equation 6: Formula 6:L total =L GAN +L mask +L FM +10*L rec +5*L ICL

[0099] Here, L total represents the total training loss, and L GAN represents the second loss value, and L mask represents the first loss value, and L FM represents the fourth loss value, and L rec represents the third loss value, L ICL represents the fifth loss value.

[0100] In practical application, the computer device trains the initial face transformation model according to the total training loss until it meets the target condition, and when it meets the target condition, stops the training to obtain the face transformation model.

[0101] It should be noted that the computer device performs iterative training on the initial face transformation model based on the above steps 501 to 504, obtains a total training loss corresponding to each iteration training, and adjusts the parameters of the initial face transformation model based on the total training loss of each iteration training. For example, optimizes parameters included in the encoder, decoder, generator, discriminator, etc. in the initial face transformation model until the total training loss meets a target condition. When the target condition is met, the computer device can stop training and use the initial face transformation model obtained by the last optimization as the face transformation model. For example, the computer device can use an Adam algorithm optimizer to perform iterative training on the initial face transformation model with a learning rate of 0.0001 until the target condition is reached, and when the target condition is reached, the training is deemed to have reached convergence and the training is stopped. For example, the target condition may be that the numerical value of the total loss is within a target numerical range, for example, that the total loss is less than 0.5, or the target condition may be that the time spent on multiple iterations of training exceeds a maximum time length.

[0102] 3 is a schematic framework diagram of a face swap model according to an embodiment of the present application. The computer device converts the face image of target A into a source image X s Let the face image of subject B be the target image X t The computing device can use a fixed face recognition network (Fixed FR Net) to detect the identity features f of the source image. id and the computing device acquires the identity feature f id The computer device inputs the initial attribute features f1 of at least one scale of the target image through an encoder and a decoder of a U-type deep network structure. att , f2 att , ..., f i att , ..., f N-1 att , f N attThe computer device executes the above steps S1 to S4 for each block until it obtains a feature map output by the last block, and the computer device calculates a final target face-swapped image Y s,t , thereby completing the face swap.

[0103] It should be noted that the image processing method of the present application can realize high-resolution face conversion, for example, 1024 2 At the same time, the generated high-resolution face transformation image can achieve both relatively high image quality and consistency with the identity of the source face in the source image, and effectively retain the key attributes of the target face in the target image with high accuracy. 2 In the image processing method of the present application, the initial attribute features and identity features of at least one scale are processed in each convolution layer of the generator, and the control mask of at least one scale is used to select the initial attribute features, so that redundant information such as the target face identity features is effectively filtered out from the obtained target attribute features, and the key attribute features of the target face are effectively reserved. And the initial attribute features of at least one scale correspond prominently to the features of different scales, and the initial attribute features of a relatively large scale correspond to the control mask of a relatively large scale, so that the key attributes can be more clearly selected, thereby accurately retaining the facial details such as hair, wrinkles, and facial occlusion of the target face, and greatly improving the accuracy and clarity of the generated face-swap image, and improving the authenticity of the face-swap image.

[0104] In addition, the image processing method of the present application can directly generate an entire face-swapped image after face swapping, which includes the face after face swapping and the background area, and does not require processing such as fusion or reinforcement in related technologies, which greatly improves the processing efficiency of the face swapping process.

[0105] In addition, the face swap model training method of the present application can perform end-to-end training on the entire generation framework for generating sample generated images in the initial face swap model during model training, and by avoiding the accumulation of errors due to multi-stage training, the face swap model trained by the present application can more stably generate face swap images and improve the stability and reliability of the face swap process.

[0106] In addition, the image processing method of the present application can generate a face-swap image with higher resolution, and can accurately retain details such as texture, skin brightness, and hair of the target face in the target image, thereby improving the accuracy, clarity, and realism of the face swap, and can be applied to scenarios with higher requirements for the quality of face swap, such as games, movies, and television. And for avatar maintenance scenarios, the image processing method of the present application can realize face swapping in which an arbitrary target face is replaced with an arbitrary target face, and for a specific avatar, the face of the specific avatar is replaced with the facial image of an arbitrary target, which makes avatar maintenance easier and improves the convenience of avatar maintenance.

[0107] The following is a comparison of the face swapping results using the image processing method of the present application with the face swapping results of related techniques. As can be seen from the comparison, the high-resolution face swapping results generated by the image processing method of the present application have obvious advantages over the related techniques in both qualitative and quantitative comparison.

[0108] As shown in Fig. 7, Fig. 7 shows a comparison between some methods in the related art (hereinafter referred to as Method A) and the high-resolution face swap results of the scheme proposed in this application. As can be seen from the comparison, Method A generates obvious skin luminance mismatch problems and cannot preserve facial hair occlusion. The results generated by the scheme proposed in this application preserve the attribute features of the target face, such as skin luminance, facial expression, skin texture, occlusion, etc., and have better image quality and are more realistic.

[0109] The following Table 1 shows a quantitative comparison between the high-resolution face swap result of the method A in the related art and the scheme proposed in the present application. The experimental data in Table 1 compares the identity similarity (ID Retrieval) between the face in the generated face swap image and the face in the source image, the pose difference (Pose Error) between the face in the face swap image and the face in the target image, and the picture quality difference (FID) between the face in the face swap image and the real face image. As can be seen from the experimental data in Table 1, the identity similarity of the high-resolution face swap result of the scheme proposed in the present application is obviously higher than that of the method A in the related art. The pose difference of the high-resolution face swap result of the scheme proposed in the present application is lower than that of the method A in the related art, and the pose difference of the scheme in the present application is lower. The picture quality difference of the high-resolution face swap result of the scheme proposed in the present application is obviously lower than that of the method A in the related art, and the picture quality difference between the face swap image obtained by the scheme in the present application and the real image is small. Therefore, the scheme proposed in this application balances image quality, identity consistency with the source face, and attribute retention for the target face, and has a significant advantage over Method A in the related art.

[0110] [Table 1]

[0111] In the image processing method of the embodiment of the present application, the identity features of a source image and the initial attribute features of at least one scale of a target image are obtained, the identity features are input into a generator in a trained face swap model, and the initial attribute features of the at least one scale are respectively input into a convolution layer of a corresponding scale in the generator to obtain a target face swap image. In each convolution layer of the generator, a second feature map is generated according to the identity features and the first feature map output by the previous convolution layer, and a control mask of the corresponding scale of the target image is determined according to the second feature map and the initial attribute features, so that the pixel points carrying the features other than the identity features of the target face in the target image can be accurately located. Based on the control mask, a target attribute feature in the initial attribute feature is selected, and based on the target attribute feature and the second feature map, a third feature map is generated and output to the next convolution layer, and through the layer-by-layer processing of at least one convolution layer, the attributes and detailed features of the target face are effectively retained in the final target face-swap image, the clarity of the face in the face-swap image is greatly improved, high-resolution face swapping is realized, and the accuracy of face swapping is improved.

[0112] 8 is a structural schematic diagram of an image processing device according to an embodiment of the present application. As shown in FIG. 8, the image processing device includes: a feature acquisition module 801 configured to acquire identity features of a source image and initial attribute features of at least one scale of a target image in response to a received face swap request, the face swap request being used to request replacing a target face in the target image with a source face in the source image, the identity features representing an object to which the source face belongs and the initial attribute features representing three-dimensional attributes of the target face; inputting the identity features and the at least one scale initial attribute features into a face swap model in a face swap module; Iteratively performing feature fusion on the identity features and the at least one scale initial attribute features through the face swap model to obtain a fusion feature; generating a target face-swapped image by the face swap model based on the fusion features, and outputting the target face-swapped image, wherein a face in the target face-swapped image is a fusion of identity features of the source face and target attribute features of the target face.

[0113] In some embodiments, the face swap model includes at least one convolutional layer, each of the convolutional layers corresponding to one of the scales, and the convolutional layer of the face swap module 802 includes an acquisition unit, a generation unit, and an attribute selection unit; The acquisition unit is configured to acquire a first feature map output by a convolutional layer previous to a current convolutional layer; a generating unit configured to generate a second feature map based on the identity features and the first feature map; The attribute selection unit is configured to select a target attribute feature from the initial attribute features of the at least one scale, and the target attribute feature is a feature other than an identity feature of the target face; The generating unit further generates a third feature map according to the target attribute feature and the second feature map, and inputs the third feature map into a next convolution layer of the current convolution layer as a first feature map of the next convolution layer; The third feature map output by a last convolutional layer of the at least one convolutional layer is configured to be determined as the fusion feature.

[0114] In some embodiments, the convolutional layer of the face swap module 802 includes: and a control mask determining unit configured to determine a control mask of the target image at a corresponding scale based on the second feature map and the initial attribute features; the control mask is used to represent pixel points carrying features other than the identity features of the target face; The generating unit is further configured to filter initial attribute features of the at least one scale based on the control mask to obtain target attribute features.

[0115] In some embodiments, the control mask determination unit further comprises: Perform feature linking on the second feature map and the initial attribute feature to obtain a linked feature map; The method is configured to map the connected feature map to the control mask based on a pre-defined mapping convolution kernel and activation function.

[0116] In some embodiments, the number of the initial attribute features and the number of the convolutional layers are both a target number, the target number of convolutional layers are connected in series, different initial attribute features correspond to different scales, each of the convolutional layers corresponds to an initial attribute feature of one of the scales, and the target number is 2 or more; The obtaining unit is further configured to obtain an initial feature map when the current convolutional layer is a first convolutional layer of the target number of convolutional layers, and use the initial feature map as a first feature map input to the current convolutional layer.

[0117] In some embodiments, the generation unit is further configured to perform an affine transformation on the identity features to obtain a first control vector, map a first convolution kernel of the current convolution layer to a second convolution kernel based on the first control vector, and perform a convolution operation on the first feature map based on the second convolution kernel to generate a second feature map.

[0118] In some embodiments, the image processor, when training the face swap model, comprises: a sample acquisition module configured to acquire a sample dataset, the sample dataset including at least one sample image pair, each sample image pair including one sample source image and one sample target image; a sample feature acquisition module configured to acquire sample identity features of a sample source image in a sample image pair and sample initial attribute features of at least one scale of a sample target image in the sample image pair; a generation module configured to iteratively perform feature fusion on the sample identity features and the sample initial attribute features at the at least one scale by a generator of the initial face swap model to obtain sample fusion features, and generate a sample generated image by the generator of the initial face swap model based on the sample fusion features; A discrimination module configured to discriminate the sample generated image and the sample source image by a discriminator of the initial face transformation model to obtain a discrimination result; a loss determination module configured to determine a loss of the initial face transformation model based on the discrimination result; and a training module configured to train the initial face transformation model based on the loss to obtain the face transformation model.

[0119] In some embodiments, the discrimination result comprises a first discrimination result for the sample source image and a second discrimination result for the sample generated image; The loss determination module is further configured to obtain a sample mask of at least one scale of the sample target image in each sample image pair, determine a first loss value based on the sample mask of the at least one scale, and determine a second loss value based on the first discrimination result and the second discrimination result; The training module is further configured to train the initial face transformation model based on the total training loss until a target condition is met, and when the target condition is met, stop training to obtain the face transformation model.

[0120] In some embodiments, the sample source image and the sample target image correspond to the same object; The loss determination module is further configured to obtain a third loss value based on the sample generated images and the sample target images, and obtain the training total loss based on the third loss value, the first loss value, and the second loss value.

[0121] In some embodiments, the classifier includes at least one convolutional layer, and the loss determination module further comprises: determining, for each sample image pair, a first similarity between non-face regions of a first discriminant feature map and non-face regions of a second discriminant feature map, the first discriminant feature map being a feature map of a sample target image output by a first partial convolutional layer of the at least one convolutional layer, and the second discriminant feature map being a feature map of a sample generated image output by the first partial convolutional layer; determining a second similarity between a third discriminant feature map and a fourth discriminant feature map, the third discriminant feature map being a feature map of a sample target image output by a second partial convolutional layer of the at least one convolutional layer, and the fourth discriminant feature map being a feature map of a sample generated image output by the second partial convolutional layer; determining a fourth loss value based on the first similarity measure and the second similarity measure; obtaining the training total loss based on the first loss value, the second loss value, and the fourth loss value.

[0122] In some embodiments, the loss determination module further comprises: extracting, for each sample image pair, a first identity feature of the sample source image, a second identity feature of the sample target image, and a third identity feature of the sample generated image; determining a first identity similarity between the sample source image and the sample generated image based on the first identity feature and the third identity feature; determining a first identity distance between the sample generated image and the sample target image based on the second identity feature and the third identity feature; determining a second identity distance between the sample source image and the sample target image based on the first identity feature and the second identity feature; determining a distance difference based on the first identity distance and the second identity distance; determining a fifth loss value corresponding to each sample image pair based on the first identity similarity and distance difference corresponding to each sample image pair; obtaining the training total loss based on the first loss value, the second loss value, and the fifth loss value.

[0123] In the image processing device of the embodiment of the present application, the identity features of a source image and the initial attribute features of at least one scale of a target image are obtained, the identity features are input into a generator in a trained face swap model, and the initial attribute features of the at least one scale are respectively input into a convolution layer of a corresponding scale in the generator to obtain a target face swap image. In each convolution layer of the generator, a second feature map is generated according to the identity features and the first feature map output by the previous convolution layer, and a control mask of the corresponding scale of the target image is determined according to the second feature map and the initial attribute features, so that the pixel points carrying the features other than the identity features of the target face in the target image can be accurately located. Based on the control mask, a target attribute feature in the initial attribute feature is selected, and based on the target attribute feature and the second feature map, a third feature map is generated and output to the next convolution layer, and through the layer-by-layer processing of at least one convolution layer, the attributes and detailed features of the target face are effectively retained in the final target face-swap image, the clarity of the face in the face-swap image is greatly improved, high-resolution face swapping is realized, and the accuracy of face swapping is improved.

[0124] Fig. 9 is a structural schematic diagram of a computer device according to an embodiment of the present application. As shown in Fig. 9, the computer device includes a memory and a processor. The memory stores a computer program. The processor executes the computer program stored in the memory to realize the image processing method provided in the embodiment of the present application.

[0125] In some embodiments, a computer device is provided. As shown in FIG. 9, the computer device 900 includes a processor 901 and a memory 903. The processor 901 is connected to the memory 903 by, for example, a bus 902. For example, the computer device 900 can further include a transceiver 904, which can be used for data interaction between the computer device and other computer devices, such as data transmission and / or data reception. It should be noted that in practical application, the transceiver 904 is not limited to one, and the structure of the computer device 900 does not constitute a limitation to the embodiments of the present application.

[0126] The processor 901 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in combination with the disclosed contents of this application. The processor 901 may also be a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc., to implement a computational function.

[0127] The bus 902 may include one path for transmitting information between the aforementioned components. The bus 902 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 902 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 9, but this does not represent only one bus or one type of bus.

[0128] The memory 903 may be, without limitation, a read only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read only memory (EEPROM), a compact disc read only memory (CD-ROM) or other optical disc memory, optical disc storage (including compressed discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disc storage medium or other magnetic storage device, or any other medium that can be used to carry or store a computer program and that can be read by a computer.

[0129] The memory 903 is used for storing a computer program for executing the embodiments of the present application, the execution of which is controlled by the processor 901. The processor 901 executes the computer program stored in the memory 903 to realize the steps shown in the above-mentioned method embodiments.

[0130] The computer equipment includes, but is not limited to, a server, a terminal, or a cloud computing center device.

[0131] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps and corresponding contents of the above-mentioned method embodiments can be realized.

[0132] The embodiments of the present application further provide a computer program product, including a computer program, and when the computer program is executed by a processor, the steps and corresponding contents of the above-mentioned method embodiments may be realized.

[0133] Terms such as "first," "second," "third," "fourth," "one," "two," and the like (when present) in the specification and claims of this application, as well as in the drawings above, are used to distinguish between similar subject matter, and not necessarily to describe a particular order or chronology. Data used in this manner are interchangeable where appropriate, so that it should be understood that the embodiments of the application described herein may be implemented in orders other than those illustrated or described in text.

[0134] It should be noted that the above are only optional embodiments of some implementation scenarios of the present application, and those skilled in the art should also note that adopting other similar implementation means based on the technical ideas of the present application without departing from the technical concept of the solution of the present application also falls within the protection scope of the embodiments of the present application. [Explanation of symbols]

[0135] 11 Server 12 Terminal 801 Feature Acquisition Module 802 Face Swap Module 900 Computer Equipment 901 Processor 902 Bus 903 Memory 904 Transceiver

Claims

1. 1. A computing device implemented image processing method, comprising: obtaining sample identity features of a sample source image in a sample image pair and sample initial attribute features of at least one scale of a sample target image in the sample image pair; performing feature fusion iteratively on the sample identity features and the sample initial attribute features of the at least one scale by a generator of an initial face swap model to obtain sample fusion features; generating a sample generated image by a generator of the initial face swap model based on the sample fusion features; using a classifier of the initial face swap model to classify the sample generated image and the sample source image to obtain a classification result; determining a loss for the initial face swap model based on the discrimination result, and training the initial face swap model based on the loss to obtain a face swap model; obtaining identity features of a source image and initial attribute features of at least one scale of a target image in response to a received face swap request, the face swap request being used to request replacement of a target face in the target image with a source face in the source image, the identity features representing an object to which the source face belongs and the initial attribute features representing three-dimensional attributes of the target face; inputting the identity features and the at least one scale initial attribute features into the face swap model; Iteratively performing feature fusion on the identity features and the at least one scale initial attribute features through the face swap model to obtain a fusion feature; generating a target face-swap image using the face-swap model based on the fusion features, and outputting the target face-swap image, wherein a face in the target face-swap image is a fusion of identity features of the source face and target attribute features of the target face.

2. The face swap model includes at least one convolution layer, each of the convolution layers corresponding to one of the scales, and the step of iteratively performing feature fusion on the identity features and the initial attribute features of the at least one scale by the face swap model to obtain a fusion feature includes: Each convolutional layer of the face swap model respectively calculates the identity features and the initial attribute features of the corresponding scale as follows: Obtaining a first feature map output by a convolutional layer previous to a current convolutional layer; generating a second feature map based on the identity features and the first feature map, and selecting target attribute features from the initial attribute features of the at least one scale, the target attribute features being features other than the identity features of the target face; generating a third feature map based on the target attribute feature and the second feature map, the third feature map being a first feature map of a next convolutional layer of the current convolutional layer; determining a third feature map output by a last convolutional layer of the at least one convolutional layer as the fusion feature. The image processing method according to claim 1 .

3. The step of selecting a target attribute feature from the initial attribute features of the at least one scale includes: determining a control mask of the target image at a corresponding scale based on the second feature map and the initial attribute features, the control mask being used to represent pixel points carrying features other than identity features of the target face; and selecting initial attribute features of the at least one scale based on the control mask to obtain target attribute features. The image processing method according to claim 2 .

4. determining a control mask of the target image at a corresponding scale based on the second feature map and the initial attribute features, performing feature concatenation on the second feature map and the initial attribute features to obtain a concatenated feature map; and mapping the connected feature map to the control mask based on a predefined mapping convolution kernel and activation function. The image processing method according to claim 3.

5. The number of the initial attribute features and the number of the convolution layers are both a target number, the target number of convolution layers are connected in series, different initial attribute features correspond to different scales, each of the convolution layers corresponds to an initial attribute feature of one of the scales, and the target number is 2 or more; The step of obtaining a first feature map output by a convolutional layer previous to the current convolutional layer includes: If the current convolutional layer is a first convolutional layer of the target number of convolutional layers, obtaining an initial feature map, and using the initial feature map as a first feature map input to the current convolutional layer. The image processing method according to claim 2 .

6. generating the second feature map based on the identity features and the first feature map, performing an affine transformation on the identity feature to obtain a first control vector; mapping a first convolution kernel of the current convolution layer to a second convolution kernel based on the first control vector; and performing a convolution operation on the first feature map based on the second convolution kernel to generate a second feature map. The image processing method according to claim 2 .

7. The discrimination result includes a first discrimination result for the sample source image and a second discrimination result for the sample generated image, and the step of determining a loss of the initial face swap model based on the discrimination result includes: obtaining a sample mask of at least one scale of the sample target image and determining a first loss value based on the sample mask of the at least one scale; determining a second loss value based on the first discrimination result and the second discrimination result; obtaining a training total loss based on the first loss value and the second loss value; training the initial face swap model based on the total training loss until a target condition is met, and stopping the training when the target condition is met to obtain the face swap model. The image processing method according to claim 1 .

8. the sampled source image and the sampled target image correspond to the same object; obtaining a training total loss based on the first loss value and the second loss value, obtaining a third loss value based on the sample generated image and the sample target image; and obtaining the training total loss based on the third loss value, the first loss value, and the second loss value. The image processing method according to claim 7.

9. The classifier includes at least one convolutional layer, and the step of obtaining a training total loss based on the first loss value and the second loss value includes: determining a first similarity between non-face regions of a first discriminant feature map and non-face regions of a second discriminant feature map, the first discriminant feature map being a feature map of a sample target image output by a first partial convolutional layer of the convolutional layer, and the second discriminant feature map being a feature map of a sample generated image output by the first partial convolutional layer; determining a second similarity between a third discriminant feature map and a fourth discriminant feature map, the third discriminant feature map being a feature map of a sample target image output by a second partial convolutional layer of the convolutional layer, and the fourth discriminant feature map being a feature map of a sample generated image output by the second partial convolutional layer; determining a fourth loss value based on the first similarity measure and the second similarity measure; and obtaining the training total loss based on the first loss value, the second loss value, and the fourth loss value. The image processing method according to claim 7.

10. obtaining a training total loss based on the first loss value and the second loss value, extracting a first identity feature of the sample source image, a second identity feature of the sample target image, and a third identity feature of the sample generated image; determining a first identity similarity between the sample source image and the sample generated image based on the first identity feature and the third identity feature; determining a first identity distance between the sample generated image and the sample target image based on the second identity feature and the third identity feature; determining a second identity distance between the sample source image and the sample target image based on the first identity feature and the second identity feature; determining a distance difference based on the first identity distance and the second identity distance; determining a fifth loss value based on the first identity similarity and the distance difference; and obtaining the training total loss based on the first loss value, the second loss value, and the fifth loss value. The image processing method according to claim 7.

11. An image processing device comprising: a feature acquisition module and a face swap module, The feature acquisition module includes: configured to obtain identity features of a source image and initial attribute features of at least one scale of a target image in response to a received face swap request, the face swap request being used to request replacement of a target face in the target image with a source face in the source image, the identity features representing an object to which the source face belongs and the initial attribute features representing three-dimensional attributes of the target face; The face swap module includes: inputting the identity features and the at least one scale initial attribute features into a face swap model in a face swap module; Iteratively performing feature fusion on the identity features and the at least one scale initial attribute features through the face swap model to obtain a fusion feature; generating a target face-swap image by the face-swap model based on the fusion features, and outputting the target face-swap image, wherein a face in the target face-swap image is a fusion of identity features of the source face and target attribute features of the target face; The face swap model is obtaining sample identity features of a sample source image in the sample image pair and sample initial attribute features of at least one scale of a sample target image in the sample image pair; performing feature fusion iteratively on the sample identity features and the sample initial attribute features of the at least one scale by a generator of an initial face swap model to obtain sample fusion features; generating a sample generated image by a generator of the initial face swap model based on the sample fusion features; Using a classifier of the initial face swap model, the sample generated image and the sample source image are classified, and a classification result is obtained; The image processing device obtains the loss of the initial face swap model based on the discrimination result, and trains the initial face swap model based on the loss.

12. A computing device comprising a memory and a processor, The memory stores a computer program; A computing device, the processor executing a computer program stored in the memory to implement the image processing method according to any one of claims 1 to 10.

13. A computer readable storage medium storing a computer program for causing a processor to execute the image processing method according to any one of claims 1 to 10.

14. A computer program product that causes a computer to execute the image processing method according to any one of claims 1 to 10.

Images