Image processing methods and equipment, computer hardware, storage media, and software products.

Abstract

This request will provide methods and sets of image processing equipment, computer equipment, storage media, and Products and software related to fields of arts and sciences such as artificial intelligence and learning. Intelligent machinery and transportation are being used to respond to requests for changes to the featured section. The unique characteristics of the source image and its prominent features indicate initial features with at least one scale. For example, the target image will be fed into the function change model where the change request is made. The page will be used to request that the target page in the target image be changed to the source page in the image. Origin: And the fusion of distinctive elements will be performed in a repetitive manner with the defining characteristics. And the highlighted part describes the initial characteristics that have at least one scale in order to obtain the subsequent highlighted part. The fusion process uses a page transformation model; the target page transformation image will be... The creation of the design is based on post-merger prominence using page and image transformation models. Changes to the target page will export the page in the image; changes to the target page will be merged. Combined with key features that define the identity of the source page and key features that describe the target characteristics. Target page;

Description

Image processing method, device, computer equipment, storage medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202210626467.1 and application date of June 2, 2022, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to technical fields such as artificial intelligence, machine learning, and smart transportation, and in particular to an image processing method, apparatus, computer equipment, storage medium, and program product. Background Art

[0004] Face swapping is an important technology in the field of computer vision and is widely used in content production, film and television portrait production, entertainment video production, virtual image creation, and privacy protection. Face swapping involves replacing the face of a subject in an image with another face.

[0005] Related art often uses neural network models to achieve face swapping. For example, an image is input into the neural network model used for face swapping, and the neural network model outputs the resulting face-swapped image. However, the images obtained by face swapping techniques in related art differ significantly from the ideal face-swapped image, resulting in poor face-swapped results.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product, which can improve the quality of images after face swapping.

[0008] The present invention provides an image processing method, which includes:

[0009] In response to the received face-swapping request, obtaining identity features of the source image and initial attribute features of at least one scale of the target image;

[0010] The face-swap request is used to request that a target face in the target image be replaced with a source face in the source image, the identity feature represents the object to which the source face belongs, and the initial attribute feature represents a three-dimensional attribute of the target face;

[0011] Inputting the identity feature and the initial attribute feature of the at least one scale into a face-changing model;

[0012] Iteratively fusing the identity features and the initial attribute features of the at least one scale using the face-changing model to obtain fused features;

[0013] Based on the fusion features, generating a target face-swapped image through the face-swapped model, and outputting the target face-swapped image;

[0014] The face in the target face-swapped image is a fusion of the identity features of the source face and the target attribute features of the target face.

[0015] The present application also provides an image processing device, comprising:

[0016] a feature acquisition module configured to acquire, in response to a received face-swapping request, an identity feature of a source image and an initial attribute feature of at least one scale of a target image;

[0017] The face-swap request is used to request that a target face in the target image be replaced with a source face in the source image, the identity feature represents the object to which the source face belongs, and the initial attribute feature represents a three-dimensional attribute of the target face;

[0018] a face-changing module configured to input the identity feature and the initial attribute feature of the at least one scale into a face-changing model in the face-changing module;

[0019] Iteratively fusing the identity features and the initial attribute features of the at least one scale using the face-changing model to obtain fused features;

[0020] Based on the fusion features, generating a target face-swapped image through the face-swapped model, and outputting the target face-swapped image;

[0021] The face in the target face-swapped image is a fusion of the identity features of the source face and the target attribute features of the target face.

[0022] An embodiment of the present application further provides a computer device, comprising a memory and a processor; wherein the memory is configured to store a computer program;

[0023] The processor is configured to execute the computer program stored in the memory to implement the image processing method provided in the embodiment of the present application.

[0024] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the image processing method provided by the embodiment of the present application is implemented.

[0025] An embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method provided in the embodiment of the present application.

[0026] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0027] The image processing method of the embodiment of the present application inputs the identity features of the source image and the initial attribute features of the target image into a face-swapping model, and then iteratively fuses the identity features and the initial attribute features of at least one scale through the face-swapping model to obtain a fused feature. That is, at the input end of the face-swapping model, the identity features and the attribute features are explicitly decoupled, so that the obtained fused feature fuses the identity features of the object in the source image and the three-dimensional attributes of the face of the object in the target image.

[0028] Based on the fusion features, a target face-swapped image is generated through a face-swapped model, and the target face-swapped image is output; wherein, the face in the target face-swapped image is fused with the identity features of the source face and the target attribute features of the target face; in this way, the target face-swapped image is generated based on the fusion features obtained by feature fusion, and on the basis of ensuring the identity consistency of the face in the target face-swapped image and the face in the source image, the attributes and detail features of the target face in the target face-swapped image are effectively retained, thereby greatly improving the clarity, accuracy and authenticity of the face in the face-swapped image, and realizing high-definition face swapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a schematic diagram of an implementation environment of an image processing method provided in an embodiment of the present application;

[0030] FIG2 is a schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0031] FIG3 is a schematic diagram of the structure of a face-changing model provided in an embodiment of the present application;

[0032] FIG4 is a schematic diagram of the structure of a block in a generator provided in an embodiment of the present application;

[0033] FIG5 is a flow chart of a face-swapping model training method provided in an embodiment of the present application;

[0034] FIG6 is a schematic diagram of a control mask of at least one dimension provided in an embodiment of the present application;

[0035] FIG7 is a schematic diagram showing a comparison of face-swapping results provided in an embodiment of the present application;

[0036] FIG8 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0037] FIG9 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0039] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] Those skilled in the art will understand that, unless otherwise specified, the singular forms "a," "an," "the," and "the" used herein may also include the plural forms. The terms "including" and "comprising" used in the embodiments of this application mean that the corresponding features can be implemented as the presented features, information, data, steps, and operations, but do not exclude the implementation of other features, information, data, steps, operations, etc. supported by the technical field.

[0041] It is understandable that in the specific implementation of the present application, any object-related data such as the source image, target image, source face, target face, and at least one pair of samples in the sample data set used for model training, as well as any object-related data such as the image to be replaced, facial features of the target face, attribute parameters, etc. used when performing face replacement using the face replacement model, are all obtained after the consent or permission of the relevant object; when the following embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. In addition, the face replacement process of the facial image of any object using the image processing method of the present application is based on the face replacement service or face replacement request triggered by the relevant object, and is performed after the permission or consent of the relevant object.

[0042] The image processing methods provided in the embodiments of this application involve the following technologies: artificial intelligence, computer vision, and others. For example, they utilize cloud computing and big data processing within artificial intelligence to implement processes such as face-swapping model training and extracting multi-scale attribute features from images. For example, computer vision technology is used to perform facial recognition on an image to obtain the identity features corresponding to the face in the image.

[0043] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also encompasses the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0044] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0045] It should be understood that computer vision (CV) is the science of making machines "see." It refers to using cameras and computers to replace the human eye in identifying and measuring objects, and performing image processing to transform the images into images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, and smart transportation. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0046] FIG1 is a schematic diagram of an implementation environment of an image processing method provided by the present application. As shown in FIG1 , the implementation environment includes: a server 11 and a terminal 12 .

[0047] The server 11 is configured with a trained face-swapping model and can provide a face-swapping function to the terminal 12 based on the model. This function can be used to generate a face-swapping image based on a source image and a target image. The generated face-swapping image incorporates the identity features of the source face in the source image and the attribute features of the target face in the template image. The identity features represent the object to which the source face belongs, and the initial attribute features represent the three-dimensional attributes of the target face.

[0048] In some embodiments, the terminal 12 is installed with an application, which may be pre-configured with a face-changing function, and the server 11 may be the background server of the application. The terminal 12 and the server 11 may exchange data based on the application to implement the face-changing process. Exemplarily, the terminal 12 may send a face-changing request to the server 11, and the face-changing request is used to request that the target face in the target image be replaced with the source face in the source image. Based on the face-changing request, the server 11 may execute the image processing method of the present application to generate a target face-changing image, and return the target face-changing image to the terminal 12. For example, the application is any application that supports the face-changing function, for example, the application includes but is not limited to: video editing applications, image processing tools, video applications, live broadcast applications, social applications, content interaction platforms, game applications, etc.

[0049] The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The aforementioned networks may include, but are not limited to, wired networks and wireless networks. Wired networks include local area networks, metropolitan area networks, and wide area networks, and wireless networks include Bluetooth, Wi-Fi, and other networks that enable wireless communication. Terminals may include smartphones (such as Android phones and iOS phones), tablets, laptops, digital broadcast receivers, MIDs (Mobile Internet Devices), PDAs (Personal Digital Assistants), desktop computers, in-vehicle terminals (such as in-vehicle navigation terminals, in-vehicle computers, etc.), smart appliances, aircraft, smart speakers, smart watches, etc. The terminals and servers may be connected directly or indirectly via wired or wireless communications, but are not limited thereto. The specific configuration may also be determined based on the actual application scenario requirements and is not limited here.

[0050] In order to make the objectives, technical solutions and advantages of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0051] The following is an introduction to the technical terms involved in this application:

[0052] Face swapping: swapping the face in an image with another face. For example, given a source image X s and the target image X t , using the image processing method of this application to generate a face-changing image Y s，tAmong them, the face-changing image Y s，t With the source image X s Identity features; while retaining the target image X t Attribute features that are not related to identity.

[0053] Face swapping model: used to swap the target face in the target image with the source face in the source image.

[0054] Source image: An image that provides identity features. The face in the generated face-swapped image has the identity features of the face in the source image.

[0055] Target image: An image that provides attributes and features. The face in the generated face-swapped image shares the attributes and features of the face in the target image. For example, if the source image is of subject A and the target image is of subject B, and subject B's face in the target image is replaced with subject A's face, the resulting face-swapped image will be the face of subject A. The face in the face-swapped image will share the same identity features as subject A, such as eye shape, distance between the eyes, and nose size. Furthermore, the face in the face-swapped image will share the same attributes and features as subject B's face, such as expression, hair, lighting, wrinkles, posture, and facial occlusion.

[0056] FIG2 is a flow chart of an image processing method provided by an embodiment of the present application. The execution subject of the method may be a computer device (which may be a terminal or a server). As shown in FIG2 , the method includes the following steps.

[0057] Step 201: In response to a received face-changing request, a computer device obtains identity features of a source image and initial attribute features of at least one scale of a target image.

[0058] The face-swap request is used to request that the target face in the target image be replaced with the source face in the source image. In practical applications, the face-swap request includes a source image and a target image, and the computer device parses the face-swap request to obtain the source image and the target image. Alternatively, the face-swap request includes an identifier of the source image and an identifier of the target image, and the computer device parses the face-swap request to obtain the identifier of the source image and the identifier of the target image, and then searches the image library for the source image and the target image based on the identifiers.

[0059] The computer device can utilize a trained face-swapping model to obtain a face-swapping image, thereby providing a face-swapping function. The identity feature characterizes the object to which the source face belongs. Exemplarily, the identity feature can be a feature that identifies the identity of the object. The identity feature can include at least one of the facial features or contour features of the target face. The facial features of the target face refer to features corresponding to the facial features, and the contour features of the target face refer to features corresponding to the contour of the target face. For example, identity features may include, but are not limited to, eye shape, interocular distance, nose size, eyebrow shape, and facial contour. The initial attribute features characterize the three-dimensional attributes of the target face. Exemplarily, the initial attribute features can characterize attributes such as the posture and spatial environment of the target face in three-dimensional space. For example, the initial attribute features can include, but are not limited to, background, lighting, wrinkles, posture, expression, hair, and facial occlusion.

[0060] In some embodiments, the face-swapping model may include an identity recognition network. The computer device may input a source image into the face-swapping model, and the identity recognition network in the face-swapping model may perform facial recognition on the source image to obtain identity features of the source image. Exemplarily, the identity recognition network is configured to identify the identity of the face in the input image based on the input image. For example, the identity recognition network may be a fixed face recognition network (FR Net) in the face-swapping model. For example, when the source image is a face, the identity recognition network may be a trained face recognition model that is configured to identify the object to which the face in the source image belongs and obtain identity features for identifying the object. The identity features may include at least one of the following features: eye shape, interocular distance, nose size, eyebrow shape, and facial contour. The identity features may be a fixed-dimensional feature vector output by the face recognition model, such as a 512-dimensional feature vector. The 512-dimensional feature vector may represent features such as eye shape, interocular distance, nose size, eyebrow shape, and facial contour.

[0061] In some embodiments, the face-swapping model further includes an attribute feature extraction network, which may include an encoder and a decoder, wherein the encoder includes at least one encoding network layer (e.g., includes at least two encoding network layers), and the decoder includes at least one decoding network layer (e.g., includes at least two decoding network layers); for example, the attribute feature extraction network is a U-shaped deep network including an encoder and a decoder. In practical applications, the computer device may obtain the initial attribute features of at least one scale of the target image in the following manner:

[0062] The computer device downsamples the target image layer by layer through at least one encoding network layer of the encoder to obtain encoding features; and upsamples the encoding features layer by layer through at least one decoding network layer of the decoder to output decoding features of different scales, and uses the decoding features of different scales output by at least one decoding network layer as initial attribute features; wherein each decoding network layer corresponds to one of the scales.

[0063] Exemplarily, each encoding network layer is used to encode the target image to obtain the encoding features, and each decoding network layer is used to decode the encoding features to obtain the initial attribute features. During operation, the decoder can perform the reverse operation according to the operating principle of the encoder. For example, the encoder can downsample the target image, and the decoder can upsample the downsampled encoding features. For example, the encoder can be an autoencoder (AE), and the decoder can be a decoder corresponding to the autoencoder.

[0064] In some embodiments, each encoding network layer is used to downsample the encoding features output by the previous encoding network layer to obtain encoding features of at least one scale, and each encoding network layer corresponds to one scale; each decoding network layer is used to upsample the decoding features output by the previous decoding network layer to obtain initial attribute features of at least one scale, and each decoding network layer corresponds to one scale. The scales of the encoding network layer and the decoding network layer at the same layer can be the same; wherein, each decoding network layer can also upsample the initial attribute features output by the previous decoding network layer in combination with the encoding features of the encoding network layer of the corresponding scale. As shown in FIG3, FIG3 adopts a U-shaped deep network to upsample the target image X t For feature extraction, for example, the target image is input into the encoder, which includes multiple (ie at least two) encoding network layers, each encoding network layer corresponds to a resolution (ie scale) of the feature map, and the target image X is output through the multiple encoding network layers of the encoder. tThe resolutions of the feature maps of the encoded features are 1024×1024, 512×512, 256×256, 128×128, and 64×64, respectively. The 64×64 feature map is input into the first decoding network layer of the decoder for upsampling to obtain a 128×128 decoding feature map. The 128×128 decoding feature map is spliced ​​with the 128×128 encoding feature map, and the spliced ​​feature map is upsampled to obtain a 256×256 decoding feature map. By analogy, the feature maps of various resolutions obtained by decoding the network structure based on the U-shaped deep network are used as the initial attribute features. In the initial attribute features, the initial attribute features at each scale are used to characterize the attribute features of the target image at the corresponding scale. The attribute features corresponding to the initial attribute features of different scales may be different. The initial attribute features of a relatively small scale can characterize the global position, posture, and other information of the target face in the target image, while the initial attribute features of a relatively large scale can characterize the local details of the target face in the target image, thereby enabling the initial attribute features of at least one scale to encompass attribute features of multiple levels of the object. For example, the initial attribute features of at least one scale can be multiple feature maps with resolutions ranging from small to large, wherein the feature map of resolution R1 can characterize the facial position of the target face in the target image, the feature map of resolution R2 can characterize the posture and expression of the target face in the target image, and the feature map of resolution R3 can characterize the facial details of the facial position of the target face in the target image; wherein resolution R1 is smaller than R2, and R2 is smaller than R3.

[0065] Step 202: The computer device iteratively performs feature fusion on the identity features and the initial attribute features of at least one scale through the face-changing model to obtain fused features.

[0066] Step 203: The computer device generates a target face-swapped image through a face-swapped model based on the fusion features, and outputs the target face-swapped image.

[0067] Here, the face in the target face-swapped image fuses the identity features of the source face and the target attribute features of the target face.

[0068] In some embodiments, the face-swapping model includes a generator, the generator including at least one convolutional layer (e.g., including at least two convolutional layers), the at least one convolutional layer being connected in series, and each convolutional layer corresponding to a scale; the computer device may iteratively perform feature fusion on the identity feature and the initial attribute feature of at least one scale using the face-swapping model in the following manner to obtain a fused feature:

[0069] The computer device performs the following processing on the identity features and the initial attribute features of the corresponding scales through each convolutional layer of the face-changing model: obtaining a first feature map output by the convolutional layer previous to the current convolutional layer; generating a second feature map based on the identity features and the first feature map, and filtering out a target attribute feature from the initial attribute features of at least one scale, the target attribute feature being a feature other than the identity feature of the target face; generating a third feature map based on the target attribute feature and the second feature map, the third feature map being the first feature map of the convolutional layer next to the current convolutional layer;

[0070] The third feature map output by the last convolution layer in the at least one convolution layer is determined as a fusion feature.

[0071] In practical applications, the target number of initial attribute features and convolutional layers is used. The target number of convolutional layers is connected in series, with different initial attribute features corresponding to different scales. Each convolutional layer corresponds to an initial attribute feature of one scale, and the target number is greater than or equal to two. If the current convolutional layer is the first of the target number of convolutional layers, an initial feature map is obtained and used as the first feature map of the current convolutional layer. In practical applications, the initial feature map can be a fixed-dimensional all-zero feature vector.

[0072] In some embodiments, the computer device may filter out the target attribute features from the initial attribute features at at least one scale in the following manner: based on the feature map and the attribute features, determine a control mask of the image at the corresponding scale, where the control mask is used to represent pixel points that carry features other than the identity features of the target face; based on the control mask, filter the initial attribute features at at least one scale to obtain the target attribute features.

[0073] Exemplarily, the computer device may input the identity feature into each convolutional layer of the generator. The computer device inputs the initial attribute feature of at least one scale into a convolutional layer in the generator that matches the scale of the initial attribute feature, wherein the scales of the feature maps output by each convolutional layer of the generator are different, and the convolutional layer that matches the scale of the initial attribute feature means that the scale of the feature map to be output by the convolutional layer is the same as the scale of the initial attribute feature. For example, if a convolutional layer in the generator is used to process a 64×64 feature map from the previous convolutional layer and output a 128×128 feature map, then the 128×128 initial attribute feature can be input into the convolutional layer.

[0074] In some embodiments, in the generator, the computer device may determine a control mask at at least one scale for the target image based on the identity feature and the initial attribute feature at at least one scale, and generate the target face-swapped image based on the identity feature, the control mask at at least one scale, and the initial attribute feature. Exemplarily, the control mask represents pixels that carry features other than the identity feature of the target face. The computer device may then determine a target attribute feature at at least one scale based on the control mask at at least one scale and the initial attribute feature, and generate the target face-swapped image based on the identity feature and the target attribute feature at at least one scale.

[0075] The computer device can process the target face-swapped image through each convolution layer of the generator. In one possible example, the computer device performs the following steps S1 to S4 on the input identity features and the initial attribute features of the corresponding scale through each convolution layer of the generator:

[0076] Step S1: The computer device obtains a first feature map output by the previous convolutional layer of the current convolutional layer.

[0077] In the generator, each convolutional layer processes the feature map output by the previous convolutional layer and outputs it to the next convolutional layer. For the first convolutional layer, the computer device may input an initial feature map into the first convolutional layer. For example, this initial feature map may be a 4×4×512 feature vector of all zeros. For the last convolutional layer, the computer device may generate the final target face-swapped image based on the feature map output by the last convolutional layer.

[0078] Step S2: The computer device generates a second feature map based on the identity feature and the first feature map, and determines a control mask of the target image at a corresponding scale based on the second feature map and the initial attribute feature.

[0079] The control mask represents pixels that carry features other than the identity features of the target face.

[0080] In some embodiments, the computer device adjusts the weights of the convolution kernel of the current convolution layer based on the identity feature, and obtains the second feature map based on the first feature map and the adjusted convolution kernel. Exemplarily, the step of the computer device generating the second feature map may include: the computer device performs an affine transformation on the identity feature to obtain a first control vector; the computer device maps the first convolution kernel of the current convolution layer to a second convolution kernel based on the first control vector, and performs a convolution operation on the first feature map based on the second convolution kernel to generate a second feature map. Exemplarily, the identity feature can be represented in the form of an identity feature vector, and the affine transformation refers to an operation of linearly transforming and translating the identity feature vector to obtain the first control vector. The affine transformation operation includes but is not limited to translation, scaling, rotation, and flipping transformations. Each convolution layer of the generator includes a trained affine parameter matrix, and the computer device can perform translation, scaling, rotation, flipping, and other transformations on the identity feature vector based on the affine parameter matrix. Exemplarily, the computer device may perform a modulation operation (Mod) and a demodulation operation (Demod) on the first convolution layer of the current convolution layer using a first control vector to obtain a second convolution kernel. The modulation operation may be a scaling process on the convolution kernel weights of the current convolution layer, and the demodulation operation may be a normalization process on the scaled convolution kernel weights. For example, the computer device may scale the convolution kernel weights using a scaling ratio corresponding to a first feature map input to the current convolution layer and the first control vector.

[0081] In some embodiments, the computer device obtains a control mask of the corresponding scale based on the second feature map and the initial attribute features of the corresponding scale input to the current convolutional layer. This process may include: the computer device concatenating the second feature map and the initial attribute features to obtain a concatenated feature map; and mapping the concatenated feature map into the control mask based on a preconfigured mapping convolution kernel and activation function. Exemplarily, the control mask is a binary image in which pixels that carry features other than the target face's identity features, such as pixels in the hair region and background region, are assigned a value of 1, while pixels that carry identity features are assigned a value of 0. Exemplarily, the mapping convolution kernel may be a 1×1 convolution kernel, and the activation function may be a sigmoid function. For example, the second feature map and the initial attribute features may be represented as feature vectors. The computer device may merge the feature vector corresponding to the second feature map with the feature vector corresponding to the initial attribute features to obtain the concatenated vector, and perform convolution and activation operations on the concatenated vector to obtain the control mask.

[0082] Exemplarily, the generator may include multiple blocks, each block includes multiple layers, and the computer device inputs the identity features and the initial attribute features of each scale into the block of the corresponding scale. In the block, the input identity features and initial attribute features can be processed layer by layer by at least one layer. Exemplarily, FIG4 shows the network structure of the i-th block (i-th GAN block, i-th adversarial network block) in the generator, where N represents the attribute injection module (AttrInjection), and the internal structure of the attribute injection module is magnified in the dotted box on the right. As shown in FIG4, the i-th block includes two layers, where the first layer is used as an example for explanation. The left side of FIG4 represents the identity feature f of the source image. id , A represents the Affine Transform operation. After performing the affine transformation operation on the identity feature vector, the first control vector is obtained. In Figure 4, Mod and Demod represent the modulation and demodulation operations on the convolution kernel Conv 3×3. After the computer device performs an upsample operation on the first feature map of the current layer input to the current block, it uses the convolution kernel Conv 3×3 after the Mod and Demod operations to perform a convolution operation on the upsampled first feature map to obtain a second feature map. Then, the computer device combines the second feature map with the initial attribute feature f input to the current block. i att Perform concatenation (Concat) operation and use convolution kernel Conv 1×1 and Sigmoid function to map the concatenated feature vector to the control mask M corresponding to the current layer. i,j att .

[0083] Step S3: The computer device screens the initial attribute features based on the control mask to obtain target attribute features.

[0084] The computer device may perform a dot product between the feature vector corresponding to the control mask and the feature vector corresponding to the initial attribute feature to filter out the target attribute feature from the initial attribute feature.

[0085] As shown in FIG4 , the computer device can control the mask M i,j att With the initial attribute feature f i att Perform dot multiplication, and add the feature vector obtained by the dot multiplication to the feature vector corresponding to the second feature map to obtain the target attribute feature.

[0086] Step S4: The computer device generates a third feature map based on the target attribute feature and the second feature map, and outputs the third feature map to the next convolution layer of the current convolution layer as the first feature map of the next convolution layer.

[0087] The computer device may add the feature vector corresponding to the second feature map and the feature vector corresponding to the target attribute feature to obtain the third feature map.

[0088] It should be noted that for each convolutional layer included in the generator, the computer device can repeatedly execute steps S1 to S4 above until the last convolutional layer of the generator is repeated to obtain the third feature map output by the last convolutional layer, and generate the target face-swapped image based on the third feature map output by the last convolutional layer.

[0089] As shown in Figure 4, if the i-th block includes two layers, the third feature map can be input into the second layer of the i-th block, and the operation of the first layer is repeated, and the feature map obtained in the second layer is output to the next block, and this cycle is repeated until the last block. As shown in Figure 3, N in Figure 3 represents the attribute injection module (AttrInjection module), and the dotted box represents the generator using the StyleGAN2 model. For the N blocks included in the generator, the source image X is input respectively. s Identity characteristics of f id , and inject the corresponding initial attribute feature f1 into the attribute injection module att 、f2 att 、……、f i att 、……、f N-1 att 、f N att Input N blocks respectively, and execute the above steps S1 to S4 in each block until the features output by the last block are obtained. Based on the feature map output by the last block, the final target face-changing image Y is generated. s,t , thus completing the face change.

[0090] FIG5 is a flow chart of a face-swapping model training method provided in an embodiment of the present application. The method may be executed by a computer device. As shown in FIG5 , the method includes:

[0091] Step 501: A computer device obtains a sample identity feature of a sample source image in a sample image pair, and a sample initial attribute feature of at least one scale of a sample target image in the sample image pair.

[0092] In actual applications, a computer device obtains a sample data set, which includes at least one sample image pair, and uses the sample data set to train a face-swapping model. Each sample image pair includes a sample source image and a sample target image. In some embodiments, the sample image pair may include a first sample image pair and a second sample image pair, wherein the first sample image pair includes a sample source image and a sample target image belonging to the same object, and the second sample image pair includes a sample source image and a sample target image belonging to different objects. For example, the sample image pair includes a source image X of object A. s and a target image X t The first sample image pair also includes a source image X of object A s and a target image X of object B t The second sample image pair is composed of the first sample image pair and the second sample image is annotated with a true value label, which represents whether the corresponding source image and target image are the same object.

[0093] Here, the acquisition of the sample identity features of the sample source image and the sample initial attribute features of the sample target image can be achieved through the initial face-changing model. In some embodiments, the initial face-changing model may include an initial identity recognition network and an attribute feature map extraction network. The computer device may use the initial identity recognition network and the attribute feature map extraction network to respectively extract the sample identity features of the sample source image and the sample initial attribute features of at least one scale of the sample target image. It should be noted that the implementation method for obtaining the sample identity features and the sample initial attribute features here is similar to the method for obtaining the identity features and initial attribute features in step 201 above, and will not be repeated here.

[0094] Step 502: The computer device iteratively performs feature fusion on the sample identity features and the sample initial attribute features of at least one scale through the generator of the initial face-changing model to obtain a sample fusion feature, and generates a sample generated image based on the sample fusion feature through the generator of the initial face-changing model.

[0095] In some embodiments, the generator of the initial face-changing model determines a sample mask of at least one scale based on the sample identity feature of the sample source image and the sample initial attribute feature of at least one scale of the sample target image, and generates a sample generated image corresponding to the sample image pair based on the sample identity feature, the sample mask of at least one scale and the sample initial attribute feature.

[0096] The generator includes multiple convolutional layers. For each sample image pair, the computer device can input the sample identity feature into each convolutional layer, and input the sample initial attribute feature of at least one scale into the convolutional layer that matches the scale of the sample initial attribute feature. After layer-by-layer processing by each convolutional layer, the sample generated image is obtained.

[0097] Exemplarily, the computer device can perform the following steps on the input sample identity features and the sample initial attribute features of the corresponding scale through each convolution layer of the generator: the computer device obtains the first sample feature map output by the previous initial convolution layer of the current initial convolution layer; based on the sample identity features and the first sample feature map, a second sample feature map is generated, and based on the second sample feature map and the sample initial attribute features, a sample mask of the sample target image at the corresponding scale is determined; based on the sample mask, the computer device filters the sample initial attribute features to obtain sample target attribute features; based on the sample target attribute features and the second sample feature map, the computer device generates a third sample feature map, and outputs the third sample feature map to the next convolution layer of the current convolution layer as the first sample feature map of the next convolution layer. This cycle is repeated until the above steps are repeated on the last convolution layer of the generator to obtain the third feature map output by the last convolution layer, and the sample generated image is obtained based on the initial feature map output by the last convolution layer.

[0098] It should be noted that during the model training phase, the steps performed by each convolutional layer are the same as the steps performed by each convolutional layer in the generator of the trained face-changing model (that is, the above steps S1-S4), and will not be repeated here.

[0099] Step 503: The computer device uses the discriminator of the initial face-changing model to discriminate the sample generated image and the sample source image to obtain a discrimination result.

[0100] Here, the sample source image and the sample generated image in the sample image pair are input into the discriminator of the initial face-changing model, and the discriminator obtains the identification results of the sample source image and the sample generated image respectively.

[0101] The initial face-swapping model may further include a discriminator. For each sample image pair, the computer device inputs the sample source image and the sample generated image into the discriminator, which then outputs a first discrimination result for the sample source image and a second discrimination result for the sample generated image. The first discrimination result may represent the probability that the sample source image is a real image, and the second discrimination result may represent the probability that the sample generated image is a real image.

[0102] In some embodiments, the discriminator includes at least one convolutional layer; each convolutional layer can be used to process the discriminant feature map output by the previous convolutional layer of the discriminator and output it to the next convolutional layer of the discriminator. Each convolutional layer can output a discriminant feature map for feature extraction of the sample source image and a discriminant feature map for feature extraction of the sample generated image, until the last convolutional layer of the discriminator. Based on the discriminant feature map of the sample source image output by the last convolutional layer, a first discrimination result is obtained; and based on the discriminant feature map of the sample generated image output by the last convolutional layer, a second discrimination result is obtained.

[0103] Step 504: The computer device determines the loss of the initial face-changing model based on the identification result, and trains the initial face-changing model based on the loss to obtain a face-changing model.

[0104] For each sample image pair, the computer device determines a first loss value based on a sample mask of at least one scale of the sample target image in the sample image pair, and determines a second loss value based on the identification results of the discriminator on the sample source image and the sample generated image (i.e., the first identification result and the second identification result), and then obtains the total training loss based on the first loss value and the second loss value, and trains the initial face-changing model based on the total training loss until the training is stopped when the target conditions are met, thereby obtaining the face-changing model.

[0105] In practical applications, the computer device may accumulate the sample masks at at least one scale and use the accumulated value corresponding to the sample masks at at least one scale as the first loss value. For example, the sample mask may be a binary image. The computer device may accumulate the values ​​of each pixel in the binary image to obtain a first sum value corresponding to each sample mask, and then accumulate the first sum values ​​corresponding to the sample masks at at least one scale to obtain the first loss value.

[0106] Exemplarily, taking the example where the generator includes at least one initial block, and each initial block includes at least one layer, for each sample image pair, the computer device may determine the first loss value based on a sample mask of at least one scale of the sample target image in each sample image pair using the following formula 1:

[0107] Formula 1: L mask =∑ i，j |M i，j |1;

[0108] Among them, L mask Represents the first loss value, i represents the i-th block of the generator, and j represents the j-th layer of the i-th block. i，jThe computer device can accumulate the sample masks of at least one layer of at least one block by using the above formula 1, and in the training phase, minimize the first loss value L mask , in order to train the generator so that the control mask it obtains can effectively represent the pixel points of key attribute features other than identity features, and then the control mask can screen out the key attribute features in the initial attribute features, filter out the redundant features in the initial attribute features, and retain the key and necessary features in the initial attribute features, thereby avoiding redundant attributes and ultimately improving the accuracy of the generated face-changing image.

[0109] It should be noted that the degree of refinement of the pixels representing features other than the identity of the target face represented by binary images at different scales varies. Figure 6 shows sample masks of different scales corresponding to three target images. Each row of sample masks represents the sample masks of each scale corresponding to one of the target images. As shown in Figure 6, for any target image, the resolution of the sample masks increases from left to right. Taking the sample masks of each scale in the first row as an example, the resolution increases from 4×4, 8×8, 16×16, and 32×32, gradually clarifying the location of the face in the target image. The corresponding pixels in the face area are set to 0, while the corresponding pixels in the background area outside the face area are also set to 0. The resolution increases from 64×64, 128×128, 16×16, 256×256, 512×512, and 1024×1024, gradually clarifying the facial posture and expression in the target image, and gradually revealing the facial details in the target image.

[0110] Exemplarily, the computer device may determine the second loss value based on the identification results of the discriminator on the sample source image and the sample generated image respectively through the following formula 2:

[0111] Formula 2: L GAN =min G max D E[log(D(X s ))]+E[log(1-D(Y s，t ))];

[0112] Among them, L GAN Denotes the second loss value, D(X s ) represents the first identification result of the discriminator on the sample source image, and the first identification result can be the sample source image X s is the probability of a real image; D(Y s，t ) represents the discriminator generating image Y for the sample s，t The second identification result may be the probability that the sample generated image is a real image; E[log(D(X s))] refers to the log(D(X s )) can represent the loss value of the discriminator; E[log(1-D(Y s，t ))] refers to the log(1-D(Y s，t )) can represent the loss value of the generator. min G Indicates that the generator expects to minimize the loss function value, max D represents the discriminator's maximized loss function. It should be noted that the initial face-swapping model includes a generator and a discriminator, which can be an adversarial network. This adversarial network learns by pitting the generator and discriminator against each other to produce the desired machine learning model, a method of unsupervised learning. The generator's training goal is to obtain the desired output based on the input; the discriminator's training goal is to distinguish the generator's generated images from real images as closely as possible. The discriminator's input includes a sample source image and a sample generated image from the generator. The two network models learn against each other, continuously adjusting parameters. The ultimate goal is for the initiator to deceive the discriminator as much as possible, forcing the discriminator to determine whether the generator's images are authentic.

[0113] In some embodiments, the computer device may use the sum of the first loss value and the second loss value as the total training loss.

[0114] In some embodiments, the computer device may also perform training based on sample images of the same object. Before determining the total training loss, the computer device may obtain a third loss value corresponding to the first sample image pair based on the sample generation image and the sample target image in the first sample image pair. The step of determining the total training loss by the computer device may include: obtaining the total training loss based on the third loss value corresponding to the first sample image pair, the first loss value, and the second loss value corresponding to the sample image pair.

[0115] Exemplarily, the computer device may obtain a third loss value based on the sample generated image and the sample target image in the first sample image by using the following formula 3:

[0116] Formula 3: L rec =|Y s，t -X t |1;

[0117] Among them, L rec Represents the third loss value, Y s，t Indicates the sample generated image corresponding to the first sample image, X tRepresents the sample target image in the first sample image pair. It should be noted that when the sample source image and the sample target image belong to the same subject, by constraining the face-swapped result to be identical to the sample target image, when the trained face-swapped model performs face-swapping on images of the same subject, the generated face-swapped image is close to the target image, thereby improving the accuracy of model training.

[0118] In some embodiments, the discriminator includes at least one convolutional layer; the computer device may perform loss calculation based on the output results of each convolutional layer of the discriminator. Before determining the total training loss, for each sample image pair, the computer device determines a first similarity between a non-face area of ​​a first discrimination feature map and a non-face area of ​​a second discrimination feature map, where the first discrimination feature map is a feature map corresponding to a sample target image output by a first portion of the convolutional layers in the at least one convolutional layer, and the second discrimination feature map is a feature map of a sample generated image output by the first portion of the convolutional layers; the computer device determines a second similarity between a third discrimination feature map and a fourth discrimination feature map, where the third discrimination feature map is a feature map of the sample target image output by a second portion of the convolutional layers in the convolutional layers, and the fourth discrimination feature map is a feature map of the sample generated image output by the second portion of the convolutional layers; the computer device determines a fourth loss value based on the first similarity and the second similarity corresponding to each sample image pair; the step of determining the total training loss may include: the computer device obtains the total training loss based on the first loss value, the second loss value, and the fourth loss value.

[0119] Exemplarily, the computer device may determine the first similarity using a trained segmentation model. For example, the computer device may obtain a segmentation mask of the first or second discriminant feature map using the segmentation model, and determine the first similarity between the non-face region of the first or second discriminant feature map based on the segmentation mask. The segmentation mask may be a binary image of the first or second discriminant feature map, where pixels corresponding to non-face regions in the binary image are assigned a value of 1, and pixels corresponding to regions outside the non-face region are assigned a value of 0, thereby effectively extracting background regions outside the face.

[0120] Exemplarily, the computer device may determine the fourth loss value corresponding to the sample image pair by using the following formula 4:

[0121] Formula 4:

[0122]

[0123] Among them, L FM Represents the fourth loss value, M bgDenotes the segmentation mask, and the discriminator includes M convolutional layers, where the 1st to mth convolutional layers are the first partial convolutional layers, and the mth to Mth convolutional layers are the second partial convolutional layers. i (X t ) represents the feature map of the sample target image output by the i-th convolutional layer in the first part of the convolutional layer; D i (Y s，t ) represents the feature map of the sample generated image output by the i-th convolutional layer in the first part of the convolutional layer; D j (X t ) represents the feature map of the sample target image output by the j-th convolutional layer in the second part of the convolutional layer; D j (Y s，t ) represents the feature map of the sample generated image output by the j-th convolutional layer in the second part of the convolutional layer. It should be noted that the value of m is a positive integer not less than 0 and not greater than M. The value of m can be configured based on needs and is not limited in this application.

[0124] In some embodiments, the computer device may further obtain similarities between identity features based on each image and perform loss calculation. For example, before determining the total training loss, for each sample image pair, the computer device may extract a first identity feature of the sample source image, a second identity feature of the sample target image, and a third identity feature of the sample generated image; determine a first identity similarity between the sample source image and the sample generated image based on the first and third identity features; determine a first identity distance between the sample generated image and the sample target image based on the second and third identity features; and determine a second identity distance between the sample source image and the sample target image based on the first and second identity features; determine a distance difference based on the first and second identity distances; and determine a fifth loss value corresponding to each sample image pair based on the first identity similarity and distance difference corresponding to each sample image pair. The step of determining the total training loss by the computer device may include: obtaining the total training loss based on the first, second, and fifth loss values.

[0125] Exemplarily, the computer device may determine the fifth loss value by using the following formula 5:

[0126] Formula 5:

[0127] L ICL =

[0128] 1-cos(z id (Y s，t ), z id (X s))+(cos(z id (Y s，t ), z id (X t ))-cos(z id (X s ), z id (X t ))) 2 ;

[0129] Among them, L ICL represents the fifth loss value, z id (X s ) represents the first identity feature of the sample source image, z id (X t ) represents the second identity feature of the sample target image, z id (Y s，t ) represents the third identity feature of the sample generated image; 1-cos(z id (Y s，t ), z id (X s )) represents the first identity similarity between the sample source image and the sample generated image; cos(z id (Y s，t ), z id (X t )) represents the first identity distance between the sample generated image and the sample target image; cos(z id (X s ), z id (X t )) represents the second identity distance between the sample source image and the sample target image; (cos(z id (Y s，t ), z id (X t ))-cos(z id (X s ), z id (X t ))) 2 Indicates the distance difference.

[0130] It should be noted that the distance difference is determined by using the first and second identity distances. Since the second identity distance is used to measure the distance between the sample source image and the sample target image, by minimizing this distance difference, the first identity distance (i.e., the distance between the sample generated image and the sample target image) is maintained at a certain distance, which is comparable to the distance between the sample source image and the sample target image. The first identity similarity ensures that the generated image possesses the identity characteristics of the target image, thereby improving the accuracy of model training and the accuracy of face swapping.

[0131] Taking the total training loss including the above five loss values ​​as an example, the computer device can determine the total training loss by the following formula 6:

[0132] Formula 6: L total =L GAN +L mask +L FM +10*L rec +5*L ICL ;

[0133] Among them, L total Represents the total training loss, L GAN Represents the second loss value, L mask Represents the first loss value, L FM Represents the third loss value, L FM Represents the fourth loss value, L ICL Represents the fifth loss value.

[0134] In actual applications, the computer device trains the initial face-changing model based on the total training loss until the training is stopped when the target conditions are met, thereby obtaining the face-changing model.

[0135] It should be noted that the computer device can iteratively train the initial face-changing model based on the above steps 501 to 504, and obtain the total training loss corresponding to each iterative training, and adjust the parameters of the initial face-changing model based on the total training loss of each iterative training. For example, the parameters of the encoder, decoder, generator, discriminator, etc. in the initial face-changing model are optimized until the total training loss meets the target condition. The computer device stops training and uses the initial face-changing model obtained by the last optimization as the face-changing model. For example, the computer device can use the Adam algorithm optimizer with a learning rate of 0.0001 to iteratively train the initial face-changing model until the target condition is met. It is considered that the training has reached convergence and the training is stopped. For example, the target condition can be that the numerical value of the total loss is within the target numerical range, for example, the total loss is less than 0.5; or the time consumed by multiple iterative training exceeds the maximum duration, etc.

[0136] FIG3 is a schematic diagram of a face-changing model provided in an embodiment of the present application. As shown in FIG3 , the computer device can use the facial image of the object A as the source image X s , take the facial image of object B as the target image X t The computer device obtains the identity feature f of the source image through Fixed FR Net (fixed face recognition network) id , the computer device uses the identity feature f idThe N blocks included in the generator are input respectively; the computer device uses the U-shaped depth to enable the encoder and decoder of the network structure to obtain the initial attribute feature f1 of at least one scale of the target image att 、f2 att 、……、f i att 、……、f N-1 att 、f N att , and input them into the blocks of corresponding scales respectively. The computer device performs the above steps S1 to S4 for each block until the feature map output by the last block is obtained. The computer device generates the final target face-changing image Y based on the feature map output by the last block. st , thus completing the face change.

[0137] It should be noted that the image processing method of this application can achieve high-definition face replacement, for example, it can generate 1024 2 The generated high-resolution face-swap image takes into account high image quality, identity consistency with the source face in the source image, and effectively retains the key attributes of the target face in the target image with high accuracy. 2 The face-swapped image is processed by the image processing method of the present application, and the initial attribute features and identity features of at least one scale are processed in the layer-by-layer convolution of the generator, and the initial attribute features are screened using a control mask of at least one scale, so that redundant information such as the identity features of the target face is effectively filtered out from the obtained target attribute features, and the key attribute features of the target face are effectively retained; and the initial attribute features of the at least one scale highlight the features corresponding to different scales, and the control mask of the larger scale corresponding to the initial attribute features of the larger scale can achieve high-definition screening of the key attributes, thereby retaining the facial detail features such as hair, wrinkles, and facial occlusion of the target face with high precision, greatly improving the accuracy and clarity of the generated face-swapped image, and improving the authenticity of the face-swapped image.

[0138] Moreover, the image processing method of the present application can directly generate the entire face-changing image after the face-changing, which includes both the face after the face-changing and the background area, without the need for fusion or enhancement processing in related technologies; greatly improving the processing efficiency of the face-changing process.

[0139] Moreover, the face-changing model training method of the present application can perform end-to-end training on the entire generation framework used to generate sample images in the initial face-changing model during model training, thereby avoiding the error accumulation caused by multi-stage training, so that the face-changing model trained by the present application can generate face-changing images more stably, thereby improving the stability and reliability of the face-changing process.

[0140] Furthermore, the image processing method of the present application can generate face-swap images with higher resolution and accurately preserve the texture, skin brightness, hair and other details of the target face in the target image, thereby improving the accuracy, clarity and realism of the face-swap, and can be applied to scenes such as games and film and television that have higher requirements for face-swap quality. Furthermore, for virtual image maintenance scenarios, the image processing method of the present application can achieve a face-swap that replaces the face of any object with the face of any object. For a specific virtual image, the face of the specific virtual image can be replaced with the face image of any object, facilitating the maintenance of the virtual image and improving the convenience of virtual image maintenance.

[0141] The following comparison shows the face-swapping results using the image processing method of this application and the face-swapping results of related technologies. As can be seen from the comparison, the high-definition face-swapping results generated by the image processing method of this application show significant superiority over related technologies in both qualitative and quantitative comparisons.

[0142] As shown in Figure 7, a comparison of high-definition face-swapping results using some related art methods (hereinafter referred to as Method A) and the proposed solution in this application is shown. The comparison shows that Method A produces significant skin brightness inconsistencies and fails to preserve facial hair occlusions. The proposed solution retains the target face's skin brightness, expression, skin texture, occlusion, and other attribute characteristics, while also providing better image quality and greater authenticity.

[0143] Table 1 below shows a quantitative comparison of high-definition face-swapping results from Method A in the related art and the proposed solution. The experimental data in Table 1 compares the identity similarity (ID Retrieval) between the face in the generated face-swapping image and the face in the source image, the pose error (Pose Error) between the face in the face-swapping image and the face in the target image, and the image quality difference (FID) between the face in the face-swapping image and the real face image. The experimental data in Table 1 show that the identity similarity of the high-definition face-swapping results from the proposed solution is significantly higher than that of Method A in the related art; the pose error of the high-definition face-swapping results from the proposed solution is lower than that of Method A in the related art, with the pose error of the proposed solution being even lower; and the image quality difference of the high-definition face-swapping results from the proposed solution is significantly lower than that of Method A in the related art, with the image quality difference between the face-swapping images obtained by the proposed solution and the real image being smaller. Therefore, the proposed solution balances image quality, identity consistency with the source face, and attribute preservation of the target face, and is significantly superior to Method A in the related art.

[0144] Table 1

[0145]

[0146]

[0147] The image processing method of the embodiment of the present application obtains the identity features of the source image and the initial attribute features of at least one scale of the target image; inputs the identity features into a generator in a trained face-swapping model, and inputs the initial attribute features of at least one scale into the convolutional layer of the generator at the corresponding scale to obtain the target face-swapping image; and in each convolutional layer of the generator, a second feature map is generated based on the identity features and the first feature map output by the previous convolutional layer; and based on the second feature map and the initial attribute features, a control mask of the target image at the corresponding scale is determined to accurately locate pixels in the target image that carry features other than the identity features of the target face; by filtering out the target attribute features from the initial attribute features based on the control mask, a third feature map is generated based on the target attribute features and the second feature map, and the third feature map is output to the next convolutional layer. After layer-by-layer processing by at least one convolutional layer, the attributes and detailed features of the target face are effectively retained in the final target face-swapping image, thereby greatly improving the clarity of the face in the face-swapping image, achieving high-definition face-swapping, and improving the accuracy of face-swapping.

[0148] FIG8 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. As shown in FIG7 , the device includes

[0149] The feature acquisition module 801 is configured to acquire, in response to a received face-swapping request, an identity feature of a source image and an initial attribute feature of at least one scale of a target image;

[0150] The face swap request is used to request that a target face in the target image be swapped with a source face in the source image, the identity feature represents the object to which the source face belongs, and the initial attribute feature represents a three-dimensional attribute of the target face;

[0151] A face-changing module 802 is configured to input the identity feature and the initial attribute feature of the at least one scale into a face-changing model in the face-changing module;

[0152] Iteratively fusing the identity features and the initial attribute features of the at least one scale using the face-changing model to obtain fused features;

[0153] Based on the fusion features, generating a target face-swapped image through the face-swapped model, and outputting the target face-swapped image;

[0154] The face in the target face-swapped image is a fusion of the identity features of the source face and the target attribute features of the target face.

[0155] In some embodiments, the face-changing model includes at least one convolutional layer, each of which corresponds to one of the scales; the convolutional layer of the face-changing module 802 includes an acquisition unit, a generation unit, and an attribute screening unit; wherein,

[0156] an acquisition unit configured to acquire a first feature map output by a previous convolutional layer of a current convolutional layer;

[0157] a generating unit configured to generate a second feature map based on the identity feature and the first feature map;

[0158] an attribute screening unit configured to screen out target attribute features from the initial attribute features of the at least one scale, wherein the target attribute features are features other than the identity features of the target face;

[0159] The generating unit is further configured to generate a third feature map based on the target attribute feature and the second feature map, and output the third feature map to a next convolutional layer of the current convolutional layer as a first feature map of the next convolutional layer;

[0160] The third feature map output by the last convolutional layer in the at least one convolutional layer is determined as the fusion feature.

[0161] In some embodiments, the convolutional layer of the face-changing module 802 further includes:

[0162] a control mask determining unit configured to determine a control mask of the target image at a corresponding scale based on the second feature map and the initial attribute feature;

[0163] The control mask is used to represent pixels that carry features other than the identity features of the target face;

[0164] The generating unit is further configured to screen the initial attribute features of the at least one scale based on the control mask to obtain target attribute features.

[0165] In some embodiments, the control mask determination unit is further configured to perform feature splicing on the second feature map and the initial attribute feature to obtain a spliced ​​feature map;

[0166] Based on a preconfigured mapping convolution kernel and activation function, the spliced ​​feature map is mapped to the control mask.

[0167] In some embodiments, the number of the initial attribute features and the convolutional layers is a target number, the target number of convolutional layers is connected in series, different initial attribute features correspond to different scales, each convolutional layer corresponds to an initial attribute feature of the scale, and the target number is greater than or equal to two;

[0168] The acquisition unit is further configured to, when the current convolutional layer is the first convolutional layer among the target number of convolutional layers, acquire an initial feature map and use the initial feature map as the first feature map input into the current convolutional layer.

[0169] In some embodiments, the generation unit is further configured to perform an affine transformation on the identity feature to obtain a first control vector; based on the first control vector, map the first convolution kernel of the current convolution layer to a second convolution kernel; and based on the second convolution kernel, perform a convolution operation on the first feature map to generate a second feature map.

[0170] In some embodiments, when training the face-changing model, the apparatus further includes:

[0171] A sample acquisition module is configured to acquire a sample data set, the sample data set including at least one pair of sample images, each sample image pair including a sample source image and a sample target image;

[0172] a sample feature acquisition module configured to acquire a sample identity feature of a sample source image in a sample image pair and a sample initial attribute feature of at least one scale of a sample target image in the sample image pair;

[0173] a generation module configured to iteratively perform feature fusion on the sample identity feature and the sample initial attribute feature of the at least one scale through the generator of the initial face-swap model to obtain a sample fusion feature; and generate a sample generated image through the generator of the initial face-swap model based on the sample fusion feature;

[0174] an identification module configured to identify the sample generated image and the sample source image through the discriminator of the initial face-changing model to obtain an identification result;

[0175] a loss determination module, configured to determine the loss of the initial face-changing model based on the identification result;

[0176] A training module is configured to train the initial face-changing model based on the loss to obtain the face-changing model.

[0177] In some embodiments, the identification result includes a first identification result for the sample source image and a second identification result for the sample generated image;

[0178] The loss determination module is further configured to obtain a sample mask of at least one scale of the sample target image in each sample image pair, determine a first loss value based on the sample mask of at least one scale, and determine a second loss value based on the first identification result and the second identification result;

[0179] The training module is further configured to train the initial face-changing model based on the total training loss until the training is stopped when the target conditions are met, thereby obtaining the face-changing model.

[0180] In some embodiments, the sample source image and the sample target image correspond to the same object;

[0181] The loss determination module is further configured to obtain a third loss value based on the sample generated image and the sample target image; and obtain the total training loss based on the third loss value, the first loss value, and the second loss value.

[0182] In some embodiments, the discriminator includes at least one convolutional layer; and the loss determination module is further configured to:

[0183] For each sample image pair, determining a first similarity between a non-face region of a first discriminant feature map and a non-face region of a second discriminant feature map, where the first discriminant feature map is a feature map of the sample target image output by a first portion of the convolutional layers in the at least one convolutional layer, and the second discriminant feature map is a feature map of the sample generated image output by the first portion of the convolutional layers;

[0184] Determining a second similarity between a third discrimination feature map and a fourth discrimination feature map, where the third discrimination feature map is a feature map of the sample target image output by a second portion of the convolutional layers in the at least one convolutional layer, and the fourth discrimination feature map is a feature map of the sample generated image output by the second portion of the convolutional layers;

[0185] determining a fourth loss value based on the first similarity and the second similarity;

[0186] The total training loss is obtained based on the first loss value, the second loss value, and the fourth loss value.

[0187] In some embodiments, the loss determination module is further configured to:

[0188] For each sample image pair, extract the first identity feature of the sample source image, the second identity feature of the sample target image, and the third identity feature of the sample generated image;

[0189] Determining a first identity similarity between the sample source image and the sample generated image based on the first identity feature and the third identity feature;

[0190] Determining a first identity distance between the sample generated image and the sample target image based on the second identity feature and the third identity feature;

[0191] Determining a second identity distance between the sample source image and the sample target image based on the first identity feature and the second identity feature;

[0192] determining a distance difference based on the first identity distance and the second identity distance;

[0193] Determining a fifth loss value corresponding to each sample image pair based on the first identity similarity and the distance difference corresponding to each sample image pair;

[0194] Based on the first loss value, the second loss value, and the fifth loss value, the total loss of the training is obtained.

[0195] The image processing device of an embodiment of the present application obtains the identity features of a source image and the initial attribute features of at least one scale of a target image; inputs the identity features into a generator in a trained face-swapping model, and inputs the initial attribute features of at least one scale into the convolutional layers of the generator at the corresponding scale, thereby obtaining a target face-swapping image; and in each convolutional layer of the generator, a second feature map is generated based on the identity features and a first feature map output by the previous convolutional layer; and based on the second feature map and the initial attribute features, a control mask of the target image at the corresponding scale is determined to accurately locate pixels in the target image that carry features other than the identity features of the target face; and by filtering out the target attribute features from the initial attribute features based on the control mask, a third feature map is generated based on the target attribute features and the second feature map, and outputted to the next convolutional layer. After layer-by-layer processing by at least one convolutional layer, the attributes and detailed features of the target face are effectively retained in the final target face-swapping image, thereby greatly improving the clarity of the face in the face-swapping image, achieving high-definition face-swapping, and improving the accuracy of face-swapping.

[0196] Figure 9 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. As shown in Figure 9, the computer device includes: a memory and a processor; wherein the memory is configured to store a computer program; and the processor is configured to execute the computer program stored in the memory to implement the image processing method provided in an embodiment of the present application.

[0197] In some embodiments, a computer device is provided, as shown in FIG9 . The computer device 900 shown in FIG9 includes a processor 901 and a memory 903 . The processor 901 and the memory 903 are connected, for example, via a bus 902 . For example, the computer device 900 may also include a transceiver 904 , which may be used for data exchange between the computer device and other computer devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 904 is not limited to one, and the structure of the computer device 900 does not constitute a limitation on the embodiments of the present application.

[0198] The processor 901 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 901 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0199] Bus 902 may include a path for transmitting information between the aforementioned components. Bus 902 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Bus 902 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG9 shows only one thick line, but this does not indicate that there is only one bus or only one type of bus.

[0200] The memory 903 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media\other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0201] The memory 903 is configured to store a computer program for executing the embodiments of the present application, and the execution is controlled by the processor 901. The processor 901 is configured to execute the computer program stored in the memory 903 to implement the steps shown in the above method embodiments.

[0202] Among them, electronic equipment includes but is not limited to: servers, terminals or cloud computing center equipment, etc.

[0203] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0204] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0205] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0206] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Images