Image face changing method and system, electronic equipment and storage medium

Abstract

The invention discloses an image face changing method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a source image and a target image; performing identity feature extraction processing on the source image to obtain an identity feature vector; performing multi-scale feature extraction processing on the target image to obtain a global feature vector; performing face positioning processing on the target image according to the global feature vector to obtain a positioning feature vector; performing feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fused feature; and inputting the fused features into a face generation model to perform feature decoding processing, and outputting to obtain a face changing image. The embodiment of the invention can improve the quality of the face changing image, and can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image face-swapping method, system, electronic device, and storage medium. Background Art

[0002] Face-swapping technology transfers the identity information of one facial image to another, while preserving non-identifying content in the target image. While face-swapping methods in related technologies can achieve a certain degree of identity transfer, they perform poorly in preserving key attributes of the target face, such as head pose, expression, and gaze direction. This results in the generated facial images bearing obvious artifacts and making them difficult to meet the high-quality and natural-looking requirements of film and television production, privacy protection, and other fields. In summary, the technical issues present in related technologies need to be addressed. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose an image face-changing method, system, electronic device and storage medium, which can improve the quality and naturalness of the image after the face-changing.

[0004] To achieve the above-mentioned purpose, an embodiment of the present application provides an image face-swapping method, the method comprising:

[0005] Get the source image and the target image;

[0006] Performing identity feature extraction processing on the source image to obtain an identity feature vector;

[0007] Performing multi-scale feature extraction processing on the target image to obtain a global feature vector;

[0008] Performing face positioning processing on the target image according to the global feature vector to obtain a positioning feature vector;

[0009] Performing feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature;

[0010] The fused features are input into a face generation model for feature decoding processing, and a face-swapped image is output.

[0011] In some embodiments, performing multi-scale feature extraction on the target image to obtain a global feature vector includes the following steps:

[0012] Performing feature extraction processing on the target image through a heterogeneous residual network to obtain a local feature vector;

[0013] Performing feature reconstruction processing on the local feature vector through a feature pyramid network to obtain a reconstructed feature vector;

[0014] The reconstructed feature vector is processed by channel weight distribution through a weight distribution network to obtain the global feature vector.

[0015] In some embodiments, the heterogeneous residual network includes a downsampling residual block and an attention residual block, the attention residual block includes a global channel attention unit and a spatial dynamic convolution unit, and the feature extraction processing of the target image through the heterogeneous residual network to obtain a local feature vector includes the following steps:

[0016] Performing downsampling and residual connection processing on the target image through the downsampling residual block to obtain an initial feature vector;

[0017] Performing frequency domain analysis and channel attention calculation processing on the initial feature vector by the global channel attention unit to obtain channel information;

[0018] Performing spatial feature fusion processing on the initial feature vector through the spatial dynamic convolution unit to obtain spatial information;

[0019] Perform residual connection processing on the initial feature vector according to the channel information and the spatial information to obtain the local feature vector.

[0020] In some embodiments, performing face location processing on the target image according to the global feature vector to obtain a location feature vector includes the following steps:

[0021] Performing facial key point extraction processing on the target image according to the global feature vector to obtain key point features;

[0022] Performing attention calculation and semantic segmentation processing on the target image according to the global feature vector to obtain regional features;

[0023] Multimodal fusion processing is performed on the key point features and the region features to obtain a positioning feature vector.

[0024] In some embodiments, performing feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fused feature includes the following steps:

[0025] Performing face style migration processing on the global feature vector according to the identity feature vector to obtain a migration feature vector;

[0026] Performing facial feature analysis on the global feature vector to obtain facial features;

[0027] Dynamic feature fusion control processing is performed on the migration feature vector according to the positioning feature vector and the facial feature to obtain the fusion feature.

[0028] In some embodiments, performing face style transfer processing on the global feature vector according to the identity feature vector to obtain a transfer feature vector includes the following steps:

[0029] Injecting the identity feature vector into the global feature vector according to a multi-scale adaptive instance normalization algorithm to obtain a normalized feature vector;

[0030] The normalized feature vector is added to the global feature vector to obtain the migration feature vector.

[0031] In some embodiments, performing dynamic feature fusion control processing on the migration feature vector according to the positioning feature vector and the facial feature to obtain the fusion feature includes the following steps:

[0032] Performing face region positioning processing on the migration feature vector according to the positioning feature vector to obtain a region feature vector;

[0033] Performing double-projection convolution processing on the facial features to obtain a scaling factor;

[0034] Adjusting the regional feature vector according to the scaling factor to obtain an offset vector;

[0035] The migration feature vector is fused according to the offset vector to obtain the fused feature.

[0036] To achieve the above objectives, another aspect of the present application provides an image face-swapping system, the system comprising:

[0037] The first module is used to obtain a source image and a target image;

[0038] The second module is used to extract identity features from the source image to obtain an identity feature vector;

[0039] The third module is used to perform multi-scale feature extraction processing on the target image to obtain a global feature vector;

[0040] A fourth module is configured to perform face location processing on the target image according to the global feature vector to obtain a location feature vector;

[0041] A fifth module is configured to perform feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature;

[0042] The sixth module is used to input the fusion features into the face generation model for feature decoding processing, and output a face-swapped image.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0044] To achieve the above objectives, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0045] The embodiments of the present application include at least the following beneficial effects: The present application provides an image face-swap method, system, electronic device, and storage medium. This solution extracts identity features from a source image to obtain an identity feature vector, and performs multi-scale feature extraction on a target image to obtain a global feature vector. This multi-scale feature extraction process extracts and fuses the image's attribute features, preserving the key attributes of the target face and thereby enhancing the naturalness and realism of the face-swap image. Furthermore, this solution performs face location processing on the target image based on the global feature vector to obtain a location feature vector, and then performs feature fusion processing on the identity feature vector and the global feature vector based on the location feature vector to obtain a fused feature. This solution allows for fine-grained control and adjustment of the global feature vector, guiding the refined reconstruction of the feature map in specific areas, ensuring that the features of the face-swap image match those of the target image, thereby improving the quality of the face-swap image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of an image face-swapping method provided by an embodiment of the present application;

[0047] Figure 2 yes Figure 1 Flowchart of step S103 in FIG.

[0048] Figure 3 yes Figure 2 Flowchart of step S201 in FIG.

[0049] Figure 4 yes Figure 1 Flowchart of step S104 in FIG.

[0050] Figure 5 yes Figure 1 Flowchart of step S105 in FIG.

[0051] Figure 6 yes Figure 5 Flowchart of step S501 in FIG.

[0052] Figure 7 yes Figure 5 Flowchart of step S503 in FIG.

[0053] Figure 8 This is a structural diagram of an image face-swapping system provided in an embodiment of the present application;

[0054] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0056] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0057] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0059] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0060] 1) Artificial Intelligence (AI) is a new technological discipline that studies and develops theories, methods, techniques, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0061] 2) Image processing refers to the use of computers to analyze images to achieve desired results. It is also known as image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array captured by devices such as industrial cameras, video cameras, and scanners. The elements of this array are called pixels, and their values are called grayscale values. Image processing techniques generally include three components: image compression, enhancement and restoration, and matching, description, and recognition.

[0062] 3) Deepfake technology refers to artificial intelligence technology that uses a machine learning model called a Generative Adversarial Network (GAN) to combine and overlay images or videos onto source images or videos, using neural network technology for large-scale learning to stitch together a person's voice, facial expressions, and body movements to create fake content. The most common form of deepfake technology is AI face-swapping, but other deepfakes include voice simulation, face synthesis, and video generation.

[0063] In related technologies, the ability to retain facial attributes in the replaced image is insufficient, that is, it cannot effectively retain important attributes of the original face, such as head posture, expression, and gaze direction, and is more sensitive to lighting, angle, and occlusion, resulting in obvious traces of artificial generation in the generated facial image.

[0064] In view of this, embodiments of the present application provide an image face-swap method, system, electronic device, and storage medium. This solution extracts identity features from a source image to obtain an identity feature vector, and performs multi-scale feature extraction on a target image to obtain a global feature vector. This multi-scale feature extraction process extracts and fuses the image's attribute features, preserving the key attributes of the target face and thereby enhancing the naturalness and realism of the face-swap image. Furthermore, this solution performs face location processing on the target image based on the global feature vector to obtain a location feature vector. Based on the location feature vector, this solution performs feature fusion processing on the identity feature vector and the global feature vector to obtain a fused feature. This allows for fine-grained control and adjustment of the global feature vector, guiding the refined reconstruction of the feature map in specific areas, ensuring that the features of the face-swap image match those of the target image, thereby improving the quality of the face-swap image.

[0065] The image face-swapping method provided in the embodiment of the present application relates to the field of image processing technology. The image face-swapping method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the image face-swapping method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0067] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0068] Figure 1 This is an optional flowchart of an image face-swapping method provided in an embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S106.

[0069] Step S101, obtaining a source image and a target image;

[0070] Step S102, performing identity feature extraction processing on the source image to obtain an identity feature vector;

[0071] Step S103, performing multi-scale feature extraction processing on the target image to obtain a global feature vector;

[0072] Step S104, performing face location processing on the target image according to the global feature vector to obtain a location feature vector;

[0073] Step S105, performing feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature;

[0074] Step S106: input the fusion features into the face generation model for feature decoding processing, and output a face-swapped image.

[0075] In steps S101 to S106 illustrated in the embodiment of the present application, a source image and a target image are obtained, wherein the source image is used to represent the facial image providing facial features, and the target image is used to represent the facial image receiving facial features. Image face swapping involves transferring the facial features of the source image to the target image to obtain a new face-swapped image. The face-swapped image includes the facial features of the source image and background features of the target image, such as environmental features. Identity feature extraction is performed on the source image to obtain specific facial features of the source image, namely, an identity feature vector. This identity feature vector may include geometric features, texture features, and other features, representing the unique identity information of the face in the source image. Multi-scale feature extraction is then performed on the target image. A global feature vector is obtained through multi-level attribute feature extraction and fusion. The global feature vector may include occlusions, accessories, posture, expression, and other features, representing the global attribute characteristics of the target image. In the embodiment of the present application, face localization is performed on the target image based on the global feature vector. By extracting facial key points and regional features corresponding to the target image from the global feature vector, a localization feature vector is obtained. This localization feature vector is used to locate the geometric structure and region of the face in the target image. The identity feature vector and the global feature vector are fused based on the localization feature vector to generate a fused feature. Specifically, the global feature vector is migrated using the identity feature vector, and the migrated feature vector is guided and adjusted using the localization feature vector. This allows the generated face-swapped image to retain fine-grained attributes of the target image, such as expression and gaze direction. Finally, the fused feature is input into a face generation model for feature decoding. The face generation model decodes the features into a high-resolution face image, which is then output as the face-swapped image.

[0076] One of the above technical solutions has the following advantages or beneficial effects: This embodiment of the application performs multi-scale feature extraction on the global attribute features of the target image, preserving the key attributes of the target face, thereby improving the naturalness and realism of the face-swapped image. Furthermore, by locating feature vectors, the feature map is guided to perform refined reconstruction in specific areas, thereby improving the quality of the face-swapped image.

[0077] In step S101 of some embodiments, the source image and the target image may be input through a camera, or may be obtained through other means, such as from a database, but the present invention is not limited thereto.

[0078] In step S102 of some embodiments, identity feature extraction is performed on the source image to obtain an identity feature vector. The source image can be input into an identity encoder, which extracts a high-dimensional identity feature vector. The identity feature vector is used to represent the unique identity information of the face in the source image. The identity encoder can use a pre-trained model based on an image recognition architecture to encode facial identity features using a deep convolutional network to obtain the identity feature vector.

[0079] See also Figure 2 In step S103 of some embodiments, performing multi-scale feature extraction on the target image to obtain a global feature vector includes the following steps:

[0080] Step S201, performing feature extraction processing on the target image through a heterogeneous residual network to obtain a local feature vector;

[0081] Step S202, performing feature reconstruction processing on the local feature vector through a feature pyramid network to obtain a reconstructed feature vector;

[0082] Step S203 , performing channel weight distribution processing on the reconstructed feature vector through a weight distribution network to obtain the global feature vector.

[0083] In step S201 of some embodiments, a heterogeneous residual network is used to extract features of the target image to obtain a local feature vector, and the local feature vector captures the global attribute relationship to achieve the extraction and preservation of multi-scale global attribute features and establish a global dependency relationship across regions.

[0084] In step S202 of some embodiments, the local feature vector is reconstructed by a feature pyramid network. The original texture features can be retained through dense skip connections, and the geometric deformation adaptability can be improved through deformable convolution. Finally, the reconstructed feature vector is obtained through cross-level attention calculation. Among them, the Feature Pyramid Network (FPN) is a network structure for target detection that aims to solve multi-scale problems, especially the problem of small target detection. FPN improves the detection performance of targets of different scales by combining bottom-up and top-down feature fusion.

[0085] In step S203 of some embodiments, a weight allocation network is used to assign channel weights to the reconstructed feature vector, and the feature significance is evaluated by calculating the weight of each channel, thereby weighting the reconstructed feature vector to obtain a global feature vector. The weight allocation network assigns weights to each channel using a multilayer perceptron (MLP), which is a forward artificial neural network model in which network layers composed of multiple neurons are connected in a fully connected manner.

[0086] One of the above technical solutions has the following advantages or beneficial effects: The embodiment of the present application performs multi-scale feature extraction on the target image, which can capture global dependencies while extracting local features, effectively maintain the key attributes of the target face, and improve the naturalness and realism of the generated face-changing image.

[0087] See also Figure 3 In step S201 of some embodiments, the heterogeneous residual network includes a downsampling residual block and an attention residual block, the attention residual block includes a global channel attention unit and a spatial dynamic convolution unit, and the feature extraction processing of the target image by the heterogeneous residual network to obtain a local feature vector includes the following steps:

[0088] Step S301, performing downsampling and residual connection processing on the target image through the downsampling residual block to obtain an initial feature vector;

[0089] Step S302: Perform frequency domain analysis and channel attention calculation on the initial feature vector through the global channel attention unit to obtain channel information.

[0090] Step S303, performing spatial feature fusion processing on the initial feature vector through the spatial dynamic convolution unit to obtain spatial information;

[0091] Step S304 : performing residual connection processing on the initial feature vector according to the channel information and the spatial information to obtain the local feature vector.

[0092] In step S301 of some embodiments, the downsampling residual block is composed of two convolutional layers, which downsamples the input target image and retains local features through residual connections to obtain an initial feature vector. Among them, downsampling is a process of reducing the spatial resolution and amount of data in deep learning and signal processing. It is often used in convolutional neural networks to reduce the amount of calculation, expand the receptive field and extract abstract features. The residual connection is to introduce the output of the previous layer in a certain layer of the neural network as part of the input of the subsequent layer. In this way, the gradient can be directly propagated, which is helpful for the training of deep networks. The residual connection makes it easier to train the network by introducing shortcut connections and residual learning mechanisms, and has achieved remarkable results in computer vision tasks.

[0093] In step S302 of some embodiments, a global channel attention (CA) unit is used to extract global information by converting the initial feature vector to the frequency domain and performing channel attention in the frequency domain. This can more efficiently capture global dependencies and avoid redundant computations in the spatial domain, thereby extracting channel information. The global channel attention unit is a channel-based attention mechanism used in convolutional neural networks. It generates a channel attention map by utilizing the channel relationship between features, and the feature map of each channel is regarded as a feature detector.

[0094] In step S303 of some embodiments, spatial information is obtained by adaptive convolution kernel prediction using spatial dynamic convolution (SDC). Dynamic convolution can generate different convolution kernels based on the input content, enhancing the spatial adaptability of the model. In the embodiment of the present application, the spatial dynamic convolution unit uses multiple convolution kernels to extract different spatial features, and then adaptively weights them to obtain spatial information.

[0095] In step S304 of some embodiments, the heterogeneous residual network includes multiple downsampling residual blocks and multiple attention residual blocks. By alternately stacking the downsampling residual blocks and the attention residual blocks, multi-scale image features are gradually extracted, and the extracted channel information and spatial information are residually connected with the initial feature vector. The channel information and spatial information output by the attention residual block are passed to the subsequent module through jump connection, which can retain the attribute characteristics of the target image.

[0096] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application downsamples the input target image through a heterogeneous residual network, retains local features through residual connections, introduces a self-attention mechanism to capture global attribute relationships, and thus better retains the image features of the target image.

[0097] See also Figure 4In step S104 of some embodiments, performing face location processing on the target image according to the global feature vector to obtain a location feature vector includes the following steps:

[0098] Step S401, performing facial key point extraction processing on the target image according to the global feature vector to obtain key point features;

[0099] Step S402, performing attention calculation and semantic segmentation processing on the target image according to the global feature vector to obtain regional features;

[0100] Step S403: Perform multimodal fusion processing on the key point features and the region features to obtain a positioning feature vector.

[0101] In an embodiment of the present application, facial key points of the target image can be extracted based on the global feature vector, and key point extraction can be performed on the global feature vector based on a detection method based on an appearance model or a deep learning method based on a deep convolutional network, so as to extract the facial key points of the target image and input them into a three-layer simple convolutional neural network to extract key point features.

[0102] The global feature vector is then used to perform attention calculation and semantic segmentation on the target image. The Transformer model performs self-attention calculations to focus on important areas of the target image. Semantic segmentation network architectures such as U-Net are then used to segment the facial features of the target image, thereby locating the facial area of the target image and obtaining regional features. Multimodal fusion of key point features and regional features can improve the accuracy of the localized feature vector.

[0103] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application obtains a positioning feature vector by performing face positioning processing on the target image, and can use the positioning feature vector to perform fine control on specific areas of the face-swapped image, thereby improving the quality of the face-swapped image.

[0104] See also Figure 5 In step S105 of some embodiments, performing feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature includes the following steps:

[0105] Step S501, performing face style transfer processing on the global feature vector according to the identity feature vector to obtain a transferred feature vector;

[0106] Step S502, performing facial feature analysis on the global feature vector to obtain facial features;

[0107] Step S503 , performing dynamic feature fusion control processing on the migration feature vector according to the positioning feature vector and the facial feature to obtain the fusion feature.

[0108] In an embodiment of the present application, a migration feature vector is obtained by injecting the identity feature vector of the source image into the global feature vector of the target image, thereby migrating the facial features of the source image to the target image. The global feature vector is then subjected to facial feature analysis to obtain facial features, which may include head posture, expression, gaze direction, etc. For example, by analyzing the facial skin color and texture features of the target image, the generated facial skin color and texture can be accurately adjusted. Or, by analyzing the facial expressions and micro-expressions of the target image, a facial image reflecting a specific emotional state can be generated. Finally, the migration feature vector is subjected to dynamic feature fusion control processing based on the positioning feature vector and the facial features. The area or position that needs to be adjusted in the migration feature vector is determined by the positioning feature vector, and the corresponding feature vector is processed by the facial features to obtain a fusion feature.

[0109] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application performs feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature. Based on the process of feature positioning and fusion, the generated face-changing image can retain the fine-grained attributes in the target image, thereby improving the quality and consistency of the face-changing image.

[0110] See also Figure 6 In step S501 of some embodiments, performing face style transfer processing on the global feature vector according to the identity feature vector to obtain a transfer feature vector includes the following steps:

[0111] Step S601, injecting the identity feature vector into the global feature vector according to a multi-scale adaptive instance normalization algorithm to obtain a normalized feature vector;

[0112] Step S602 : Add the normalized feature vector and the global feature vector to obtain the migration feature vector.

[0113] In the embodiment of the present application, the identity feature vector of the source image is injected into the global feature vector of the target image through the multi-scale adaptive instance normalization (AdaIN) method to achieve facial style transfer. The specific formula can be expressed as:

[0114]

[0115] Where AdaIN represents multi-scale adaptive instance normalization, F S represents the identity feature vector, Ft Denotes the global eigenvector, σ() denotes the standard deviation of the feature, and μ() denotes the mean of the feature. The normalized eigenvector is then added to the global eigenvector to preserve the integrity of the target attribute and obtain the migration eigenvector.

[0116] One of the above technical solutions has the following advantages or beneficial effects: The embodiment of the present application realizes the transfer of facial style by injecting the identity feature vector of the source image into the global feature vector of the target image, so that the generated face-swapped image has the identity information of the source image, thereby improving the quality of the face-swapped image.

[0117] See also Figure 7 In step S503 of some embodiments, performing dynamic feature fusion control processing on the migration feature vector according to the positioning feature vector and the facial feature to obtain the fusion feature includes the following steps:

[0118] Step S701, performing face region positioning processing on the migration feature vector according to the positioning feature vector to obtain a region feature vector;

[0119] Step S702, performing double-projection convolution processing on the facial features to obtain a scaling factor;

[0120] Step S703, adjusting the regional feature vector according to the scaling factor to obtain an offset vector;

[0121] Step S704: performing fusion processing on the migration feature vector according to the offset vector to obtain the fusion feature.

[0122] In an embodiment of the present application, the migration feature vector can be processed for facial region positioning based on the positioning feature vector, the corresponding regional feature vector is obtained by matching or positioning from the migration feature vector, and the facial feature is subjected to double projection convolution processing to obtain a scaling factor, the intensity of the regional feature vector is adjusted by the scaling factor, and an offset vector is generated, which is used to supplement the detail information. In an embodiment of the present application, the migration feature vector is fused by the offset vector to obtain a fused feature. In an embodiment of the present application, the face-swapped image can be fine-grained controlled by the regional feature vector. For example, the eye region feature vector is used to correspondingly enhance the feature response of the pupil position to ensure that the eyeball direction of the face in the generated face-swapped image is consistent with the target; the mouth region feature vector is used to guide the fine reconstruction of the feature map in the mouth corner and lip shape area, so that the curvature of the mouth corner matches the target expression.

[0123] One of the above technical solutions has the following advantages or beneficial effects: The embodiment of the present application performs dynamic feature fusion control processing on the migration feature vector by positioning the feature vector and the facial features, which can effectively retain and adjust the key attributes in the target image when generating the face-swapped image, thereby improving the visual quality and authenticity of the generated face-swapped image.

[0124] In step S106 of some embodiments, the fused features are input into a face generation model for feature decoding processing, and a face-swapped image is output.

[0125] This embodiment of the application decodes the fused features into a high-resolution face image using a face generation model to obtain a face-swapped image. The face generation model consists of a decoder architecture and an output layer. The decoder structure includes a deconvolution layer, a batch normalization layer, and a LeakyReLU activation function. The output layer converts the feature map into a 3-channel RGB image using 1×1 convolution, with the same resolution as the input target image.

[0126] The following is a detailed description of the embodiments of the present application with reference to specific application examples:

[0127] The embodiments of the present application can be applied to application fields such as computer technology, artificial intelligence, and face swapping. Specifically, it can be widely used in scenarios such as film and television special effects production, digital content generation, biometric authentication, and privacy data desensitization. For example, the embodiments of the present application are applied to face swapping processing of images or videos. By selecting a video containing a face as the production material for a video ringtone, the face in the video is replaced with the face you want to replace it with. The embodiments of the present application can help users generate high-quality face-swapped videos and effectively save computing resources. It is conceivable that the embodiments of the present application can also combine the face-swapped task with the domain adaptation task, and optimize the face-swapped effect and domain adaptation ability at the same time through a multi-task learning framework, thereby further improving the quality and consistency of the generated faces.

[0128] See also Figure 8 The embodiment of the present application further provides an image face-swapping system that can implement the above-mentioned image face-swapping method, and the system includes:

[0129] The first module 801 is used to obtain a source image and a target image;

[0130] The second module 802 is configured to perform identity feature extraction processing on the source image to obtain an identity feature vector;

[0131] The third module 803 is used to perform multi-scale feature extraction processing on the target image to obtain a global feature vector;

[0132] The fourth module 804 is configured to perform face location processing on the target image according to the global feature vector to obtain a location feature vector;

[0133] The fifth module 805 is configured to perform feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature;

[0134] The sixth module 806 is used to input the fusion features into the face generation model for feature decoding processing, and output a face-swapped image.

[0135] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described image face-swapping method. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0137] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0139] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0140] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the image face swapping method of the embodiments of this application.

[0141] Input / output interface 903, used to implement information input and output;

[0142] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0143] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0144] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0145] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned image face-changing method is implemented.

[0146] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0148] The embodiments of the present application provide a method, system, electronic device, and storage medium for face-swapping an image. This solution extracts identity features from a source image to obtain an identity feature vector, and performs multi-scale feature extraction on a target image to obtain a global feature vector. This multi-scale feature extraction process extracts and fuses the image's attribute features, preserving the key attributes of the target face and thereby enhancing the naturalness and realism of the face-swapping image. Furthermore, this solution performs face location processing on the target image based on the global feature vector to obtain a location feature vector. Based on the location feature vector, this solution performs feature fusion processing on the identity feature vector and the global feature vector to obtain a fused feature. This allows for fine-grained control and adjustment of the global feature vector, guiding the refined reconstruction of the feature map in specific areas, ensuring that the features of the face-swapping image match those of the target image, thereby improving the quality of the face-swapping image.

[0149] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0150] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0151] The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0152] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0153] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0154] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0156] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0159] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A face-swapping method for an image, characterized in that: The method comprises the following steps: Get the source image and the target image; Performing identity feature extraction processing on the source image to obtain an identity feature vector; Performing multi-scale feature extraction processing on the target image to obtain a global feature vector; Performing face positioning processing on the target image according to the global feature vector to obtain a positioning feature vector; Performing feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature; The fused features are input into a face generation model for feature decoding processing, and a face-swapped image is output.

2. The method according to claim 1, characterized in that The multi-scale feature extraction process is performed on the target image to obtain a global feature vector, comprising the following steps: Performing feature extraction processing on the target image through a heterogeneous residual network to obtain a local feature vector; Performing feature reconstruction processing on the local feature vector through a feature pyramid network to obtain a reconstructed feature vector; The reconstructed feature vector is processed by channel weight distribution through a weight distribution network to obtain the global feature vector.

3. The method according to claim 2, characterized in that The heterogeneous residual network includes a downsampling residual block and an attention residual block, the attention residual block includes a global channel attention unit and a spatial dynamic convolution unit, and the feature extraction processing of the target image by the heterogeneous residual network to obtain a local feature vector includes the following steps: Performing downsampling and residual connection processing on the target image through the downsampling residual block to obtain an initial feature vector; Performing frequency domain analysis and channel attention calculation processing on the initial feature vector by the global channel attention unit to obtain channel information; Performing spatial feature fusion processing on the initial feature vector through the spatial dynamic convolution unit to obtain spatial information; Perform residual connection processing on the initial feature vector according to the channel information and the spatial information to obtain the local feature vector.

4. The method according to claim 1, wherein The step of performing face positioning processing on the target image according to the global feature vector to obtain a positioning feature vector comprises the following steps: Performing facial key point extraction processing on the target image according to the global feature vector to obtain key point features; Performing attention calculation and semantic segmentation processing on the target image according to the global feature vector to obtain regional features; Multimodal fusion processing is performed on the key point features and the region features to obtain a positioning feature vector.

5. The method according to claim 1, wherein The step of performing feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature comprises the following steps: Performing face style migration processing on the global feature vector according to the identity feature vector to obtain a migration feature vector; Performing facial feature analysis on the global feature vector to obtain facial features; Dynamic feature fusion control processing is performed on the migration feature vector according to the positioning feature vector and the facial feature to obtain the fusion feature.

6. The method according to claim 5, characterized in that The step of performing face style transfer processing on the global feature vector according to the identity feature vector to obtain a transfer feature vector comprises the following steps: Injecting the identity feature vector into the global feature vector according to a multi-scale adaptive instance normalization algorithm to obtain a normalized feature vector; The normalized feature vector is added to the global feature vector to obtain the migration feature vector.

7. The method according to claim 5, characterized in that The step of performing dynamic feature fusion control processing on the migration feature vector according to the positioning feature vector and the facial feature to obtain the fusion feature comprises the following steps: Performing face region positioning processing on the migration feature vector according to the positioning feature vector to obtain a region feature vector; Performing double-projection convolution processing on the facial features to obtain a scaling factor; Adjusting the regional feature vector according to the scaling factor to obtain an offset vector; The migration feature vector is fused according to the offset vector to obtain the fused feature.

8. An image face-swapping system, characterized in that: The system comprises: The first module is used to obtain a source image and a target image; The second module is used to extract identity features from the source image to obtain an identity feature vector; The third module is used to perform multi-scale feature extraction processing on the target image to obtain a global feature vector; A fourth module is configured to perform face location processing on the target image according to the global feature vector to obtain a location feature vector; A fifth module is configured to perform feature fusion processing on the identity feature vector and the global feature vector according to the positioning feature vector to obtain a fusion feature; The sixth module is used to input the fusion features into the face generation model for feature decoding processing, and output a face-swapped image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images