Face image exchange method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a face image exchange method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and the field of financial science and technology. The method comprises the following steps: acquiring a first face image and a second face image; performing first image coding on the first face image to obtain a first face information potential vector, and performing second image coding on the second face image to obtain a second face information potential vector; performing diffusion processing on the first face information potential vector and the second face information potential vector through a diffusion model to obtain a target fusion face information potential vector; performing face generation according to the target fusion face information potential vector to obtain a first fusion face image; performing transparency processing on the first face image to obtain a face attribute transparent image; and performing image fusion according to the first fusion face image and the face attribute transparent image to obtain a target face image. According to the embodiment of the invention, the face attribute of the first face image can be reserved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to a facial image exchange method and device, electronic device, and storage medium. Background Art

[0002] Face swapping refers to the technology of replacing the facial identity (such as the facial features of a facial image) in one facial image with another facial image. For example, in the financial technology scenario, users need to pass identity authentication to view or modify financial data. Face swapping technology can be applied in the identity authentication system to detect and prevent fake faces generated by face swapping technology, thereby improving the security of financial data.

[0003] The current face image exchange method is difficult to preserve facial details of face images, such as lighting, expression, etc. For example, when the lighting information of the first face image and the second face image is significantly different, the target face image finally generated by the current face image exchange method based on the first face image and the second face image often has unnatural shadows, and cannot well preserve the lighting information in the first face image and / or the second face image.

[0004] Therefore, how to preserve the facial details of face images has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a facial image exchange method and device, electronic device, and storage medium, aiming to preserve the facial details of facial images.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a face image exchange method, the face image exchange method comprising:

[0007] Acquire a first face image and a second face image;

[0008] Performing first image encoding on the first face image to obtain a first face information latent vector, and performing second image encoding on the second face image to obtain a second face information latent vector;

[0009] Diffusion processing is performed on the first face information latent vector and the second face information latent vector through a preset diffusion model to obtain a target fused face information latent vector; wherein the target fused face information latent vector is used to represent the identity information of the first face image and the background information of the second face image;

[0010] Generate a face according to the target fused face information latent vector to obtain a first fused face image;

[0011] Performing transparency processing on the first face image to obtain a transparent face attribute map;

[0012] Image fusion is performed based on the first fused face image and the face attribute transparent map to obtain a target face image.

[0013] In some embodiments, the step of performing diffusion processing on the first face information latent vector and the second face information latent vector by using a preset diffusion model to obtain a target fused face information latent vector includes:

[0014] Using the diffusion model to perform identity information diffusion on the first face information latent vector to obtain an identity information latent vector, and using the diffusion model to perform background information diffusion on the second face information latent vector to obtain a background information latent vector;

[0015] Feature fusion is performed according to the identity information latent vector and the background information latent vector to obtain the target fused face information latent vector.

[0016] In some embodiments, the step of performing identity information diffusion on the first face information latent vector using the diffusion model to obtain the identity information latent vector, and performing background information diffusion on the second face information latent vector using the diffusion model to obtain the background information latent vector, includes:

[0017] Performing a first noise addition process on the first face information latent vector according to a preset first noise coefficient to obtain a first face information noisy latent vector;

[0018] Performing a second noise addition process on the second face information latent vector according to a preset second noise coefficient to obtain a second face information noisy latent vector; wherein the first noise coefficient is not equal to the second noise coefficient;

[0019] Performing a first denoising process on the first face information denoised latent vector to obtain the identity information latent vector;

[0020] A second denoising process is performed on the second facial information denoised latent vector to obtain the background information latent vector.

[0021] In some embodiments, the performing transparency processing on the first face image to obtain a transparent face attribute map includes:

[0022] Performing semantic segmentation on the first face image to obtain an initial mask image;

[0023] Performing corrosion expansion on the initial mask image to obtain a target mask image;

[0024] Gaussian blur is performed on the target mask image to obtain the transparent image of the facial attributes of the human face.

[0025] In some embodiments, the face attribute transparent map has illumination information, facial occlusion information, and face angle information of the first face image;

[0026] The step of performing image fusion according to the first fused face image and the face attribute transparent map to obtain a target face image includes:

[0027] updating the first fused face image according to the illumination information and the facial occlusion information to obtain a second fused face image;

[0028] The angle of the second fused face image is adjusted according to the face angle information to obtain the target face image.

[0029] In some embodiments, before generating a face according to the target fused face information latent vector to obtain a first fused face image, the face image exchange method further includes:

[0030] Obtain the potential vector of sample face information;

[0031] Inputting the sample face information latent vector into a preset initial generative adversarial model to generate a sample face, thereby obtaining a sample face image;

[0032] Perform loss calculation based on the sample face image to obtain target loss data;

[0033] Adjusting the model parameters of the initial generative adversarial model according to the target loss data to obtain a target generative adversarial model;

[0034] The step of generating a face according to the target fused face information latent vector to obtain a first fused face image includes:

[0035] The target fused face information latent vector is input into the target generative adversarial model to generate a fused face, so as to obtain the first fused face image.

[0036] In some embodiments, before performing first image encoding on the first face image to obtain a first face information latent vector, and performing second image encoding on the second face image to obtain a second face information latent vector, the face image exchange method further includes:

[0037] Performing a first facial correction on the first facial image according to the first facial key points of the first facial image and the preset standard facial key point positions to obtain a first corrected facial image;

[0038] Performing a second face correction on the second face image according to the second face key points of the second face image and the positions of the standard face key points to obtain a second corrected face image;

[0039] The first corrected facial image is updated to the first facial image, and the second corrected facial image is updated to the second facial image.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a facial image exchange device, the device comprising:

[0041] An image acquisition module, used to acquire a first face image and a second face image;

[0042] An image encoding module, configured to perform a first image encoding on the first face image to obtain a first face information latent vector, and to perform a second image encoding on the second face image to obtain a second face information latent vector;

[0043] A diffusion module, configured to perform diffusion processing on the first face information latent vector and the second face information latent vector through a preset diffusion model to obtain a target fused face information latent vector; wherein the target fused face information latent vector is used to represent the identity information of the first face image and the background information of the second face image;

[0044] A face generation module, used for performing face generation according to the target fused face information latent vector to obtain a first fused face image;

[0045] A transparency module, used for performing transparency processing on the first face image to obtain a transparent face attribute map;

[0046] The image fusion module is used to perform image fusion based on the first fused face image and the face attribute transparent map to obtain a target face image.

[0047] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] The face image exchange method and device, electronic device, and storage medium proposed in the present application perform first image encoding on the first face image to obtain the first face information latent vector; perform second image encoding on the second face image to obtain the second face information latent vector; then, diffuse the first face information latent vector and the second face information latent vector through a preset diffusion model to obtain the target fused face information latent vector, thereby more fully integrating the identity information of the first face image and the background information of the second face image. In addition, after generating a face according to the target fused face information latent vector to obtain the first fused face image, the first face image is also transparently processed to obtain a face attribute transparent map, and then image fusion is performed according to the first fused face image and the face attribute transparent map to obtain the target face image. It can be seen that in addition to retaining the identity information of the first face image and the background information of the second face image in the target face image, the present application also retains the face attribute information of the first face image in the target face image, thereby fully retaining the facial details of the first face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a facial image exchange method provided in an embodiment of the present application;

[0051] Figure 2 yes Figure 1 Flow chart of step 103 in;

[0052] Figure 3 yes Figure 2 Flow chart of step 201 in;

[0053] Figure 4 yes Figure 1 Flowchart of step 105 in;

[0054] Figure 5 yes Figure 1 Flow chart of step 106 in;

[0055] Figure 6 is a flow chart of a facial image exchange method provided by another embodiment of the present application;

[0056] Figure 7 is a flow chart of a facial image exchange method provided by another embodiment of the present application;

[0057] Figure 8 is a structural schematic diagram of a face image exchange device provided in an embodiment of the present application;

[0058] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] Face swapping: refers to the technology of replacing the identity features of a face image, such as facial features and facial expressions, with another face image. Face swapping has a wide range of applications in entertainment, film and television production, human-computer interaction and other fields.

[0065] Diffusion Models (DM): A generative model. In the field of machine learning and deep learning, diffusion models are used to generate data. Diffusion models first gradually add noise to the original data through a forward process, and then remove the noise from the data through a reverse process, gradually restoring the noise-free data, thereby generating new data similar to the original data.

[0066] Generative Adversarial Network (GAN): A deep learning model. The GAN consists of two parts: the generator and the discriminator, and can generate realistic images, text, or audio content.

[0067] The facial image exchange method and device, electronic device, and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the facial image exchange method in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The face image exchange method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the face image exchange method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0073] Figure 1 is an optional flowchart of the face image exchange method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 101 to 106.

[0074] Step 101, obtaining a first face image and a second face image;

[0075] Step 102, performing first image coding on the first face image to obtain a first face information latent vector, and performing second image coding on the second face image to obtain a second face information latent vector;

[0076] Step 103, performing diffusion processing on the first face information latent vector and the second face information latent vector through a preset diffusion model to obtain a target fused face information latent vector; wherein the target fused face information latent vector is used to represent the identity information of the first face image and the background information of the second face image;

[0077] Step 104, generating a face according to the target fused face information latent vector to obtain a first fused face image;

[0078] Step 105, performing transparency processing on the first face image to obtain a transparent face attribute map;

[0079] Step 106, performing image fusion based on the first fused face image and the face attribute transparent map to obtain a target face image.

[0080] The beneficial effects of the embodiments of the present application include but are not limited to: performing first image encoding on the first face image to obtain a first face information latent vector; performing second image encoding on the second face image to obtain a second face information latent vector; then, performing diffusion processing on the first face information latent vector and the second face information latent vector through a preset diffusion model to obtain a target fused face information latent vector, thereby more fully integrating the identity information of the first face image and the background information of the second face image. In addition, after performing face generation according to the target fused face information latent vector to obtain the first fused face image, the first face image is also transparently processed to obtain a face attribute transparent map, and then image fusion is performed according to the first fused face image and the face attribute transparent map to obtain the target face image. It can be seen that in addition to retaining the identity information of the first face image and the background information of the second face image in the target face image, the present application also retains the face attribute information of the first face image in the target face image, thereby fully retaining the facial details of the first face image.

[0081] In step 101 of some embodiments, the first face image and the second face image may be obtained from a face image database. Alternatively, the first face image may be obtained by photographing the first face through a camera device, and the second face image may be obtained by photographing the second face through a camera device. The first face image and the second face image may also be obtained by capturing face images from a video. The first face image and the second face image may also be obtained by other methods, not limited thereto.

[0082] In some embodiments, the image size of the first face image and the image size of the second face image may be the same, for example, both are 256x256. Wherein, 256 refers to the number of pixels. In another embodiment, the face angle of the first face image and the face angle of the second face image may be the same, so as to facilitate subsequent image processing steps.

[0083] It should be noted that the face image (such as the first face image and the second face image) contains identity information, facial attribute information and background information. Specifically, identity information refers to facial features, such as the color and shape of the facial features. Facial attribute information refers to lighting, head posture, facial expression, facial angle information, facial occlusion information (such as glasses, hair, etc.). Background information refers to the area other than the person area in the face image, such as the sky, houses, streets, etc.

[0084] In step 102 of some embodiments, the first face image may be encoded by an encoder to obtain a first face information latent vector, and the second face image may be encoded by an encoder to obtain a second face information latent vector. Specifically, the encoder may be an E4E encoder. It should be noted that the E4E encoder has a good encoding effect and can accurately extract the latent information in the face image to generate a latent vector containing the face information, thereby generating a high-quality target face image based on the latent vector. Other encoders may also be used to encode the face image, but are not limited to this.

[0085] In step 103 of some embodiments, specifically, the diffusion model may be a Diffusion diffusion model. Diffusion processing is performed on the first face information latent vector and the second face information latent vector, specifically, vector fusion is performed on the first face information latent vector and the second face information latent vector, for example, weighted summation or vector concatenation is performed on the first face information latent vector and the second face information latent vector, so as to generate a target fused face information latent vector. It should be noted that the target fused face information latent vector retains the identity information of the first face image, as well as the background information and facial attributes (such as the head posture and facial angle of the face, etc.) of the second face image.

[0086] In step 104 of some embodiments, the target fused face information latent vector may be input into a decoder for face generation to obtain a first fused face image. The decoder corresponds to the encoder, and the encoder is used to perform a first image encoding on the first face image to obtain a first face information latent vector, and to perform a second image encoding on the second face image to obtain a second face information latent vector.

[0087] In step 105 of some embodiments, the first face image is transparentized to obtain a face attribute transparent map. It should be noted that the face attribute transparent map may have transparent channel data (such as the value of the Alpha channel). Each pixel of the face attribute transparent map corresponds to a transparent channel data, and the transparent channel data is used to indicate the transparency of the pixel. For example, the value range of the transparent channel data may be [0, 255]; the transparent channel data is 0, indicating complete transparency; the transparent channel data is 255, indicating complete opacity.

[0088] In step 106 of some embodiments, image fusion may be performed based on the deep learning model, the first fused face image, and the face attribute transparent map to obtain a target face image. Specifically, the deep learning model may be a generative adversarial network model, such as a StyleGAN2 model.

[0089] In some embodiments, the face image exchange method in the embodiments of the present application can be applied in financial technology scenarios. For example, the target face image generated by the face image exchange method can be used as training data, and the identity authentication system can be trained with the training data, so that the identity authentication system can identify the target face image generated by the face image exchange method as a false face, thereby preventing malicious third parties from using false faces to pass identity authentication and modify financial data, thereby improving the security of financial data.

[0090] In some embodiments, the face image exchange method in the embodiment of the present application can also be applied in digital medical scenarios, such as medical education, psychotherapy, cosmetic surgery, etc. For example, in medical education, the face image exchange method can be used to generate target face images with different diseases or injuries, and the learning effect and interest of medical students can be improved by displaying the target face images. In psychotherapy, the target face images generated by the face image exchange method can be used to allow patients with facial disabilities to see their faces exchanged with others to relieve patients' anxiety or depression and other psychological problems. In cosmetic surgery, the face image exchange method can be used to generate target face images corresponding to different plastic surgery schemes, thereby helping patients choose the most suitable plastic surgery scheme. In addition, in the field of science and technology, the face exchange algorithm can also be used in entertainment, education, human-computer interaction, etc., such as making face-changing videos, simulating different expressions and postures, and enhancing the experience of virtual reality and augmented reality. The application of the face image exchange method in the embodiment of the present application is not limited.

[0091] It should be noted that the current face image exchange method is difficult to retain the facial details of the face image, such as lighting, expression, etc. For example, when the lighting difference between the first face image and the second face image is large, the target face image finally generated by the current face image exchange method based on the first face image and the second face image often has unnatural shadows, resulting in unstable image quality of the target face image. In the embodiment of the present application, in addition to retaining the identity information of the first face image and the background information of the second face image, the target face image can also retain the facial attribute information of the first face image, such as lighting, facial angle, facial occlusion hair and other facial details, so as to fully retain the facial details of the first face image in the target face image. In addition, it can avoid generating a target face image with wrong identity information when the facial attributes of the first face image and the second face image are greatly different, improve the stability of the image quality of the target face image, and improve the reliability of the face image exchange method.

[0092] See also Figure 2 In some embodiments, step 103 may include but is not limited to steps 201 to 202:

[0093] Step 201, using a diffusion model to diffuse identity information on a first face information latent vector to obtain an identity information latent vector, and using a diffusion model to diffuse background information on a second face information latent vector to obtain a background information latent vector;

[0094] Step 202, feature fusion is performed based on the identity information latent vector and the background information latent vector to obtain a target fused face information latent vector.

[0095] The advantage of this embodiment is that the identity information of the first face information latent vector is diffused by using a diffusion model to obtain an identity information latent vector, and the background information of the second face information latent vector is diffused by using a diffusion model to obtain a background information latent vector; then, feature fusion is performed based on the identity information latent vector and the background information latent vector to obtain a target fused face information latent vector, so that the identity information of the first face image and the background information of the second face image are represented by the target fused face information latent vector, which can more fully fuse the identity information of the first face image and the background information of the second face image, thereby improving the reliability of the face image exchange method.

[0096] In step 201 of some embodiments, the first face information latent vector may be subjected to identity information diffusion in the latent vector space, and the second face information latent vector may be subjected to background information diffusion in the latent vector space.

[0097] It should be noted that the latent vector space is a low-dimensional space used to represent data in machine learning, that is, a latent representation space. In the latent vector space, each point represents a compressed representation of a data, and similar data are close to each other in the latent space. The latent vector space has a decoupling characteristic, so it is possible to use the latent vector space to extract different types of information from the first face information latent vector, such as extracting a vector representing identity information and a vector representing background information, thereby generating a target fused face information latent vector, and then using the target fused face information latent vector to generate a target face image, thereby realizing face image exchange.

[0098] It should be noted that the embodiment of the present application encodes the first facial image into a first facial information latent vector, and encodes the second facial image into a second facial information latent vector, and then diffuses the first facial information latent vector and the second facial information in the latent vector space, rather than directly diffusing the original first facial image and the second facial image. This can greatly reduce the computing cost and time, and improve the efficiency of the facial image exchange method.

[0099] In step 202 of some embodiments, feature fusion is performed based on the identity information latent vector and the background information latent vector, which may be vector concatenation of the identity information latent vector and the background information latent vector. It may also be weighted summation of the identity information latent vector and the background information latent vector, or feature fusion of the identity information latent vector and the background information latent vector is performed in other ways, which is not limited in the embodiments of the present application.

[0100] See also Figure 3 In some embodiments, step 201 may include but is not limited to steps 301 to 304:

[0101] Step 301, performing a first noise addition process on a first face information latent vector according to a preset first noise coefficient to obtain a first face information noisy latent vector;

[0102] Step 302, performing a second noise addition process on the second face information latent vector according to a preset second noise coefficient to obtain the second face information noisy latent vector; wherein the first noise coefficient is not equal to the second noise coefficient;

[0103] Step 303, performing a first denoising process on the first face information denoised latent vector to obtain an identity information latent vector;

[0104] Step 304: Perform a second denoising process on the second facial information denoised latent vector to obtain a background information latent vector.

[0105] The advantage of this embodiment is that a first noise addition process is performed on the first facial information latent vector according to a preset first noise coefficient to obtain the first facial information noisy latent vector; and a second noise addition process is performed on the second facial information latent vector according to a second noise coefficient different from the second noise coefficient to obtain the second facial information noisy latent vector; then, a first denoising process is performed on the first facial information noisy latent vector to obtain an identity information latent vector; a second denoising process is performed on the second facial information noisy latent vector to obtain a background information latent vector, so that feature fusion can be performed based on the identity information latent vector and the background information latent vector to obtain a target fused facial information latent vector; and then, a target facial image is generated based on the target fused facial information latent vector, so that the target facial image retains the identity information of the first facial image and the background information of the second facial image.

[0106] In step 301 of some embodiments, the first face information latent vector may be subjected to a first noise addition process according to a preset first noise coefficient and a preset noise addition relational expression to obtain the first face information noisy latent vector. It should be noted that the noise addition relational expression is a noise addition function of a diffusion model.

[0107] In step 302 of some embodiments, a second noise addition process may be performed on the second facial information latent vector according to a preset second noise coefficient and a noise addition relationship to obtain a second facial information noisy latent vector.

[0108] In step 303 of some embodiments, the first face information denoised latent vector may be subjected to a first denoising process according to a denoising relation to obtain an identity information latent vector. It should be noted that the denoising relation is an inverse relation of the denoising relation, such as an inverse function of the denoising relation.

[0109] In step 304 of some embodiments, a second denoising process may be performed on the second facial information denoised latent vector according to a denoising relationship to obtain a background information latent vector.

[0110] See also Figure 4 In some embodiments, step 105 may include but is not limited to steps 401 to 403:

[0111] Step 401, performing semantic segmentation on the first face image to obtain an initial mask image;

[0112] Step 402, corroding and expanding the initial mask image to obtain a target mask image;

[0113] Step 403, Gaussian blur is performed on the target mask image to obtain a transparent image of facial attributes.

[0114] The advantage of this embodiment is that semantic segmentation is performed on the first face image to obtain an initial mask map, and then the initial mask map is corroded and puffed to obtain a target mask map; Gaussian blur is performed on the target mask map to obtain a face facial attribute transparent map, so that the facial attributes of the first face image are retained through the face facial attribute transparent map, and then the face facial attribute transparent map is used to perform image fusion with the first fused face image, and the facial attributes of the first face image can also be fully retained in the obtained target face image.

[0115] In step 401 of some embodiments, semantic segmentation can be performed on the face area and the background area of ​​the first face image to obtain an initial mask map, thereby separating the face area and the background area of ​​the first face image. It should be noted that the first face image has a face area and a background area, so the initial mask map obtained by semantic segmentation of the first face image has different mask areas, such as a face mask area corresponding to the face area, and a background mask area corresponding to the background area. Different mask values ​​can be set for different mask areas to distinguish them. For example, in the initial mask map, the mask value of each pixel in the face mask area is 1, and the mask value of each pixel in the background mask area is 0. Different colors can also be used to distinguish different mask areas in the initial mask map, which is not limited in the embodiments of the present application.

[0116] Specifically, the Bisenet model can be used to perform semantic segmentation on the first face image and the second face image. The Bisenet model is a semantic segmentation model.

[0117] In step 402 of some embodiments, the initial mask image may be first subjected to image corrosion processing, and then subjected to image expansion processing to obtain a target mask image, so that the edge of the mask area in the target mask image is smoother.

[0118] In step 403 of some embodiments, Gaussian blur is performed on the target mask image to obtain a transparent face attribute map to reduce the noise of the transparent face attribute map, achieve image denoising, and further improve the image quality of the target face image generated according to the transparent face attribute map, for example, making the light and shadow transition of the target face image smoother and more natural.

[0119] It should be noted that Gaussian blur is an image processing technique that uses a Gaussian function to reduce image noise.

[0120] See also Figure 5 In some embodiments, the face attribute transparent map has the illumination information, face occlusion information and face angle information of the first face image. Step 106 may include but is not limited to steps 501 to 502:

[0121] Step 501, updating the first fused face image according to the illumination information and the face occlusion information to obtain a second fused face image;

[0122] Step 502: Adjust the angle of the second fused face image according to the face angle information to obtain a target face image.

[0123] The advantage of this embodiment is that the first fused face image is updated according to the illumination information and the facial occlusion information to obtain the second fused face image; and the angle of the second fused face image is adjusted according to the facial angle information to obtain the target face image, thereby retaining the facial attributes of the first face image such as the illumination information, facial occlusion information and facial angle information in the target face image.

[0124] In step 501 of some embodiments, illumination information of the first fused facial image may be updated according to illumination information, and facial occlusion information of the first fused facial image may be updated according to facial occlusion information.

[0125] In step 502 of some embodiments, the angle of the second fused face image is adjusted according to the face angle information, and face correction is performed on the face key points of the second fused face image according to the face angle information, so as to obtain a target face image that retains the face angle information of the first face image.

[0126] See also Figure 6 In some embodiments, before step 104, the face image exchange method further includes:

[0127] Step 601, obtaining a potential vector of sample face information;

[0128] Step 602, inputting the sample face information latent vector into a preset initial generative adversarial model to generate a sample face, thereby obtaining a sample face image;

[0129] Step 603, performing loss calculation based on the sample face image to obtain target loss data;

[0130] Step 604, adjusting the model parameters of the initial generative adversarial model according to the target loss data to obtain a target generative adversarial model;

[0131] Step 104 includes:

[0132] Step 605: input the target fused face information latent vector into the target generative adversarial model to generate a fused face, and obtain a first fused face image.

[0133] The advantage of this embodiment is that before the target fused face information latent vector is input into the target generative adversarial model for fused face generation, the initial generative adversarial model is first trained with the sample face information latent vector to obtain the trained target generative adversarial model, thereby improving the quality and reliability of the first fused face image generated according to the target fused face information latent vector.

[0134] In step 601 of some embodiments, the sample face information latent vector may be obtained from a sample database. Other methods may also be used to obtain the sample face information latent vector, which is not limited.

[0135] In step 602 of some embodiments, the initial generative adversarial model may specifically be an untrained StyleGAN2 model.

[0136] In step 603 of some embodiments, the target loss data can be obtained by calculation based on the loss function of the initial generative adversarial model and the sample face image. The loss function can specifically be a non-saturated loss function. It should be noted that the target loss data refers to the function value calculated based on the loss function.

[0137] In step 604 of some embodiments, the model parameters of the initial generative adversarial model are adjusted according to the target loss data. Specifically, the target loss data may be minimized to adjust the model parameters of the initial generative adversarial model to model parameters corresponding to the minimum target loss data.

[0138] In step 605 of some embodiments, the target generative adversarial model may be a trained StyleGAN2 model.

[0139] See also Figure 7 In some embodiments, before step 102, the face image exchange method may further include but is not limited to steps 701 to 703:

[0140] Step 701, performing a first face correction on the first face image according to the first face key points of the first face image and the preset standard face key point positions, to obtain a first corrected face image;

[0141] Step 702, performing a second face correction on the second face image according to the second face key points and the positions of the standard face key points of the second face image to obtain a second corrected face image;

[0142] Step 703: update the first corrected face image to a first face image, and update the second corrected face image to a second face image.

[0143] The advantage of this embodiment is that, according to the first facial key points of the first facial image and the preset facial standard key point positions, the first facial correction is performed on the first facial image to obtain the first corrected facial image; and according to the second facial key points of the second facial image and the facial standard key point positions, the second facial correction is performed on the second facial image to obtain the second corrected facial image, and then the first corrected facial image is updated to the first facial image, and the second corrected facial image is updated to the second facial image. By adjusting the facial key points of the first facial image and the second facial image to the same facial standard key point positions, subsequent image encoding processing can be performed.

[0144] In step 701 of some embodiments, the first facial key points refer to the key points of the facial features in the first facial image, and there may be five first facial key points. For example, the first facial key points include two eye key points, one nose key point, and one mouth key point in the first facial image.

[0145] In step 702 of some embodiments, the second facial key points refer to the key points of the facial features in the second facial image. It should be noted that the facial key points of the first corrected facial image and the facial key points of the second corrected facial image are at the same position, both of which are standard facial key point positions.

[0146] In step 703 of some embodiments, the first rectified facial image is updated to the first facial image, and the second rectified facial image is updated to the second facial image, so that image encoding can be performed using the rectified images.

[0147] In some embodiments, before step 102, the face image exchange method may further include:

[0148] Performing a first image cropping on the first face image using a preset face detection and recognition algorithm to obtain a first face region map;

[0149] Performing second image cropping on the second face image by using a face detection and recognition algorithm to obtain a second face region map;

[0150] The first face region map is updated to a first face image, and the second face region map is updated to a second face image.

[0151] The advantage of this embodiment is that the area where the face is located is detected from the first face image and the second face image by using a face detection and recognition algorithm, and the image is cropped to obtain the first face area map and the second face area map.

[0152] See also Figure 8 The embodiment of the present application also provides a face image exchange device, which can implement the above face image exchange method, and the device includes:

[0153] An image acquisition module 801 is used to acquire a first face image and a second face image;

[0154] An image encoding module 802 is configured to perform a first image encoding on the first face image to obtain a first face information latent vector, and to perform a second image encoding on the second face image to obtain a second face information latent vector;

[0155] Diffusion module 803, used to perform diffusion processing on the first face information latent vector and the second face information latent vector through a preset diffusion model to obtain a target fused face information latent vector; wherein the target fused face information latent vector is used to represent the identity information of the first face image and the background information of the second face image;

[0156] A face generation module 804 is used to generate a face according to the target fused face information latent vector to obtain a first fused face image;

[0157] A transparency module 805 is used to perform transparency processing on the first face image to obtain a transparent face attribute map;

[0158] The image fusion module 806 is used to perform image fusion according to the first fused face image and the face attribute transparent map to obtain a target face image.

[0159] In one embodiment, the face image exchange device also includes a model training module, which is used to: obtain a latent vector of sample face information; input the latent vector of the sample face information into a preset initial generative adversarial model to generate a sample face and obtain a sample face image; perform loss calculation based on the sample face image to obtain target loss data; adjust the model parameters of the initial generative adversarial model based on the target loss data to obtain a target generative adversarial model.

[0160] In one embodiment, the facial image exchange device also includes an image correction module, which is used to: perform a first facial correction on the first facial image according to the first facial key points of the first facial image and the preset standard facial key point positions to obtain a first corrected facial image; perform a second facial correction on the second facial image according to the second facial key points of the second facial image and the standard facial key point positions to obtain a second corrected facial image; update the first corrected facial image to the first facial image, and update the second corrected facial image to the second facial image.

[0161] The specific implementation of the facial image exchange device is basically the same as the specific implementation of the facial image exchange method described above, and will not be repeated here.

[0162] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned face image exchange method when executing the computer program. The electronic device may include any intelligent terminal such as a tablet computer and a car computer.

[0163] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0164] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0165] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the face image exchange method of the embodiment of this application;

[0166] Input / output interface 903, used to implement information input and output;

[0167] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0168] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0169] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0170] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned face image exchange method is implemented.

[0171] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0172] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0173] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0176] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0177] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0182] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for exchanging facial images, characterized in that: The face image exchange method comprises: Acquire a first face image and a second face image; Performing first image encoding on the first face image to obtain a first face information latent vector, and performing second image encoding on the second face image to obtain a second face information latent vector; Diffusion processing is performed on the first face information latent vector and the second face information latent vector through a preset diffusion model to obtain a target fused face information latent vector; wherein the target fused face information latent vector is used to represent the identity information of the first face image and the background information of the second face image; Generate a face according to the target fused face information latent vector to obtain a first fused face image; Performing transparency processing on the first face image to obtain a transparent face attribute map; Image fusion is performed based on the first fused face image and the face attribute transparent map to obtain a target face image.

2. The facial image exchange method according to claim 1, characterized in that: The step of performing diffusion processing on the first face information latent vector and the second face information latent vector by using a preset diffusion model to obtain a target fused face information latent vector includes: Using the diffusion model to perform identity information diffusion on the first face information latent vector to obtain an identity information latent vector, and using the diffusion model to perform background information diffusion on the second face information latent vector to obtain a background information latent vector; Feature fusion is performed according to the identity information latent vector and the background information latent vector to obtain the target fused face information latent vector.

3. The facial image exchange method according to claim 2, characterized in that: The step of using the diffusion model to diffuse the identity information of the first face information latent vector to obtain the identity information latent vector, and using the diffusion model to diffuse the background information of the second face information latent vector to obtain the background information latent vector, includes: Performing a first noise addition process on the first face information latent vector according to a preset first noise coefficient to obtain a first face information noisy latent vector; Performing a second noise addition process on the second face information latent vector according to a preset second noise coefficient to obtain a second face information noisy latent vector; wherein the first noise coefficient is not equal to the second noise coefficient; Performing a first denoising process on the first face information denoised latent vector to obtain the identity information latent vector; A second denoising process is performed on the second facial information denoised latent vector to obtain the background information latent vector.

4. The facial image exchange method according to any one of claims 1 to 3, characterized in that: The step of performing transparency processing on the first face image to obtain a transparent face attribute map includes: Performing semantic segmentation on the first face image to obtain an initial mask image; Performing corrosion expansion on the initial mask image to obtain a target mask image; Gaussian blur is performed on the target mask image to obtain the transparent image of the facial attributes of the human face.

5. The facial image exchange method according to any one of claims 1 to 3, characterized in that: The face attribute transparent map has illumination information, face occlusion information and face angle information of the first face image; The step of performing image fusion according to the first fused face image and the face attribute transparent map to obtain a target face image includes: updating the first fused face image according to the illumination information and the facial occlusion information to obtain a second fused face image; The angle of the second fused face image is adjusted according to the face angle information to obtain the target face image.

6. The facial image exchange method according to any one of claims 1 to 3, characterized in that: Before generating a face according to the target fused face information latent vector to obtain a first fused face image, the face image exchange method further includes: Obtain the potential vector of sample face information; Inputting the sample face information latent vector into a preset initial generative adversarial model to generate a sample face, thereby obtaining a sample face image; Perform loss calculation based on the sample face image to obtain target loss data; Adjusting the model parameters of the initial generative adversarial model according to the target loss data to obtain a target generative adversarial model; The step of generating a face according to the target fused face information latent vector to obtain a first fused face image includes: The target fused face information latent vector is input into the target generative adversarial model to generate a fused face, so as to obtain the first fused face image.

7. The facial image exchange method according to any one of claims 1 to 3, characterized in that: Before performing first image encoding on the first face image to obtain a first face information latent vector, and performing second image encoding on the second face image to obtain a second face information latent vector, the face image exchange method further includes: Performing a first facial correction on the first facial image according to the first facial key points of the first facial image and the preset standard facial key point positions to obtain a first corrected facial image; Performing a second face correction on the second face image according to the second face key points of the second face image and the positions of the standard face key points to obtain a second corrected face image; The first corrected facial image is updated to the first facial image, and the second corrected facial image is updated to the second facial image.

8. A facial image exchange device, characterized in that: The device comprises: An image acquisition module, used to acquire a first face image and a second face image; An image encoding module, configured to perform a first image encoding on the first face image to obtain a first face information latent vector, and to perform a second image encoding on the second face image to obtain a second face information latent vector; A diffusion module, configured to perform diffusion processing on the first face information latent vector and the second face information latent vector through a preset diffusion model to obtain a target fused face information latent vector; wherein the target fused face information latent vector is used to represent the identity information of the first face image and the background information of the second face image; A face generation module, used for performing face generation according to the target fused face information latent vector to obtain a first fused face image; A transparency module, used for performing transparency processing on the first face image to obtain a transparent face attribute map; The image fusion module is used to perform image fusion based on the first fused face image and the face attribute transparent map to obtain a target face image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the facial image exchange method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the facial image exchange method according to any one of claims 1 to 7 is implemented.

Images