Facial image replacement method and apparatus, and device, storage medium and program product

Abstract

A facial image replacement method and apparatus, and a device, a storage medium and a program product. The method comprises: acquiring a first sample image, a second sample image and a sample replacement image (210); using sample noise data to perform n instances of noise addition on the sample replacement image in a time dimension, so as to obtain a sample noise-added image (220); in the process of performing facial area replacement on the first sample image and the second sample image by means of a facial image replacement model, performing prediction on the basis of the sample noise-added image, so as to obtain predicted noise data (230); and using the difference between the sample noise data and the predicted noise data to train the facial image replacement model, so as to obtain a trained facial image replacement model (240). By means of the above manner, a facial replacement process can be implemented while removing noise, thus solving the problem of the facial replacement generation effect being relatively poor caused by the relatively low image definition, and improving the robustness of a trained facial image replacement model, which can be applied to various scenarios such as cloud technology, artificial intelligence and smart traffic.

Description

Facial image replacement method, device, equipment, storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 3, 2023, with application number 202310969507.7 and titled “Method, device, apparatus and storage medium for replacing facial images,” the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of computer technology, and in particular to a facial image replacement method, apparatus, device, storage medium, and program product. Background Art

[0003] Facial image replacement has a wide range of applications, including film and television portrait production, game character design, and virtual image creation. For example, in film and television production, when an actor is unable to perform specialized movements, a professional can first perform the movements. Later, the professional's face can be replaced using the facial image replacement process, allowing the production to be completed while ensuring the actor's safety.

[0004] In the related art, after obtaining the first image and the second image for facial image replacement, facial key points are usually extracted from the first image and the second image respectively, and multiple first facial key points corresponding to the first facial area on the first image are determined, and multiple second facial key points corresponding to the second facial area on the second image are determined. Then, based on the image registration results of the first image and the second image, and the correspondence between the multiple first facial key points and the multiple second facial key points, the first facial area on the first image is replaced with the second facial area of ​​the second image, thereby realizing the facial replacement process.

[0005] However, the above method does not fully consider the image clarity of the first image and the second image. When the image clarity of the first image and the second image is low, the accuracy of the extraction results of the above first and second facial key points will also be low. After the face replacement process is implemented according to the correspondence between the first and second facial key points, the quality of the obtained face replacement image is poor, which greatly affects the face replacement effect.

[0006] Summary of the Invention

[0007] The present invention provides a facial image replacement method, apparatus, device, storage medium, and program product that can simultaneously remove noise and perform facial replacement, thereby avoiding the problem of poor facial replacement results due to low image clarity and improving the robustness of the trained facial image replacement model. The technical solution is as follows.

[0008] In one aspect, a method for replacing a facial image is provided, the method comprising:

[0009] Acquire a first sample image, a second sample image, and a sample replacement image, wherein the sample replacement image is an image obtained by replacing a first facial region in the first sample image with a second facial region in the second sample image;

[0010] Noising the sample replacement image n times in the time dimension using the sample noise data, where n is a positive integer to obtain the sample noisy image;

[0011] In the process of performing facial region replacement on the first sample image and the second sample image using the facial image replacement model, predicting noise data based on the sample noisy image;

[0012] The facial image replacement model is trained using the difference between the sample noise data and the predicted noise data to obtain a trained facial image replacement model, wherein the trained facial image replacement model is used to replace a first facial region in a first image with a second facial region in a second image.

[0013] In another aspect, a method for replacing a facial image is provided, the method comprising:

[0014] Acquire a first image and a second image;

[0015] Acquire a noise map, where the noise map is a randomly acquired or pre-set map representing noise data;

[0016] In the process of performing facial image replacement on the first image and the second image using the trained facial image replacement model, the first image and the second image are used as image guidance information, and the first facial area of ​​the first image is replaced with the second facial area of ​​the second image by performing noise reduction processing on the noise map to obtain the facial replacement image.

[0017] In another aspect, a facial image replacement device is provided, the device comprising:

[0018] an image acquisition module, configured to acquire a first sample image, a second sample image, and a sample replacement image, wherein the sample replacement image is an image obtained by replacing a first facial region in the first sample image with a second facial region in the second sample image;

[0019] An image denoising module, configured to perform n-times denoising on the sample replacement image using sample noise data in a time dimension to obtain a sample noisy image, where n is a positive integer;

[0020] a noise prediction module, configured to obtain predicted noise data based on the sample noisy image during the process of performing facial region replacement on the first sample image and the second sample image using a facial image replacement model;

[0021] A model training module is used to train the facial image replacement model using the difference between the sample noise data and the predicted noise data to obtain a trained facial image replacement model, wherein the trained facial image replacement model is used to replace the first facial area in the first image with the second facial area in the second image.

[0022] In another aspect, a facial image replacement device is provided, the device comprising:

[0023] An image acquisition module, configured to acquire a first image and a second image;

[0024] A noise acquisition module is used to acquire a noise map, where the noise map is a randomly acquired or pre-set map representing noise data;

[0025] A denoising processing module is configured to, in a process of performing facial image replacement on the first image and the second image using the trained facial image replacement model, use the first image and the second image as image guidance information, and replace the first facial region of the first image with the second facial region of the second image by performing denoising on the noise map to obtain the facial replacement image.

[0026] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the facial image replacement method as described in any of the above-mentioned embodiments of the present application.

[0027] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the facial image replacement method as described in any of the above-mentioned embodiments of the present application.

[0028] In another aspect, a computer program product or computer program is provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the facial image replacement method described in any of the above embodiments.

[0029] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0030] In the process of replacing the facial region of a first sample image and a second sample image using a facial image replacement model, predicted noise data is obtained based on a sample noisy image after the sample replacement image is noisyed n times with sample noise data, and then the facial image replacement model is trained based on the difference between the sample noise data and the predicted noise data. The process of predicting the noise distribution of the sample noisy image with the help of the facial image replacement model is conducive to enabling the facial image replacement model to learn the noise relationship between the sample image (the first sample image and the second sample image) and the sample noisy image. Combined with the reference replacement situation between the sample replacement image and the first sample image and the second sample image, it is conducive to improving the facial image replacement model's targeted analysis process of noise, and then using the noise prediction process to make targeted adjustments to images with lower clarity, so as to achieve the facial replacement process while removing noise, avoid the problem of poor facial replacement generation effect due to low image clarity, improve the robustness of the trained facial image replacement model, and facilitate the application of the trained facial image replacement model to a wider range of facial replacement scenarios.

[0031] BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] FIG1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0034] FIG2 is a flow chart of a facial image replacement method provided by an exemplary embodiment of the present application;

[0035] FIG3 is a flow chart of a facial image replacement method provided by an exemplary embodiment of the present application;

[0036] FIG4 is a schematic diagram of obtaining a sample replacement image provided by an exemplary embodiment of the present application;

[0037] FIG5 is a schematic diagram of a noise addition process provided by an exemplary embodiment of the present application;

[0038] FIG6 is a flowchart of obtaining predicted noise data provided by an exemplary embodiment of the present application;

[0039] FIG7 is a flowchart of a facial image replacement method provided by an exemplary embodiment of the present application;

[0040] FIG8 is a schematic structural diagram of a facial image replacement model provided by an exemplary embodiment of the present application;

[0041] FIG9 is a schematic diagram of an encoding network and a decoding network provided by an exemplary embodiment of the present application;

[0042] FIG10 is a flowchart of a facial image replacement method provided by an exemplary embodiment of the present application;

[0043] FIG11 is a schematic diagram of a noise reduction process provided by an exemplary embodiment of the present application;

[0044] FIG12 is a structural block diagram of a facial image replacement apparatus provided by an exemplary embodiment of the present application;

[0045] FIG13 is a structural block diagram of a facial image replacement apparatus provided by an exemplary embodiment of the present application;

[0046] FIG14 is a structural block diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0048] The following is a brief introduction to the terms involved in the embodiments of this application.

[0049] In an embodiment of the present application, a sample replacement image representing reference information is obtained, and the sample replacement image is denoised n times in the temporal dimension using sample noise data to obtain a sample noisy image. During facial region replacement of a first sample image and a second sample image using a facial image replacement model, the sample noisy image is used to predict predicted noise data corresponding to the facial region replacement process, and the facial image replacement model is trained based on the difference between the sample noise data and the predicted noise data. Predicting the noise distribution of the sample noisy image using the facial image replacement model facilitates the facial image replacement model to learn the noise relationship between the sample images (including the first sample image and the second sample image) and the sample noisy image. Combined with the reference replacement between the sample replacement image and the first sample image and the second sample image, this facilitates improving the facial image replacement model's targeted noise analysis process. Furthermore, the noise prediction process facilitates targeted adjustments to images with lower clarity, thereby achieving the facial replacement process while removing noise, thereby avoiding the problem of poor facial replacement generation results due to lower image clarity, improving the robustness of the trained facial image replacement model, and facilitating the application of the trained facial image replacement model to a wider range of facial replacement scenarios.

[0050] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant region. For example, the first sample image, second sample image, sample replacement image, facial image replacement model, first image, second image and other contents involved in this application are all obtained with full authorization.

[0051] The following describes the implementation environment involved in the embodiments of the present application. The facial image replacement method provided in the embodiments of the present application can be executed by a terminal alone, by a server, or by a terminal and a server jointly through data exchange, and the embodiments of the present application are not limited thereto. The following description uses the facial image replacement method executed interactively by a terminal and a server as an example.

[0052] Please refer to FIG. 1 , the implementation environment includes a terminal 110 and a server 120 , and the terminal 110 and the server 120 are connected via a communication network 130 .

[0053] In some embodiments, terminal 110 has an image acquisition function for acquiring at least one of a first sample image, a second sample image, and a sample replacement image. The sample replacement image is an image obtained by replacing a first facial region in the first sample image with a second facial region in the second sample image. After acquiring the first and second sample images, terminal 110 performs facial replacement on the first and second sample images using a pre-trained facial image replacement model to obtain the sample replacement image.

[0054] In an embodiment, after acquiring the first sample image, the second sample image, and the sample replacement image, the terminal 110 sends the first sample image, the second sample image, and the sample replacement image to the server 120 via the communication network 130, so that the server 120 obtains the first sample image, the second sample image, and the sample replacement image. Alternatively, after acquiring the first sample image and the second sample image, the terminal 110 sends the first sample image and the second sample image to the server 120 via the communication network 130, and the server 120 performs facial replacement on the first sample image and the second sample image using a pre-trained facial image replacement model to obtain the sample replacement image, thereby allowing the server 120 to obtain the first sample image, the second sample image, and the sample replacement image.

[0055] In some embodiments, the server 120 performs noise addition on the sample replacement image n times in the time dimension using the sample noise data to obtain a sample noisy image, where n is a positive integer.

[0056] In an embodiment, a facial image replacement model 121 is configured on the server 120, which is a model to be trained. The facial image replacement model 121 is trained using a first sample image, a second sample image, and a sample noisy image obtained by noisying the sample replacement image n times.

[0057] In this embodiment, when facial image replacement model 121 performs facial region replacement on a first sample image and a second sample image, predicted noise data is obtained based on the sample noisy image prediction. Furthermore, server 120 trains facial image replacement model 121 using the difference between the sample noise data and the predicted noise data, generating a trained facial image replacement model 121. The trained facial image replacement model 121 is used to replace the facial region in the first image with the second image. The above process is an example of a non-unique scenario for training facial image replacement model 121.

[0058] In some embodiments, terminal 110 transmits the first and second images to be face-replaced to server 120 via a communication network. Server 120 then performs the face-replacement process on the first and second images using a trained facial image replacement model 121, generating a face-replacement image after the face-replacement. This face-replacement image can largely eliminate noise interference that may be present during the face-replacement process, and can generate a high-quality face-replacement image even when the clarity of the first and second images is low. In some embodiments, server 120 can transmit the generated face-replacement image to terminal 110 via communication network 130, so that terminal 110 can render and display the face-replacement image on the terminal 110 screen.

[0059] In an embodiment, the terminals include but are not limited to mobile terminals such as mobile phones, tablet computers, portable laptop computers, intelligent voice interaction devices, smart home appliances, and vehicle-mounted terminals, and may also be implemented as desktop computers, etc. The servers may be independent physical servers, or server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0060] Cloud technology refers to a hosting technology that unifies hardware, applications, networks, and other resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network, information, integration, management platform, and application technologies used in the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand.

[0061] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system.

[0062] In combination with the above-mentioned terminology introduction and application scenarios, the facial image replacement method provided in this application is explained. Taking the application of this method to a server as an example, as shown in FIG2 , the method includes the following steps 210 to 240 .

[0063] Step 210: Acquire a first sample image, a second sample image, and a sample replacement image.

[0064] In an embodiment, the first sample image and the second sample image are images containing facial regions.

[0065] In an embodiment, the first sample image and the second sample image are images pre-stored in an image library; or, the first sample image and the second sample image are images captured using an image capture device; or, the first sample image and the second sample image are images randomly acquired from the Internet.

[0066] In an embodiment, the first sample image and the second sample image may also be image frames in a video work. For example, the first sample image and the second sample image may be image frames pre-selected or randomly selected from a video work such as a film or television work, a variety show, or an animation work.

[0067] In an embodiment, the sample replacement image is an image obtained by replacing a first facial region in a first sample image with a second facial region in a second sample image.

[0068] In an embodiment, the facial region included in the first sample image is referred to as a first facial region, and the facial region included in the second sample image is referred to as a second facial region. A facial region is used to represent an area occupied by a face and is used to refer to an area for face replacement.

[0069] In an embodiment, the first sample image is an image used to replace the second sample image, so the first sample image may be referred to as a source image, and the second sample image may be referred to as a target image.

[0070] In some embodiments, the sample replacement image is an image obtained by a pre-trained facial image replacement model.

[0071] In one embodiment, a pre-trained facial image replacement model is obtained. This model is trained using a large number of sample images and has a facial replacement function, capable of performing a highly accurate facial replacement process based on two sample images. During the training of the facial image replacement model provided in the embodiments of the present application, a first sample image and a second sample image are input into the model to be trained, and a sample replacement image corresponding to the first sample image and the second sample image is obtained. The sample replacement image is used as a reference image so that the model to be trained can be trained based on the sample replacement image.

[0072] In an embodiment, the sample replacement image is annotated with a sample label, and the sample label may be at least one of a true label and a pseudo label.

[0073] In this embodiment, if a sample replacement image obtained through a pre-trained model is used as a reference image, the sample label annotated with the sample replacement image is a pseudo label. After the sample replacement image is obtained through a pre-trained model, the sample replacement image is manually analyzed for accuracy and labeled with a sample label. If the sample replacement image that meets the preset accuracy requirements is used as the reference image, the sample label is then considered the true label.

[0074] In an embodiment, multiple sample images are obtained. If any one of the sample images is used as the first sample image, any sample image other than the first sample image among the multiple sample images can be used as the second sample image, and then a sample replacement image after face replacement is obtained based on the first sample image and the second sample image, that is, a face replacement relationship exists between the sample replacement image and the first sample image and the second sample image.

[0075] In an embodiment, the multiple sample images obtained include sample image A, sample image B, and sample image C. If sample image A is used as the first sample image, then any one of sample image B and sample image C can be used as the second sample image for face replacement with the first sample image. Taking sample image A and sample image B as an example of face replacement, the second sample image is sample image B. Based on the face replacement of sample image A and sample image B, sample replacement image 1 is obtained. Sample replacement image 1 has a face replacement relationship with sample image A and sample image B. Similarly, taking sample image A and sample image C as an example of face replacement, the second sample image is sample image C. Based on the face replacement of sample image A and sample image C, sample replacement image 2 is obtained. Sample replacement image 2 has a face replacement relationship with sample image A and sample image C.

[0076] Similarly, if sample image B is used as the first sample image, any one of sample image A and sample image C can be used as the second sample image for face replacement with the first sample image. Based on the above process, a sample replacement image 3 having a face replacement relationship with sample image B and sample image A can also be obtained, and / or a sample replacement image 4 having a face replacement relationship with sample image B and sample image C can be obtained.

[0077] Taking sample image A and sample image B as an example, sample image A can be used as the first sample image and sample image B as the second sample image, or sample image B can be used as the first sample image and sample image A as the second sample image. This choice will affect the direction of face replacement, and the two resulting sample replacement images will also be different. For example, when sample image A is the first sample image and sample image B is the second sample image, sample image A is replaced with sample image B; when sample image B is the first sample image and sample image A is the second sample image, sample image B is replaced with sample image A. Therefore, the face replacement relationships represented by sample replacement image 1 and sample replacement image 3 are different.

[0078] It is worth noting that the above are merely illustrative examples. The above-mentioned first sample image, second sample image, sample replacement image and other images are all obtained with full authorization, and the embodiments of the present application do not limit this.

[0079] Step 220 , performing n-times noise addition on the sample replacement image in the time dimension using the sample noise data to obtain a sample noisy image.

[0080] In an embodiment, after the sample replacement image is acquired, the sample replacement image is denoised n times in the time dimension using sample noise data, and the image after the nth denoising is referred to as a sample noisy image.

[0081] In an embodiment, the sample noise data is pre-selected noise data.

[0082] In some embodiments, the sample noise data is implemented as a fixed noise value. When the sample replacement image is denoised n times in the time dimension using the sample noise data, the value of the sample noise data used in each denoising process is fixed, that is, the noise difference between any two adjacent noisy images is the sample noise data of a fixed value.

[0083] In some embodiments, the sample noise data is implemented as a noise value having a certain variation pattern. When the sample noise data is used to perform n times of noise addition on the sample replacement image in the temporal dimension, the sample noise data value used in each noise addition process is determined according to a preset variation pattern. That is, the noise difference between any two adjacent noisy images is a value determined based on the preset variation pattern and may be the same or different. For example, the preset variation pattern may indicate that in 10 noise addition processes, the sample noise data used in the first five noise addition processes is a first noise value, and the sample noise data used in the last five noise addition processes is a second noise value different from the first noise value.

[0084] In some embodiments, the sample replacement image is denoised at each of n moments represented by a time series distribution using sample noise data, so that each moment corresponds to a noisy image, the noisy image being an image content determined based on the moment (time series information) and the sample noise data.

[0085] In this embodiment, two adjacent noisy images are noisy images corresponding to two adjacent moments, and thus the two adjacent noisy images are adjacent in the time dimension. For example, if time t5 and time t6 are adjacent in the time dimension, the noisy image corresponding to time t5 among the n noisy images is noisy image P5, and the noisy image corresponding to time t6 among the n noisy images is noisy image P6. Thus, noisy image P5 and noisy image P6 are two adjacent noisy images in the time dimension.

[0086] In some embodiments, when the sample-replaced image is denoised n times in the time dimension, an iterative denoising method with n iterations is employed to denoise the sample-replaced image. For example, at time t1, the sample-replaced image is denoised to obtain a noisy image P1. Subsequently, at time t2, the noisy image P1 is denoised to obtain a noisy image P2, and so on. Thus, after n iterations, the nth noisy image after the nth iteration of denoising is used as the sample noisy image.

[0087] In some embodiments, when a sample-replaced image is noised n times in the time dimension, n time moments are selected based on the time series distribution. At each of these n time moments, the sample-replaced image is noised using a preset noise strategy. The preset noise strategy is determined based on time parameters associated with the time moments and noise parameters associated with the sample noise data. For example, n time moments are selected, and at time t1, the sample-replaced image is noised using the preset noise strategy to obtain a noisy image P1. Subsequently, at time t2, the sample-replaced image is noised using the preset noise strategy to obtain a noisy image P2. Based on the time parameters in the preset noise strategy, the noisy images P1 and P2 corresponding to different time moments t1 and t2 are different. The sample noisy image is the noisy image obtained by noise-processing the sample-replaced image using the preset noise strategy at time tn. Because the preset noise strategy is associated with time tn and the sample noise data, the sample noisy image is associated with time tn and the sample noise data.

[0088] In this embodiment, among n noisy images distributed in a time series, the noise intensity exhibits a positive correlation with the time series distribution. For example, if the time series distribution is time t1, time t2, etc., the noise intensity of the noisy image P2 corresponding to time t2 is higher than the noise intensity of the noisy image P1 corresponding to time t1, and the sample noisy image corresponding to time tn has the highest noise intensity.

[0089] In some embodiments, the noise addition process is implemented by adding noise. In some embodiments, the noise is implemented as Gaussian noise, and the noise addition process is implemented by adding Gaussian noise. The intensity of the Gaussian noise is adjusted by changing a parameter affecting the Gaussian noise, and the parameter includes at least one of a mean value, a standard deviation, a noise intensity, and a smoothing degree.

[0090] Step 230 : In the process of performing facial region replacement on the first sample image and the second sample image using the facial image replacement model, predicted noise data is obtained based on the sample noisy image prediction.

[0091] In this embodiment, the facial image replacement model is a neural network model that improves the Stable Diffusion model. The facial image replacement model retains the Convolutional Networks for Biomedical Image Segmentation (UNet) used in the Stable Diffusion model, implementing the reverse noise reduction process represented by the Stable Diffusion model with the help of the UNet. The specific structure of the facial image replacement model is described below, see Figure 8.

[0092] In the embodiment, n moments are selected based on the time series distribution, each moment corresponds to a noisy image, and each noisy image can be referred to as a noisy image after this noise addition.

[0093] In some embodiments, the noisy image after the n-th noisy process is used in the model training process, that is, the facial image replacement model is trained using the sample noisy image obtained after the n-th noisy process.

[0094] The facial image replacement model is a model to be trained, and is used to perform a facial replacement process between a first sample image and a second sample image during the model training process.

[0095] In an embodiment, in the process of performing facial region replacement on the first sample image and the second sample image using the facial image replacement model, predicted noise data is obtained based on the prediction of the sample noisy image.

[0096] The predicted noise data is used to restore the sample replacement image based on the sample noisy image. In an embodiment, the predicted noise data is used to perform noise reduction processing on the sample noisy image to restore the sample replacement image as much as possible using the first sample image and the second sample image.

[0097] In an embodiment, during facial region replacement of a first sample image and a second sample image using a facial image replacement model, a denoised noise value is predicted when the sample noisy image is subjected to at least one denoising process, and the denoised noise value is used as predicted noise data.

[0098] In some embodiments, during the process of performing n facial region replacements on the first sample image and the second sample image using the facial image replacement model, n predicted noise data are obtained based on the sample noisy image prediction.

[0099] In an embodiment, the predicted noise data is used to represent the predicted results corresponding to each restoration process when gradually restoring the sample noisy image to the sample replacement image. The predicted noise data is used to perform multiple iterative noise reduction processes on the sample noisy image, thereby gradually reducing the noise on the sample noisy image as much as possible using the first sample image and the second sample image, and restoring the sample replacement image.

[0100] For example, the n predicted noise data are prediction results of the noise added in the n noise adding processes, and each predicted noise data is a prediction result of the noise added in the corresponding noise adding process.

[0101] Based on this, there is a corresponding relationship between at least one predicted noise data and the sample noise data, so the facial image replacement model can be trained based on the predicted noise data and the sample noise data.

[0102] In an embodiment, the facial image replacement model is trained with the help of n noisy images, the facial image replacement model is trained once with each noisy image, and the facial image replacement model is iteratively trained n times with the help of n noisy images with a time series distribution relationship, so as to realize the process of iteratively training the facial image replacement model n times under one sample replacement image.

[0103] In the embodiment, taking the process of training a facial image replacement model for the i-th time using the i-th noisy image as an example, since the facial image replacement model is trained to replace facial regions of a first sample image and a second sample image, the i-th training process of the facial image replacement model is the process of replacing facial regions of the first sample image and the second sample image using the facial image replacement model. That is, during the i-th facial region replacement process of the first sample image and the second sample image using the facial image replacement model, the first sample image, the second sample image, and the i-th noisy image are used as inputs to the facial image replacement model. Based on the first sample image, the second sample image, and the i-th noisy image, the predicted noise data corresponding to the i-th facial region replacement process in which the i-th noisy image participates in the model training is determined.

[0104] In an embodiment, when a sample noisy image is known, the goal of learning a facial image replacement model is to learn how much noise to reduce on the sample noisy image so that a more accurate image replacement relationship exists between the noise-reduced image and the first sample image and the second sample image, so as to restore the sample replacement image.

[0105] In an embodiment, the image replacement relationship is used to represent the replacement conditions followed when obtaining a face replacement image using the first sample image and the second sample image. The image replacement relationship includes at least one of multiple image information replacement relationships, such as a facial key point replacement relationship, a background replacement relationship, and an expression replacement relationship.

[0106] In an embodiment, the facial key point replacement relationship between the first sample image and the second sample image is implemented as follows: in the desired facial replacement image, the facial key points corresponding to the first facial area in the first sample image need to be replaced with the second facial area in the second sample image.

[0107] The background replacement relationship is implemented as follows: in the face replacement image to be obtained, the image background of the first sample image is not displayed, and the image background of the second sample image is kept displayed.

[0108] The expression replacement relationship is implemented by retaining the facial expression of the second facial area in the second sample image in the desired facial replacement image. For example, the facial expression of the first facial area in the first sample image is calm, and the facial expression of the second facial area in the second sample image is laughing. The expression replacement relationship indicates that the facial expression of the facial replacement image should be laughing.

[0109] In some embodiments, based on the image replacement relationship represented by the first sample image and the second sample image, the facial image replacement model predicts predicted noise data based on the sample noisy image, and the predicted noise data is used to predict the sample replacement image.

[0110] In an embodiment, a facial region replacement process is performed on the first sample image and the second sample image n times by using a facial image replacement model, and predicted noise data corresponding to the n facial region replacement processes are obtained based on the sample noisy image prediction.

[0111] In an embodiment, n predicted noise data are obtained by predicting the sample noisy image, and each predicted noise data is used to restore the noisy image before the noisy process, thereby predicting the sample replacement image using the n predicted noise data. For example, after performing the noisy process on the sample replacement image three times, noisy image 1, noisy image 2, and sample noisy image (noisy image 3) are obtained. Three predicted noise data are obtained by predicting the sample noisy image, representing the predicted noise data when restoring the sample noisy image to noisy image 2, the predicted noise data when restoring the noisy image 2 to noisy image 1, and the predicted noise data when restoring the noisy image 1 to the sample replacement image, etc.

[0112] In an embodiment, the UNet network in the Stable Diffusion model for predicting noise is retained in the facial image replacement model. When the predicted noise data is obtained based on the sample noisy image prediction, the image replacement relationship represented by the first sample image and the second sample image is used to perform a noise prediction process with the help of the UNet network in the facial image replacement model, thereby obtaining the predicted noise data.

[0113] In an embodiment, the facial image replacement model can perform at least one noise reduction process on the sample noisy image based on the predicted noise data and the sample noisy image, thereby predicting the predicted noise data.

[0114] In some embodiments, the predicted noise data is implemented as a predicted noise map. Noise in an image typically manifests as isolated pixels or blocks of pixels that cause a strong visual effect. Therefore, when describing noise in a sample noisy image, a predicted noise map can be used to characterize the noise conditions corresponding to different pixel locations when analyzing the sample noisy image. For example, the image size of the predicted noise map is the same as that of the sample noisy image.

[0115] In the embodiment, taking the predicted noise data as a predicted noise map as an example, since the predicted noise data is obtained after analyzing the sample noisy image under the condition of the image replacement relationship represented by the first sample image and the second sample image, based on the information represented by the image replacement relationship, the noise values ​​corresponding to different pixel positions in the predicted noise map may be different.

[0116] For example, in the facial key point replacement relationship in the image replacement relationship, it is indicated that pixel 1 in the sample noisy image should be realized as information a at the nose in the first sample image. Based on this information a, the noise value of pixel 1' corresponding to pixel 1 in the predicted noise map is determined. The noise value indicates: if the noise value is removed at pixel 1 in the sample noisy image, an image that is more conducive to clearly showing information a at the nose will be obtained.

[0117] In an embodiment, based on the above process, noise values ​​corresponding to multiple pixels are determined using the image replacement relationship, thereby generating a predicted noise map. The multiple pixels in the predicted noise map correspond one-to-one to the multiple pixels in the sample noisy image, and the noise values ​​corresponding to the multiple pixels in the predicted noise map are values ​​determined based on the image replacement relationship, thereby facilitating a more accurate face replacement process using the predicted noise.

[0118] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0119] Step 240 : training the facial image replacement model using the difference between the sample noise data and the predicted noise data to obtain a trained facial image replacement model.

[0120] In an embodiment, after the sample noise data and the predicted noise data are obtained, the difference between the sample noise data and the predicted noise data is determined to obtain a noise loss value.

[0121] In an embodiment, the sample noise data is implemented as a sample noise map, and the predicted noise data is implemented as a predicted noise map. Because both the sample noise map and the predicted noise map are obtained based on the sample noisy image, multiple pixels in the sample noise map correspond one-to-one to multiple pixels in the predicted noise map. Based on the correspondence between the pixels, the difference in pixel values ​​corresponding to each pixel is determined, and thus the noise loss value between the sample noise data and the predicted noise data is obtained based on the difference in pixel values ​​corresponding to the multiple pixels.

[0122] In an embodiment, the facial image replacement model is trained using the noise loss value until a trained facial image replacement model is obtained.

[0123] In some embodiments, model parameters of the facial image replacement model are adjusted based on the noise loss value to obtain an intermediate model; in response to the training of the intermediate model based on the noise loss value reaching a training target, a trained facial image replacement model is obtained.

[0124] In an embodiment, the model parameters of the facial image replacement model are adjusted with the goal of reducing the noise loss value, for example, by using a gradient descent method to reduce the noise loss value; or by using a back propagation algorithm to reduce the noise loss value.

[0125] In an embodiment, during the process of training the intermediate model using the noise loss value, when the training of the intermediate model reaches the training target, a trained facial image replacement model is obtained.

[0126] In an embodiment, in response to the noise loss value reaching a convergence state, the intermediate model obtained by the most recent iterative training is used as the trained facial image replacement model.

[0127] In an embodiment, the noise loss value reaching a convergence state is used to indicate that the value of the noise loss value obtained by the loss function no longer changes or the change amplitude is less than a preset threshold.

[0128] The trained facial image replacement model is used to replace the facial area in the first image with the second image.

[0129] In an embodiment, after obtaining the trained facial image replacement model, the first image and the second image for performing face replacement are input into the trained facial image replacement model, so that the face replacement process is performed using the first image and the second image.

[0130] Among them, the trained facial image replacement model is used to perform a facial replacement process through the image replacement relationship between the first image and the second image during the noise reduction process. Therefore, in addition to inputting the first image and the second image into the trained facial image replacement model, random noise data is also input into the trained facial image replacement model, so that the trained facial image replacement model uses the image replacement relationship between the first image and the second image to perform multiple noise reduction processes on the random noise data, thereby replacing the facial area in the first image with the second image.

[0131] In an embodiment, the image obtained after replacing the facial area in the first image with the second image is called a face replacement image. The face replacement image is an image obtained by performing noise reduction processing on random noise data using the image replacement relationship between the first image and the second image. It can not only avoid noise interference in the face replacement process between the first image and the second image, but also focus on more essential information (information represented by the image replacement relationship) when the first image and the second image have lower clarity, so that the face replacement image has stronger stability.

[0132] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0133] In summary, the process of predicting the noise distribution of the sample noisy image with the help of the facial image replacement model is conducive to enabling the facial image replacement model to learn the noise relationship between the sample image and the sample noisy image. Combined with the reference replacement situation between the sample replacement image and the first sample image and the second sample image, it is conducive to improving the targeted analysis process of the facial image replacement model for noise, and then using the noise prediction process to make targeted adjustments to the image with lower clarity, so as to realize the facial replacement process while removing noise, avoid the problem of poor facial replacement effect due to low image clarity, improve the robustness of the trained facial image replacement model, and facilitate the application of the trained facial image replacement model to a wider range of facial replacement scenarios.

[0134] In an embodiment, during facial region replacement of a first sample image and a second sample image using a facial image replacement model, predicted noise data is obtained using at least one piece of information represented in the first sample image and the second sample image. As shown in FIG3 , the embodiment shown in FIG2 can also be implemented as steps 310 to 360; step 230 can also be implemented as steps 330 to 350.

[0135] Step 310: Acquire a first sample image, a second sample image, and a sample replacement image.

[0136] In an embodiment, the first sample image and the second sample image are images containing facial regions.

[0137] The sample replacement image is an image obtained by replacing the first facial area in the first sample image with the second facial area in the second sample image.

[0138] In some embodiments, before acquiring the first sample image and the second sample image, two images including facial regions are acquired. Considering that the facial region in an image often occupies a relatively small area, after acquiring the image, facial detection is first performed on the image to obtain the facial region. Facial registration is then performed within the facial region to obtain facial key points, including at least key points for the eyes and mouth corners. Furthermore, based on the facial key points, a cropped facial image is obtained and used as the first and second sample images. That is, after preprocessing the image, a facial image with a more distinct facial region is obtained, which is used as the first and second sample images.

[0139] In an embodiment, based on the facial replacement relationship among the first sample image, the second sample image and the sample replacement image, the first sample image and the second sample image for generating the sample replacement image are determined, and the first sample image, the second sample image and the sample replacement image are combined into a triplet. When there are multiple sample images, multiple triplets will be obtained based on this process.

[0140] In an embodiment, it is determined that the first sample image used to generate the sample replacement image 1 is sample image A, and the second sample image is sample image B. The resulting triplet can be expressed as "sample image A-sample image B-sample replacement image 1".

[0141] Figure 4 is a schematic diagram of obtaining a sample replacement image using a first sample image 410 and a second sample image 420. While maintaining the identity represented by the first sample image 410, the expression, angle, and background represented by the second sample image 420 are analyzed. Thus, when the first sample image 410 is replaced with the second sample image 420, the identity of the first sample image is maintained, and information such as the facial features represented in the first sample image 410 is replaced with the second sample image 420, resulting in a sample replacement image 430.

[0142] Step 320 , performing n-times noise addition on the sample replacement image in the time dimension using the sample noise data to obtain a sample noisy image.

[0143] Where n is a positive integer.

[0144] In an embodiment, the sample replacement image is iteratively denoised n times in the time dimension using sample noise data with the same noise value to obtain n noisy images, wherein the nth noisy image is the sample noisy image.

[0145] The noise difference between two adjacent noisy images in the n noisy images is the sample noise data.

[0146] In an embodiment, during n-times iterative denoising of a sample-replaced image, the sample-replaced image is first denoised, and then the denoised sample-replaced image is further denoised. For example, during the first denoising process (e.g., implemented at time t1 representing temporal information), the sample-replaced image is denoised to obtain a noisy image P1; during the second denoising process (e.g., implemented at time t2 representing temporal information), the noisy image P1 is denoised to obtain a noisy image P2, and so on. Thus, after n-times iterative denoising, a sample-noised image is obtained, where the n noisy images are distributed in a temporal sequence.

[0147] In an embodiment, the sample replacement image is iteratively denoised n times in the time dimension using sample noise data with different noise values ​​to obtain n noisy images distributed in a time series, wherein the nth noisy image is the sample noisy image.

[0148] The noise difference between the vth noisy image and the v+1th noisy image is the sample noise data used in the vth iterative noisy process, and v is a positive integer not greater than n.

[0149] In an embodiment, n moments are selected in the time dimension according to the time series distribution; at the n moments, the sample replacement image is denoised using a preset denoising strategy to obtain n noisy images distributed in a time series, wherein the nth noisy image is the sample noisy image.

[0150] In the embodiment, n moments are selected in the time dimension according to the time series distribution, the n moments are different from each other, and different moments represent different time series information.

[0151] In an embodiment, the preset noise addition strategy is implemented as a pre-set noise addition formula. The noise addition formula includes a time parameter representing the temporal information, and the value of the time parameter corresponding to different time points is different. When the sample replacement image is noised using the preset noise addition strategy at n time points, the noisy images obtained after noise addition based on the noise addition formula are also different due to the differences in the n time points.

[0152] In an embodiment, the preset noise addition strategy (noising formula) further includes a noise parameter representing the sample noise data. The noise parameter is implemented as noise intensity. The sample noise data between two adjacent moments is determined based on a preset noise variation pattern and may be the same or different. In an embodiment, when the sample replacement image is subjected to n noise addition processes using the noise addition formula, the value of the noise parameter representing the noise intensity remains unchanged during each noise addition process; alternatively, when the sample replacement image is subjected to n noise addition processes using the noise addition formula, the value of the noise parameter representing the noise intensity may vary between different noise addition processes.

[0153] In an embodiment, the noise addition formula is implemented as the following formula (1).

[0154] yt=αty0+(1-αt)ε (1)

[0155] Among them, yt is used to represent the sample noisy image obtained after noise processing at time t; αt is used to represent the time series information corresponding to time t; y0 is used to represent the sample replacement image; ε is used to represent the noise parameter (e.g., the noise intensity of Gaussian noise).

[0156] In some embodiments, as shown in FIG5 , a schematic diagram of a noise addition process is shown. The noise addition process is performed based on a sample replacement image 510. FIG5 can be implemented as a schematic diagram of iterative noise addition on the sample replacement image 510, or as a schematic diagram of noise addition on the sample replacement image 510 using a preset noise addition strategy.

[0157] In this embodiment, FIG5 is a schematic diagram illustrating iterative noise addition on a sample replacement image 510. In the first noise addition process (step 1), noise 1 is added to the sample replacement image 510 to obtain a noisy image 511. In the second noise addition process (step 2), noise 2 is added to the noisy image 511 to obtain a noisy image 512. In the third noise addition process (step 3), noise 3 is added to the noisy image 512 to obtain a noisy image 513. In the fourth noise addition process (step 4), noise 4 is added to the noisy image 513 to obtain a noisy image 514. If noisy image 514 is the image obtained after the last noise addition process, then noisy image 514 is the sample noisy image.

[0158] It is worth noting that the above four noise addition processes are only illustrative examples, and the number of noise addition processes can be randomly set or pre-set, and is not limited here.

[0159] Step 330 : In the process of performing facial region replacement on the first sample image and the second sample image using the facial image replacement model, facial key point information in the first sample image is obtained.

[0160] In an embodiment, the UNet network in the Stable Diffusion model is retained in the facial image replacement model to implement the reverse noise reduction process represented by the Stable Diffusion model with the help of UNet.

[0161] In an embodiment, the facial image replacement model is trained at least once using a sample noisy image. For example, the facial image replacement model is trained n times using the sample noisy image, representing n processing situations represented by n noisy processes, to implement n iterative training of the facial image replacement model using one sample noisy image.

[0162] In an embodiment, the input of the facial image replacement model includes a first sample image, a second sample image, and a sample noisy image. The facial image replacement model performs noise reduction processing on the sample noisy image based on the first sample image and the second sample image, and learns information during the facial replacement process of the first sample image and the second sample image.

[0163] In an embodiment, the facial image replacement model analyzes the first sample image and obtains facial key point information in the first sample image.

[0164] The facial key point information includes multiple facial key points, and the facial key point information is used to describe the key points of the facial features in the first facial area.

[0165] In an embodiment, after obtaining the first sample image, facial key points corresponding to the first facial region are identified using a facial image replacement model to obtain facial key points representing facial features in the first facial region.

[0166] In an embodiment, multiple facial landmarks can be used to obtain various information about the shape, color, and thickness of facial features. For example, multiple lip landmarks among the multiple facial landmarks can be used to roughly determine information about the lip shape, color, and position relative to the first facial region; or multiple nose landmarks among the multiple facial landmarks can be used to roughly determine information about the nose's shape, height, color, and position relative to the first facial region.

[0167] Step 340: Acquire global image information in the second sample image.

[0168] The global image information includes at least one of image angle information, image background information and facial expression information.

[0169] In an embodiment, the image angle information is used to characterize the orientation of the face in the second facial area in the second sample image, such as the front orientation, the side orientation, etc.; the image background information is used to characterize the background area other than the second facial area in the second sample image; and the facial expression information is used to characterize the facial expression in the second facial area in the second sample image, such as joy, anger, sorrow, happiness, etc.

[0170] In some embodiments, the second sample image is analyzed using a facial image replacement model to obtain global image information corresponding to the second sample image.

[0171] In an embodiment, the image background of the second sample image is analyzed by the facial image replacement model to obtain image background information corresponding to the second sample image; the facial angle of the second sample image is analyzed by the facial image replacement model to obtain image angle information corresponding to the second sample image; and the facial expression of the second sample image is analyzed by the facial image replacement model to obtain facial expression information corresponding to the second sample image.

[0172] Step 350 : Predicting noise data based on at least one of the facial key point information and the global image information, and the sample noisy image.

[0173] In an embodiment, predicted noise data is predicted based on facial key point information and a sample noisy image; or, predicted noise data is predicted based on image global information and a sample noisy image; or, predicted noise data is predicted based on facial key point information, image global information and a sample noisy image.

[0174] In an embodiment, at least one of facial key point information and image global information is used as image guidance information.

[0175] In an embodiment, the purpose of determining the facial key point information and the image global information is to determine the role of the facial key point information and the image global information in synthesizing the facial replacement image through the image replacement relationship between the first sample image and the second sample image.

[0176] For example, facial key point information is used to determine the facial features that characterize the first facial area when synthesizing a facial replacement image; image global information is used to determine the background, expression, facial angle, etc. in the second sample image when synthesizing a facial replacement image.

[0177] In an embodiment, in the process of replacing facial regions of a first sample image and a second sample image through a facial image replacement model, facial key point information is used as image guidance information, or, image global information is used as image guidance information, or, facial key point information and image global information are used as image guidance information.

[0178] The image guidance information is used to determine the noise prediction when performing noise reduction processing on the sample noisy image.

[0179] In an embodiment, facial key points corresponding to the first sample image and global image information corresponding to the second sample image are used as image guidance information, so that the facial image replacement model can predict noise as accurately as possible to more comprehensively present the image guidance information.

[0180] In an embodiment, with the image guidance information as reference information and with the goal of reducing noise in the sample noisy image, predicted noise data is predicted.

[0181] In an embodiment, after selecting at least one information as image guidance information, the image guidance information is used as reference information, which is used as a reference basis for reducing noise in the sample noisy image while learning the first sample image and the second sample image.

[0182] In an embodiment, after determining the image guidance information, the sample noisy image is approached to the image guidance information, so that the image obtained after denoising the sample noisy image can better display the image guidance information (facial key point information and / or image global information) than the sample noisy image as much as possible, and the noise predicted in the approaching process and requiring denoising of the sample noisy image is called predicted noise data, which is the predicted noise data determined for the sample noisy image based on the image guidance information as reference information.

[0183] Step 360 : training the facial image replacement model using the difference between the sample noise data and the predicted noise data to obtain a trained facial image replacement model.

[0184] In an embodiment, after obtaining sample noise data and predicted noise data, the difference between the sample noise data and the predicted noise data is determined to obtain a noise loss value; the facial image replacement model is trained using the noise loss value until a trained facial image replacement model is obtained.

[0185] The trained facial image replacement model is used to replace the facial area in the first image with the second image.

[0186] In some embodiments, random noise data is obtained; the random noise data, a first image, and a second image are input into a trained facial image replacement model; the trained facial image replacement model analyzes the facial key point information corresponding to the first image and / or the image global information corresponding to the second image based on the random noise data; with the help of the image replacement relationship between the first image and the second image, and with the facial key point information corresponding to the first image and / or the image global information corresponding to the first image as guiding information, the random noise data is subjected to multiple noise reduction processes, thereby replacing the facial area in the first image with the second image to obtain a facial replacement image.

[0187] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0188] In the embodiment of the present application, the trained facial image replacement model is obtained from the two perspectives of noise reduction and facial replacement, and has stronger robustness and prediction accuracy, thereby being able to obtain a facial replacement image in which the facial replacement effect is less affected by interference. By using the facial key point information corresponding to the first image and / or the image global information corresponding to the second image as guidance information, the accuracy of the facial image replacement model in obtaining the sample replacement image by restoring the sample noisy image is improved as much as possible. In addition, by analyzing the predicted noise data during the noise reduction process, it is possible to avoid noise interference during the facial replacement process between the first image and the second image, and also to focus on more essential information (information represented by the image replacement relationship) when the first image and the second image have lower clarity, so that the facial replacement image has stronger stability.

[0189] In an embodiment, when predictive noise data is obtained using a facial image replacement model, it is also possible to perform facial landmark analysis on the first sample image and facial region analysis on the second sample image, thereby obtaining the predicted noise data more accurately under certain conditions. As shown in FIG6 , when predictive noise data is obtained by performing facial landmark analysis on the first sample image, step 230 shown in FIG2 can be implemented as follows: steps 611 to 612; when predictive noise data is obtained by performing facial region analysis on the second sample image, step 230 shown in FIG2 can be implemented as follows: steps 621 to 622.

[0190] Step 611: Acquire facial identification information from the first sample image based on the first facial region in the first sample image.

[0191] The facial identification information is used to represent the identity information represented by the first facial region. The following embodiment describes how to obtain the facial identification information.

[0192] In an embodiment, the first sample image is input into a logo recognition network to obtain a first logo feature representation.

[0193] The identification network is a pre-trained neural network.

[0194] In an embodiment, the identification network includes at least one of the following pre-trained models: an Additive Angular Margin Loss for Deep Face Recognition (ArcFace) model, a Large Margin Cosine Loss for Deep Face Recognition (CosFace) model, a Deep Hypersphere Embedding for Face Recognition (SphereFace) model, etc.

[0195] In an embodiment, the first sample image is input into the identification recognition network to obtain a first identification feature representation representing the identity information of the first facial region. For example, the first identification feature representation corresponding to the first sample image, such as a 1*512 dimensional ID feature, is obtained through the net Arc in FIG8 .

[0196] In one embodiment, the first identification feature representation is subjected to nonlinear mapping through a transformer network to obtain an encoded feature representation. For example, a five-layer transformer network is used to perform multi-layer nonlinear mapping on the first identification feature representation. This encoded feature representation is more relevant to the face replacement process.

[0197] In an embodiment, the encoded feature representation is presented in a matrix format, and the number of first sample images input to the identification network is at least one. If the number of first sample images input to the identification network is multiple, for example, a batch includes 20 first sample images, then an additional dimension is added to the matrix format, and different columns in this dimension represent different first sample images.

[0198] In an embodiment, the encoded feature representation is passed through a normalization (layerNorm) layer to normalize the encoded feature representation in the feature dimension to obtain a normalized feature representation. When the matrix-format encoded feature representation is normalized by layerNorm, different eigenvalues ​​of the first sample image are processed. When multiple first sample images are included, the different eigenvalues ​​of each first sample image are processed with the column being the analysis object.

[0199] For example, taking the analysis of the encoded feature representation corresponding to any first sample image x as an example, the eigenvalue corresponding to the first sample image x in the encoded feature representation is determined, and a mean E[X] and a variance Var[x] are calculated based on the multiple eigenvalues ​​corresponding to the first sample image x, and then a normalized feature representation is obtained after the encoded feature representation is normalized, as shown in the following formula (2).

[0200] Among them, γ and β are additional learning parameters; ε is a small parameter that ensures that the variance Var[x] is not zero.

[0201] In an embodiment, the normalized feature representation is passed through a fully connected layer to obtain a second identification feature representation corresponding to the first sample image, and the second identification feature representation is used as facial identification information corresponding to the first sample image.

[0202] In an embodiment, the facial identification information is implemented as a second identification feature representation represented in a matrix form.

[0203] Step 612: Using the facial identification information as reference information and aiming at reducing the noise in the sample noisy image, predict noise data.

[0204] In an embodiment, after determining the facial identification information, the sample noisy image is approached to the facial identification information, so that the image obtained after noise reduction processing of the sample noisy image can better display the facial identification information than the sample noisy image as much as possible, and the noise predicted in the approaching process and requiring noise reduction processing of the sample noisy image is called predicted noise data. The predicted noise data is predicted noise data determined for the sample noisy image based on the facial identification information as guiding information.

[0205] In an embodiment, the facial identification information is implemented as the image guidance information, which is used to determine noise prediction when performing noise reduction processing on a sample noisy image.

[0206] In an embodiment, facial key point information corresponding to the first sample image, image global information corresponding to the second sample image, and facial identification information are used as image guidance information, and the sample noisy image is brought closer to the image guidance information, so that the image obtained after denoising the sample noisy image can better display the image guidance information than the sample noisy image, so that the facial image replacement model can predict the noise that can more comprehensively display the image guidance information as accurately as possible.

[0207] Through the facial identification information acquisition process described in steps 611 to 612 above, face replacement can be performed while ensuring the accuracy of identity information.

[0208] Step 621: Perform facial segmentation on the second sample image to obtain a second facial region corresponding to the second sample image.

[0209] In an embodiment, facial segmentation is performed on the second sample image using a pre-trained image segmentation model, thereby obtaining a second facial region corresponding to the second sample image.

[0210] The image segmentation model may be a high-resolution network HRNet, which is used to determine the second facial region in the second sample image.

[0211] Step 622 : within the region of the second facial region, obtain predicted noise data based on the sample noisy image.

[0212] In an embodiment, after the second facial region is determined, a face replacement image is generated using the first sample image and the second sample image, with the region range of the second facial region as a boundary.

[0213] In an embodiment, facial key point information corresponding to the first sample image and image global information corresponding to the second sample image are used as image guidance information, and a face replacement process is performed within the boundary of the second facial area, so that the sample noisy image is brought closer to the image guidance information, so that the image obtained after denoising the sample noisy image can better display the image guidance information than the sample noisy image. This not only enables the facial image replacement model to predict the noise that can more comprehensively display the image guidance information as accurately as possible, but also enables the facial area in the generated face replacement image to be limited within the second facial area, thereby improving the generation accuracy of the face replacement image.

[0214] In an embodiment, the facial identification information is implemented as the above-mentioned image guidance information, and the facial key point information corresponding to the first sample image, the image global information corresponding to the second sample image, and the facial identification information are used as the image guidance information. The face replacement process is performed within the boundary of the second facial area, and the sample noisy image is brought closer to the image guidance information, so that the image obtained after the noise reduction processing of the sample noisy image can not only better display the image guidance information than the sample noisy image, but also can limit the facial area in the generated face replacement image to within the second facial area, so that the facial image replacement model can predict the noise that can more comprehensively display the image guidance information as accurately as possible, that is, predict the noise data.

[0215] Through the process of determining the second facial region described in steps 621 to 622 , face replacement can be performed while improving region standardization.

[0216] It is worth noting that the above steps 611 to 612 and steps 621 to 622 can be implemented in a sequential relationship (e.g., first executing steps 611 to 612, then executing steps 621 to 622; or first executing steps 621 to 622, then executing steps 611 to 612); or in a parallel relationship (e.g., executing steps 611 to 612; or, executing steps 621 to 622). This embodiment of the present application is not limited to this.

[0217] In an embodiment of the present application, the first sample image and the second sample image are analyzed, and the facial identification information corresponding to the first sample image and / or the image global information corresponding to the second sample image are used as image guidance information. When the first sample image and the second sample image are subjected to facial replacement through a facial image replacement model, the first sample image and the second sample image are analyzed more comprehensively, thereby facilitating the facial image replacement model to learn richer image information, thereby improving the accuracy of the facial image replacement model in acquiring predicted noise data.

[0218] In this embodiment, the facial image replacement model includes an encoding network, a noise prediction network, and a decoding network. The encoding network is used to extract deep feature representations from the first sample image, the second sample image, and the sample noisy image. Taking the example of obtaining predicted noise data from the sample noisy image at the current moment, as shown in Figure 7, step 230 shown in Figure 2 can also be implemented as steps 710 to 740; and step 240 shown in Figure 2 can also be implemented as steps 750 to 770.

[0219] Step 710: Input the sample replacement image into the encoding network to obtain the image feature representation corresponding to the sample replacement image; and perform n-times noise addition on the image feature representation in the time dimension using the sample noise data to obtain the noisy image feature representation.

[0220] In this embodiment, after obtaining sample replacement images corresponding to the first sample image and the second sample image, the sample replacement images are input into an encoding network, which performs feature extraction on the sample replacement images to obtain image feature representations corresponding to the sample replacement images. The image feature representations are then noised n times in the temporal dimension using sample noise data, and the nth noisy image feature representation is used as the noisy image feature representation.

[0221] Step 720: Input the first sample image and the second sample image into the encoding network to obtain sample image feature representations representing the first sample image and the second sample image.

[0222] The sample image feature representation is the feature matrix obtained by combining the first sample feature representation corresponding to the first sample image and the second sample feature representation corresponding to the second sample image using the concat function. The concat function can be used to concatenate the first sample feature representation and the second sample feature representation without changing the two sample feature representations.

[0223] As shown in Figure 8, a schematic diagram of the model structure of the facial image replacement model is shown. When the facial image replacement model is trained, the facial image replacement process is implemented through the first sample image, the second sample image and the sample replacement image.

[0224] In an embodiment, a sample replacement image 830 (image not shown, represented by yt) is input into the encoding network of the facial image replacement model to obtain a noisy image feature representation corresponding to the sample noisy image obtained by performing n-times of noise processing on the sample replacement image 830.

[0225] In an embodiment, a first sample image 810 and a second sample image 820 are input into a facial image replacement model. In an embodiment, the image sizes of the first sample image 810, the second sample image 820, and the sample replacement image 830 are cropped to 512*512. Considering that the first sample image 810, the second sample image 820, and the sample replacement image 830 are color (three-channel) images, each image is represented as 3*512*512. Here, 6*512*512 represents the sample image feature representation obtained by combining the 3*512*512 image of the first sample image 810 and the 3*512*512 image of the second sample image 820 using the concat function. "6" can be understood as concatenating the two three-channel images. scr is the abbreviation for the source represented by the first sample image 810. tar is the abbreviation for the target represented by the second sample image 820. yt is used to represent the sample replacement image 830.

[0226] The encoder network in the facial image replacement model performs deep feature extraction on the sample noisy images after the first sample image 810, the second sample image 820 and the sample replacement image 830 are noisy.

[0227] Figure 9 shows the schematic diagram of the encoding network-decoding network framework. After the original image 910 is input into the encoding network 920 (image encoding network), the encoding network 920 compresses the high-resolution original image and ultimately converts it into a low-dimensional hidden feature representation. The data dimension of this hidden feature representation can be described as a latent space. This latent feature representation is the low-dimensional feature representation obtained after processing by the encoding network. Subsequently, the low-dimensional feature representation is input into the decoding network 930 to restore the high-resolution generated image 940.

[0228] In the facial image replacement model shown in FIG8 , the encoding network and the decoding network are symmetrically arranged. The following embodiment illustrates the network structure between the encoding network and the decoding network in FIG8 .

[0229] As shown in Figure 8, after deep feature extraction is performed on the first sample image 810 and the second sample image 820 through the encoding network, a sample image feature representation representing the first and second sample images is obtained, namely, 8*64*64. The sample image feature representation is the feature matrix obtained by combining the first sample feature representation corresponding to the first sample image and the second sample feature representation corresponding to the second sample image using the concat function. That is, the sample image feature representation 8*64*64 is the feature matrix obtained by combining the first sample feature representation 4*64*64 and the second sample feature representation 4*64*64 using the concat function. To reduce memory consumption and computational complexity, the UNet network shown in Figure 8 operates in a low-dimensional latent space. Latent space is a low-dimensional space used to represent data in machine learning. It refers to a compressed representation of all useful information contained in the data. Latent space can be generated by various methods, such as principal component analysis (PCA) and deep neural networks (DNN). PCA is a linear dimensionality reduction method that generates a latent space by finding linear combinations of the original data. DNN is a nonlinear dimensionality reduction method that generates a latent space by learning nonlinear relationships in the data. Therefore, the latent space is typically lower dimensional than the original data space. For example, the first sample image 810 and the second sample image 820 are both three-channel images of 512*512 pixels, represented as 3*512*512 pixels respectively. After compression at the encoding layer, they are represented as 4*64*64 pixels in the latent space, resulting in a sample image feature representation of 8*64*64 pixels representing the first and second sample images.

[0230] In addition, after deep feature extraction is performed on the sample noisy image after the sample replacement image 830 is noisy through the encoding network, a noisy image feature representation representing the sample noisy image is obtained, that is, 4*64*64.

[0231] In some embodiments, as shown in FIG8 , a pre-trained face-changing network 800 is shown, which is used to obtain a sample replacement image based on a first sample image 810 and a second sample image 820 .

[0232] In an embodiment, the faces of the first sample image 810 and the second sample image 820 are replaced by a pre-trained face-changing network 800 to obtain a sample replacement image; then, the sample replacement image is denoised n times in the time dimension to obtain a sample noisy image, where n is a positive integer.

[0233] Step 730 : resize the second facial region in the second sample image to obtain a mask feature representation corresponding to the second facial region.

[0234] In an embodiment, the second sample image 820 is input into a pre-trained image segmentation network (i.e., the above-mentioned image segmentation model, not shown in FIG8 ) to obtain a second facial region (mask) corresponding to the face, and the mask is used to represent the area where the face needs to be replaced.

[0235] In an embodiment, the mask is resized to obtain an image of size 64*64, which is represented as 1*64*64. 1*64*64 is called mask feature representation, and 1 represents a single channel.

[0236] Step 740: Input the sample image feature representation, the noisy image feature representation, and the mask feature representation into a noise prediction network to obtain predicted noise data.

[0237] As shown in FIG8 , the sample image feature representation 8*64*64, the noisy image feature representation 4*64*64, and the mask feature representation 1*64*64 are input into the noise prediction network to perform noise prediction using the sample image feature representation 8*64*64, the mask feature representation 1*64*64, and the noisy image feature representation 4*64*64.

[0238] In this embodiment, the sample image feature representation 8*64*64 and the mask feature representation 1*64*64 are used as image guidance information, and the noisy image feature representation 4*64*64 is approximated to the image guidance information to predict noise data. The image guidance information is input into the reference information of the UNet network.

[0239] In an embodiment, the sample image feature representation and the mask feature representation are used as image guidance information to determine a first feature distance between the noisy image feature representation and the sample image feature representation, and a second feature distance between the noisy image feature representation and the mask feature representation. The feature distance refers to the distance between feature vectors represented by the two image feature representations.

[0240] In an embodiment, the sample image feature representation and the mask feature representation are used as image guidance information to bring the noisy image feature representation and the sample image feature representation closer together while also bringing the noisy image feature representation and the mask feature representation closer together. In an embodiment, a first characteristic distance between the noisy image feature representation and the sample image feature representation, and a second characteristic distance between the noisy image feature representation and the mask feature representation, are determined in a vector space.

[0241] In some embodiments, the predicted noise data is obtained with the goal of reducing the first characteristic distance to a first preset threshold and the second characteristic distance to a second preset threshold.

[0242] In an embodiment, while reducing the first characteristic distance, the second characteristic distance is also reduced, thereby predicting the predicted noise data; or, the sum of the first characteristic distance and the second characteristic distance is determined, with the goal of reducing the sum of the first characteristic distance and the second characteristic distance to reach a preset threshold, thereby predicting the predicted noise data.

[0243] In an embodiment, the facial image replacement model further includes a logo acquisition layer; the logo acquisition layer performs logo analysis on the first sample image to obtain an image logo feature representation corresponding to the first sample image.

[0244] As shown in FIG8 , the facial image replacement model further includes an identification acquisition layer, which is used to analyze the identity of the first sample image 810 .

[0245] In an embodiment, the identification acquisition layer uses a pre-trained ArcFace model to perform identity (ID) analysis, and is therefore represented as ID ArcFace in Figure 8. Acquiring the image identification feature representation corresponding to the first sample image through ID ArcFace includes the following four steps.

[0246] (1) After the pre-trained identification recognition network (net Arc), the ID feature of 1*512 dimension (the first identification feature representation) is obtained;

[0247] (2) The ID features are subjected to multi-layer nonlinear mapping through a mapper (a 5-layer transformer network) to obtain an encoded feature representation, which is more relevant to the face-swapping process;

[0248] (3) Through the layerNorm layer, the encoded feature representation is normalized in the feature dimension to obtain the normalized feature representation;

[0249] (4) Since the image identification feature representation needs to be analyzed through the noise prediction network, and the condition module of the noise prediction network requires the input size of the feature representation to be 768, that is, corresponding to a 768-dimensional vector, the normalized feature representation can be mapped to 768 dimensions through a fully connected layer, that is, Proj_out to 768 dimensions, so that the second identification feature representation corresponding to the first sample image is represented as 1*1*768. Among them, "1*1" is the padded dimension.

[0250] Among them, the second identification feature representation is used to characterize the facial identification information of the first sample image. For example, the second identification feature representation is a feature representation determined based on the distribution of facial features, facial width and size, etc., and is used to more accurately confirm the identity of the person shown in the first sample image. For different images of the same person, the value of each of the 768 dimensions represented by the second identification feature is the same or basically the same, while for images of different people, the value of each of the 768 dimensions represented by the second identification feature is significantly different. Therefore, the second identification feature representation can be used as the facial identification information corresponding to the first sample image to characterize the identity information of the first facial area. The first sample feature representation corresponding to the first sample image is used to characterize the image information of the first sample image as a whole, including not only the identity information of the person shown in the first sample image, but also background information such as the building and location shown in the first sample image.

[0251] In the embodiment, the noise prediction network uses a UNet network in a stable diffusion, and the image identification feature representation 1*1*768 is input into the UNet network as a condition of the UNet network. The condition here can also be understood as the input data of the UNet network.

[0252] In an embodiment, the image identification feature representation, the sample image feature representation, the noisy image feature representation, and the mask feature representation are input into a noise prediction network to obtain predicted noise data.

[0253] In the embodiment, the second identification feature representation 1*1*768, the sample image feature representation 8*64*64 and the mask feature representation 1*64*64 are used as image guidance information, and the noisy image feature representation 4*64*64 is approached to the image guidance information to predict the predicted noise data.

[0254] In an embodiment, the sample image feature representation, the mask feature representation and the second identification feature representation are used as image guidance information to determine a first feature distance between the noisy image feature representation and the sample image feature representation, a second feature distance between the noisy image feature representation and the mask feature representation, and a third feature distance between the noisy image feature representation and the second identification feature representation.

[0255] In an embodiment, the sample image feature representation and the mask feature representation are used as image guidance information, and while making the noisy image feature representation and the sample image feature representation as close as possible, the noisy image feature representation and the mask feature representation are made close, and the noisy image feature representation and the second identification feature representation are made close.

[0256] In some embodiments, the predicted noise data is obtained with the goal of reducing the first characteristic distance to a first preset threshold, the second characteristic distance to a second preset threshold, and the third characteristic distance to a third preset threshold.

[0257] In an embodiment, while reducing the first characteristic distance, the second characteristic distance and the third characteristic distance are also reduced, thereby predicting the predicted noise data; alternatively, the sum of the first characteristic distance, the second characteristic distance and the third characteristic distance is determined, with the goal of reducing the sum of the first characteristic distance, the second characteristic distance and the third characteristic distance to reach a preset threshold, thereby predicting the predicted noise data.

[0258] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0259] Step 750: Obtain a noise loss value based on the difference between the sample noise data and the predicted noise data.

[0260] In an embodiment, the predicted noise data and the sample noise data when performing noise addition processing on the sample replacement image are acquired, and the difference between the sample noise data and the predicted noise data is determined as the noise loss value.

[0261] In some embodiments, the sample image feature representation, the noisy image feature representation and the mask feature representation are input into the noise prediction network to obtain n predicted noise data, and the n predicted noise data correspond one-to-one to the n noisy processes. Each of the n noisy processes corresponds to a sample noise data, so the n noisy processes correspond to n sample noise data, and the n predicted noise data correspond one-to-one to the n sample noise data.

[0262] In this embodiment, when the n sample noise data are identical, it means that during n iterative noise additions to the sample replacement image, the noise values ​​of the sample noise data used in each addition are identical. Based on the differences between the n predicted noise data and the sample noise data, n noise loss values ​​are obtained.

[0263] In this embodiment, when the n sample noise data are different, this indicates that during n iterative noise additions to the sample replacement image, the noise values ​​of the sample noise data used in each addition may differ. A one-to-one correspondence is determined between the n predicted noise data and the n sample noise data. Based on the correspondence, the difference between the predicted noise data and the corresponding sample noise data is determined to obtain a noise loss value, and n noise loss values ​​are determined.

[0264] In some embodiments, when the n sample noise data are different, the n sample noise data can also be determined using the noise difference relationship between the n noisy images. For example, the first sample noise data is determined based on the sample replacement image and the first noisy image; the i-th sample noise data is determined based on the i-th noisy image and the i+1-th noisy image; the n-th sample noise data is determined based on the n-1-th noisy image and the n-th noisy image, and so on, where i is a positive integer not greater than n.

[0265] The following formula (3) is the loss function calculation formula for the noise loss value.

[0266] L=E t,y0,ε ||εθ(y t ,Xsrc,Xtar,Xmask,c,t)-ε|| 2 (3)

[0267] Among them, L is used to represent the noise loss value; t is used to represent the temporal information, which is regarded as the facial region replacement process at any moment; y0 represents the sample noisy image; ε is the noise; E t,y0,ε It is used to represent the mean square error calculated based on the time series information t, the sample noise image y0 and the noise ε; εθ is used to represent the noise prediction network (UNet network), εθ(y t ,Xsrc,Xtar,Xmask,c,t) is used to represent the sample noise image y t After the first sample image Xsrc, the second sample image Xtar, the second facial area Xmask, the image identification feature representation c, and the time series information t are input into the UNet network, the noise loss value under the time series information t is predicted with the second facial area Xmask and the image identification feature representation c as conditions (input) and the first sample image Xsrc and the second sample image Xtar as image guidance information (it can also be called the second facial area Xmask, the image identification feature representation c, the first sample image Xsrc and the second sample image Xtar as image guidance information); ε is used to characterize the sample noise data.

[0268] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0269] Step 760 , training the noise prediction network in the facial image replacement model with the noise loss value, and obtaining the trained noise prediction network when the noise loss value calculated by the loss function reaches a convergence state.

[0270] In an embodiment, the facial image replacement model is trained n times to obtain n noise loss values; the noise prediction network in the facial image replacement model is trained with the n noise loss values, and the trained noise prediction network is obtained after the n training times.

[0271] As shown in FIG8 , the noise prediction network UNet in the facial image replacement model is trained n times using sample noisy images to obtain a trained noise prediction network.

[0272] Step 770: Use the facial image replacement model including the trained noise prediction network as the trained facial image replacement model.

[0273] In an embodiment, a facial image replacement model including a trained noise prediction network is referred to as a trained facial image replacement model.

[0274] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0275] In an embodiment of the present application, the structure of the facial image replacement model is described. After deep feature extraction of the first sample image and the second sample image is performed with the help of the encoding network, the hidden feature representation of the first sample image and the second sample image is obtained, and then based on the sample image feature representation, the noisy image feature representation, the mask feature representation and the image identification feature representation, a comprehensive analysis of the first sample image and the second sample image is achieved. In this way, with the help of the UNet network in the stable diffusion model, the sample image feature representation, the mask feature representation and the image identification feature are used as image guidance information (conditions), and the noisy image feature representation is analyzed more specifically, and the noise prediction network is trained more comprehensively with the help of the predicted noise data to obtain the trained facial image replacement model.

[0276] In an embodiment, after obtaining a trained facial image replacement model, a facial replacement process is performed on the first and second images using the trained facial image replacement model. Using the obtained noise map, the first and second images are used as guidance information, and a noise reduction process is performed on the noise map to obtain a more stable facial replacement image. As shown in FIG10 , step 240 shown in FIG2 may further include steps 1010 to 1060.

[0277] Step 1010: Acquire a first image and a second image.

[0278] The first image is used to replace the first facial region with the second facial region of the second image.

[0279] In an embodiment, the first image is a source image, the second image is a target image, and a face replacement image is generated by the first image and the second image. In the face replacement image, it is necessary to retain the facial features information, identity information, etc. on the first image as accurately as possible, and retain the expression information, background information, angle information, etc. on the second image as much as possible.

[0280] In an embodiment, the first image and the second image are two randomly selected images containing facial regions, such as the first image and the second image captured by a capture device, or captured by video screenshots, or downloaded from the Internet.

[0281] Step 1020: Obtain a noise map.

[0282] In an embodiment, the noise map is a randomly acquired map representing noise data; or, the noise map is a preset map representing noise data, etc.

[0283] The noise map is used to replace the first facial region with the second facial region of the second image through noise reduction processing.

[0284] In some embodiments, the image size of the first image is the same as the image size of the second image. For example, the image size of the first image and the image size of the second image are the same after cropping.

[0285] In an embodiment, the size of the noise map is the same as the image size of the first image and the second image.

[0286] In an embodiment, in the process of performing facial image replacement on the first image and the second image using the trained facial image replacement model, the first image and the second image are used as image guidance information, and a facial replacement image is obtained based on denoising the noise map.

[0287] In an embodiment, the process of determining predicted noise data related to the noise map and performing noise reduction processing includes the following steps 1030 to 1060 .

[0288] Step 1030 , in a process of performing a first facial image replacement on the first image and the second image using the trained facial image replacement model, the first image and the second image are used as image guidance information, and predicted noise data is obtained based on the noise map.

[0289] The predicted noise data is data representing the noise situation predicted by the trained facial image replacement model.

[0290] In an embodiment, the first image, the second image, and the noise map are used as inputs of a facial image replacement model, and the first image and the second image are used as image guidance information to determine predicted noise data for performing more targeted noise reduction processing on the noise map.

[0291] In an embodiment, facial key point information in the first image and image global information in the second image are obtained, at least one of the facial key point information and the image global information is used as image guidance information, the noise map is approached to the image guidance information, and predicted noise data is predicted.

[0292] In an embodiment, facial identification information and facial key point information in the first image and image global information in the second image are obtained, and the facial identification information, facial key point information and image global information are used as image guidance information, and the noise map is approached to the image guidance information to predict the predicted noise data.

[0293] In an embodiment, facial identification information and facial key point information in a first image, as well as image global information and a second facial region in a second image are obtained, and the facial identification information, facial key point information, image global information and the second facial region are used as image guidance information, and a noise map is brought closer to the image guidance information to predict noise data, etc.

[0294] As shown in Figure 8, after the facial image replacement model is trained, the trained facial image replacement model is obtained. When the trained facial image replacement model is applied, the sample image feature representation 8*64*64 and the noisy image feature representation 4*64*64 output by the encoding network are received through the UNet network, and the size-transformed mask feature representation 1*64*64 and the image identification feature representation 1*1*768 from the identification acquisition layer ID ArcFace are received, thereby obtaining predicted noise data based on multiple feature representation predictions.

[0295] Step 1040 , performing noise reduction processing on the noise image using the predicted noise data to obtain noise-reduced predicted data.

[0296] Figure 11 is a schematic diagram of the noise reduction process. In the initial stage (stage zero), the noise map 1110 and the predicted noise data 1120 are included. The noise map 1110 is subjected to noise reduction processing based on the predicted noise data 1120 to obtain the noise reduction predicted data 1130 in the first stage.

[0297] Step 1050 , in the process of performing the m-th facial image replacement on the first image and the second image using the trained facial image replacement model, the first image and the second image are used as image guidance information, and the m-th predicted noise data is obtained based on the m-1-th noise reduction prediction data.

[0298] Wherein, m is a positive integer greater than 1.

[0299] In an embodiment, after obtaining the predicted noise data and noise reduction prediction data after the first face replacement process, the noise reduction prediction data after the first face replacement process is referred to as the first noise reduction prediction data, and the first noise reduction prediction data is used as the input of the second face image replacement process, that is, the second face image replacement process is performed based on the first noise reduction prediction data, the first image and the second image, so that the first image and the second image are used as image guidance information, and the predicted noise data after the second face replacement process is obtained based on the first noise reduction prediction data, which is referred to as the second predicted noise data.

[0300] The noise image is subjected to denoising processing using the second predicted noise data to obtain the second denoised predicted data.

[0301] Similarly, after obtaining the second predicted noise data and the second noise reduction prediction data, the second noise reduction prediction data is used as the input of the third facial image replacement process, that is, the second facial image replacement process is performed based on the second noise reduction prediction data, the first image and the second image, so that the first image and the second image are used as image guidance information, and the predicted noise data after the third facial replacement process is obtained based on the second noise reduction prediction data, which is called the third predicted noise data, etc.

[0302] Step 1060 : In response to the mth predicted noise data meeting a preset replacement condition, the first facial region on the first image is replaced with the second facial region on the second image to obtain a facial replacement image.

[0303] In an embodiment, the preset replacement condition is a pre-set replacement condition, for example, the preset replacement condition is a preset number of times, such as the preset number of times is 50 times; after obtaining 50 times of predicted noise data, a facial replacement image is obtained after replacing the first facial area with the second facial area of ​​the second image.

[0304] That is, the above process of performing noise reduction processing on the noise map based on the first image and the second image is an iterative process, and the iterative process can also be summarized as the following steps.

[0305] (1) Input the first image Xsrc and the second image Xtar of size 512*512 into the encoder of the trained facial image replacement model to obtain an 8*64*64 image feature representation;

[0306] (2) Input the second image Xtar map into the pre-trained image segmentation network to obtain a 64*64 image region mask;

[0307] (3) Input the first image Xsrc into the IDArcFace network in the trained facial image replacement model to obtain a 1*768-dimensional identification feature representation;

[0308] (4) in the process of performing face replacement on the first image and the second image using the trained face image replacement model for the first time, obtaining a noise map (e.g., a noise map composed of random noise data), inputting the noise map into an encoder, and obtaining a 4*64*64 noise reduction feature representation;

[0309] (5) Inputting the denoising feature representation, image feature representation, image region mask, and identification feature representation into UNet, using the image feature representation, image region mask, and identification feature representation as image guidance information, and obtaining predicted noise data based on the denoising feature representation;

[0310] (6) Processing the noise map with the predicted noise data to obtain the noise-reduced predicted data (prediction result Xt);

[0311] (7) After obtaining the denoised prediction data for the first time, in the process of performing face replacement on the first image and the second image using the trained facial image replacement model, the denoised prediction data is input into the encoder to obtain a 4*64*64 denoised feature representation;

[0312] (8) Inputting the denoising feature representation, the image feature representation, the image region mask, and the identification feature representation into the UNet, using the image feature representation, the image region mask, and the identification feature representation as image guidance information, and obtaining predicted noise data based on the denoising feature representation;

[0313] (9) Process the noise reduction prediction data with the predicted noise data to obtain the next noise reduction prediction data; Xt-1 = Xt-noise, where noise is the predicted noise data;

[0314] (10) Repeat steps 7 to 9 until a preset number of times (e.g., 50 times, 100 times, etc.) is reached to obtain X0;

[0315] (11) Input X0 into the decoder network to obtain a 512*512 face replacement image.

[0316] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0317] In an embodiment, a facial image replacement model includes an encoding network, a noise prediction network, and a decoding network; the encoder performs feature extraction on the first image and the second image; the decoding network obtains predicted noise data based on the feature extraction results, and thus obtains noise reduction prediction data based on the predicted noise data; when the mth predicted noise data meets the preset replacement condition, the decoding network decodes the mth predicted noise data, thereby obtaining a facial replacement image after replacing the first facial area on the first image with the second facial area of ​​the second image.

[0318] In an embodiment, the above method is applied to the face replacement process in a video scene. An image is obtained through video capture, and the image is input into an image segmentation model to determine the facial area and crop the facial area. Then, a stable diffusion-based face replacement process is performed based on the trained facial image replacement model, and a face replacement image is obtained and displayed.

[0319] In actual use, the trained facial image replacement model can interact with other modules. For example, it can first receive image input from the video acquisition module, then perform face detection and crop the facial area, and then perform face replacement using the above method and display it.

[0320] In the embodiments of the present application, by utilizing image guidance information based on the first and second images, a face replacement image is restored from the noise map through noise reduction processing. This can avoid the problem of being unable to accurately and high-quality generate a face replacement image when the first and second images have low image quality. The noise reduction process of the noise map can further reduce noise interference. Even in scenes with large poses and occlusion, a more robust trained face image replacement model can be used to generate more stable face replacement images, thus having a wider range of application scenarios.

[0321] FIG12 is a structural block diagram of a facial image replacement device provided by an exemplary embodiment of the present application. As shown in FIG12 , the device includes the following parts:

[0322] An image acquisition module 1210 is configured to acquire a first sample image, a second sample image, and a sample replacement image, wherein the sample replacement image is an image obtained by replacing a first facial region in the first sample image with a second facial region in the second sample image;

[0323] An image denoising module 1220 is configured to perform n-times denoising on the sample replacement image using sample noise data in a time dimension to obtain a sample noisy image, where n is a positive integer;

[0324] The noise prediction module 1230 is used to obtain predicted noise data based on the sample noisy image during the process of replacing the facial area of ​​the first sample image and the second sample image through the facial image replacement model, and the predicted noise data is used to restore the sample replacement image based on the sample noisy image.

[0325] The model training module 1240 is used to train the facial image replacement model using the difference between the sample noise data and the predicted noise data to obtain a trained facial image replacement model, and the trained facial image replacement model is used to replace the first facial area in the first image with the second facial area in the second image.

[0326] In an embodiment, the noise prediction module 1230 is also used to obtain facial key point information in the first sample image, the facial key point information includes multiple facial key points, and the multiple facial key points are used to describe the key points of the facial features in the first facial area; obtain image global information in the second sample image, the image global information includes at least one of image angle information, image background information and facial expression information; based on the facial key point information and at least one of the image global information, and the sample noisy image, predict the predicted noise data.

[0327] In an embodiment, the noise prediction module 1230 is further used to use at least one of the facial key point information and the image global information as image guidance information, and the image guidance information is used to determine the noise prediction situation when the sample noisy image is subjected to noise reduction processing; under the condition that the image guidance information is used as reference information, with the goal of reducing the noise in the sample noisy image, the predicted noise data is predicted.

[0328] In an embodiment, the noise prediction module 1230 is also used to obtain facial identification information from the first sample image based on the first facial area of ​​the first sample image, and the facial identification information is used to characterize the identity information represented by the first facial area; under the condition that the facial identification information is used as reference information, with the goal of reducing the noise in the sample noisy image, the predicted noise data is predicted.

[0329] In an embodiment, the noise prediction module 1230 is also used to pass the first sample image through an identification recognition network to obtain a first identification feature representation, which is a pre-trained neural network; the first identification feature representation is nonlinearly mapped through a conversion layer network to obtain an encoded feature representation; the encoded feature representation is passed through a normalization layer to normalize the encoded feature representation in the feature dimension to obtain a normalized feature representation; the normalized feature representation is passed through a fully connected layer to obtain a second identification feature representation corresponding to the first sample image, and the second identification feature representation is used as the facial identification information corresponding to the first sample image.

[0330] In an embodiment, the noise prediction module 1230 is further used to perform facial segmentation on the second sample image using a pre-trained image segmentation model to obtain the second facial area corresponding to the second sample image; within the area range of the second facial area, the predicted noise data is obtained based on the sample noisy image prediction.

[0331] In an embodiment, the facial image replacement model includes an encoding network and a noise prediction network;

[0332] The noise prediction module 1230 is also used to input the sample replacement image into the encoding network to obtain the image feature representation corresponding to the sample replacement image; perform n-times noise addition on the image feature representation in the time dimension with the sample noise data to obtain the noisy image feature representation; input the first sample image and the second sample image into the encoding network to obtain the sample image feature representation characterizing the first sample image and the second sample image, the sample image feature representation being a feature matrix obtained by combining the first sample feature representation corresponding to the first sample image and the second sample feature representation corresponding to the second sample image through a concat function; perform size transformation on the second facial area in the second sample image to obtain the mask feature representation corresponding to the second facial area; input the sample image feature representation, the noisy image feature representation and the mask feature representation into the noise prediction network to obtain the predicted noise data.

[0333] In an embodiment, the image replacement model further includes a logo acquisition layer;

[0334] The noise prediction module 1230 is further configured to perform identification analysis on the first sample image through the identification acquisition layer to obtain a second identification feature representation corresponding to the first sample image; and input the second identification feature representation, the sample image feature representation, the noisy image feature representation, and the mask feature representation into the noise prediction network to obtain the predicted noise data.

[0335] In an embodiment, the image denoising module 1220 is further configured to perform n iterative denoising on the sample replacement image in the time dimension to obtain the n noisy images distributed in a time series, wherein the nth noisy image is the sample noisy image.

[0336] In an embodiment, the image denoising module 1220 is further used to select n moments in the time dimension according to a time series distribution; at the n moments, the sample replacement image is denoised according to a preset denoising strategy to obtain the n noisy images distributed in a time series, wherein the nth noisy image is the sample noisy image, and the preset denoising strategy is a strategy determined based on moment parameters related to the moment and noise parameters related to the sample noise data.

[0337] In an embodiment, the model training module 1240 is also used to obtain a noise loss value based on the difference between the sample noise data and the predicted noise data; train the noise prediction network in the facial image replacement model with the noise loss value, and when the noise loss value calculated by the loss function reaches a convergence state, obtain the trained noise prediction network; and use the facial image replacement model including the trained noise prediction network as the trained facial image replacement model.

[0338] FIG13 is a structural block diagram of a facial image replacement device provided by an exemplary embodiment of the present application. As shown in FIG13 , the device includes the following parts:

[0339] An image acquisition module 1310 is configured to acquire a first image and a second image, wherein the first image is used to replace the first facial region onto the second image;

[0340] a noise acquisition module 1320 for acquiring a noise map, wherein the noise map is a randomly acquired or pre-set map representing noise data, and is used to replace the first facial region into the second image through noise reduction processing;

[0341] The noise reduction processing module 1330 is used to replace the first facial region of the first image with the second facial region of the second image by performing noise reduction processing on the noise image during the process of performing facial image replacement on the first image and the second image using the trained facial image replacement model, using the first image and the second image as image guidance information, to obtain the facial replacement image.

[0342] In an embodiment, the noise reduction processing module 1330 is also used to, in the process of performing the first facial image replacement on the first image and the second image using the trained facial image replacement model, use the first image and the second image as image guidance information, and obtain predicted noise data based on the noise map; perform noise reduction processing on the noise map using the predicted noise data to obtain noise reduction prediction data; in the process of performing the mth facial image replacement on the first image and the second image using the trained facial image replacement model, use the first image and the second image as image guidance information, and obtain the mth predicted noise data based on the m-1th noise reduction prediction data, where m is a positive integer greater than 1; in response to the mth predicted noise data reaching a preset replacement condition, replace the first facial area on the first image with the second facial area of ​​the second image to obtain the facial replacement image.

[0343] It should be noted that the facial image replacement device provided in the above embodiment is merely illustrated by the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, i.e., the internal structure of the device can be divided into different functional modules to perform all or part of the functions described above. Furthermore, the facial image replacement device provided in the above embodiment and the facial image replacement method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0344] FIG14 shows a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. The server 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including a random access memory (RAM) 1402 and a read-only memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. The server 1400 also includes a mass storage device 1406 for storing an operating system 1413, application programs 1414, and other program modules 1415.

[0345] The mass storage device 1406 is connected to the central processing unit 1401 through a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1406 and its associated computer-readable media provide non-volatile storage for the server 1400. In other words, the mass storage device 1406 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0346] In embodiments, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The system memory 1404 and mass storage device 1406 described above may be collectively referred to as memory.

[0347] According to various embodiments of the present application, the server 1400 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 1400 may be connected to the network 1412 via the network interface unit 1411 connected to the system bus 1405, or the network interface unit 1411 may be used to connect to other types of networks or remote computer systems (not shown).

[0348] The memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0349] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the facial image replacement method provided by the above-mentioned method embodiments.

[0350] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the facial image replacement method provided by the above-mentioned method embodiments.

[0351] Embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the facial image replacement method described in any of the above embodiments.

[0352] The above descriptions are merely some embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for replacing a facial image, comprising: Acquire a first sample image, a second sample image, and a sample replacement image, wherein the sample replacement image is an image obtained by replacing a first facial area in the first sample image with a second facial area in the second sample image; Noising the sample replacement image n times in the time dimension using the sample noise data to obtain a sample noisy image, where n is a positive integer; In the process of performing facial region replacement on the first sample image and the second sample image by using the facial image replacement model, predicted noise data is obtained based on the sample noisy image prediction, and the predicted noise data is used to restore the sample replacement image based on the sample noisy image; The facial image replacement model is trained with the difference between the sample noise data and the predicted noise data to obtain a trained facial image replacement model, wherein the trained facial image replacement model is used to replace a first facial area in a first image with a second facial area in a second image.

2. The method according to claim 1, wherein: The step of obtaining predicted noise data based on the sample noisy image prediction includes: Acquire facial key point information in the first sample image, wherein the facial key point information includes a plurality of facial key points, and the plurality of facial key points are used to describe key points of facial features in the first facial region; Acquire image global information in the second sample image, where the image global information includes at least one of image angle information, image background information, and facial expression information; The predicted noise data is predicted based on at least one of the facial key point information and the image global information, and the sample noisy image.

3. The method according to claim 2, wherein: The predicting the predicted noise data based on at least one of the facial key point information and the image global information, and the sample noisy image, comprises: using at least one of the facial key point information and the image global information as image guidance information, wherein the image guidance information is used to determine a noise prediction condition when performing a noise reduction process on the sample noisy image; Under the condition that the image guidance information is used as reference information and the noise in the sample noisy image is reduced, the predicted noise data is predicted.

4. The method according to any one of claims 1 to 3, wherein: The step of obtaining predicted noise data based on the sample noisy image prediction includes: Based on the first facial region of the first sample image, acquiring facial identification information from the first sample image, where the facial identification information is used to represent identity information represented by the first facial region; Under the condition that the facial identification information is used as reference information and with the goal of reducing the noise in the sample noisy image, the predicted noise data is predicted.

5. The method according to claim 4, wherein: The acquiring facial identification information from the first sample image based on the first facial region of the first sample image includes: Passing the first sample image through a logo recognition network to obtain a first logo feature representation, wherein the logo recognition network is a pre-trained neural network; The first identification feature representation is nonlinearly mapped through a conversion layer network to obtain an encoded feature representation; The encoded feature representation is passed through a normalization layer to normalize the encoded feature representation in the feature dimension. Perform normalization processing to obtain normalized feature representation; The normalized feature representation is passed through a fully connected layer to obtain a second identification feature representation corresponding to the first sample image, and the second identification feature representation is used as the facial identification information corresponding to the first sample image.

6. The method according to any one of claims 1 to 3, wherein: The step of obtaining predicted noise data based on the sample noisy image prediction includes: Performing facial segmentation on the second sample image by using a pre-trained image segmentation model to obtain the second facial region corresponding to the second sample image; The predicted noise data is predicted based on the sample noisy image within the area of ​​the second facial area.

7. The method according to any one of claims 1 to 3, wherein: The facial image replacement model includes an encoding network and a noise prediction network; The step of adding noise to the sample replacement image n times in the time dimension using the sample noise data to obtain the sample noisy image includes: Inputting the sample replacement image into the encoding network to obtain an image feature representation corresponding to the sample replacement image; performing n-times noise addition on the image feature representation in the time dimension with the sample noise data to obtain a noisy image feature representation; The step of obtaining predicted noise data based on the sample noisy image prediction includes: Inputting the first sample image and the second sample image into the encoding network to obtain a sample image feature representation characterizing the first sample image and the second sample image, wherein the sample image feature representation is a feature matrix obtained by combining a first sample feature representation corresponding to the first sample image and a second sample feature representation corresponding to the second sample image through a concat function; performing a size transformation on the second facial region in the second sample image to obtain a mask feature representation corresponding to the second facial region; The sample image feature representation, the noisy image feature representation and the mask feature representation are input into the noise prediction network to obtain the predicted noise data.

8. The method according to claim 7, wherein: The facial image replacement model also includes a logo acquisition layer; The step of inputting the sample image feature representation, the noisy image feature representation and the mask feature representation into the noise prediction network to obtain the predicted noise data comprises: Performing identification analysis on the first sample image through the identification acquisition layer to obtain a second identification feature representation corresponding to the first sample image; The second identification feature representation, the sample image feature representation, the noisy image feature representation and the mask feature representation are input into the noise prediction network to obtain the predicted noise data.

9. The method according to any one of claims 1 to 3, wherein: The step of adding noise to the sample replacement image n times in the time dimension using the sample noise data to obtain the sample noisy image includes: Iteratively adding noise to the sample replacement image n times in the time dimension using the sample noise data with the same noise value, to obtain n noisy images distributed in a time series, wherein the nth noisy image is the sample noisy image, and the noise difference between two adjacent noisy images in the n noisy images is the sample noise data; or, The sample replacement image is iterated n times with the sample noise data of different noise values ​​in the time dimension to obtain n noisy images distributed in a time series, wherein the nth noisy image is the sample noisy image Image; wherein the noise difference between the vth noisy image and the v+1th noisy image is the sample noise data used in the vth iterative noisy process, and v is a positive integer not greater than n.

10. The method according to any one of claims 1 to 3, wherein: The step of adding noise to the sample replacement image n times in the time dimension using the sample noise data to obtain the sample noisy image includes: Selecting n moments in the time dimension according to the time series distribution; At the n moments, the sample replacement image is denoised using a preset denoising strategy to obtain n noisy images distributed in a time series, wherein the nth noisy image is the sample noisy image, and the preset denoising strategy is a strategy determined based on moment parameters related to the moment and noise parameters related to the sample noise data.

11. The method according to any one of claims 1 to 10, wherein: The facial image replacement model is trained based on the difference between the sample noise data and the predicted noise data to obtain a trained facial image replacement model, comprising: Obtaining a noise loss value based on a difference between the sample noise data and the predicted noise data; The noise prediction network in the facial image replacement model is trained with the noise loss value, and when the noise loss value calculated by the loss function reaches a convergence state, a trained noise prediction network is obtained; The facial image replacement model including the trained noise prediction network is used as the trained facial image replacement model.

12. A method for replacing a facial image, comprising: Acquire a first image and a second image; Acquire a noise map, wherein the noise map is a randomly acquired or pre-set map representing noise data; In the process of performing facial image replacement on the first image and the second image using the trained facial image replacement model, the first image and the second image are used as image guidance information, and the first facial area of ​​the first image is replaced with the second facial area of ​​the second image by performing noise reduction processing on the noise map to obtain the facial replacement image.

13. The method according to claim 12, wherein: In the process of performing facial image replacement on the first image and the second image using the trained facial image replacement model, the first image and the second image are used as image guidance information, and a first facial region of the first image is replaced with a second facial region of the second image by performing noise reduction processing on the noise map to obtain the facial replacement image, including: In a process of performing a first facial image replacement on the first image and the second image using the trained facial image replacement model, taking the first image and the second image as image guidance information, and acquiring predicted noise data based on the noise map; Performing noise reduction processing on the noise map using the predicted noise data to obtain noise reduction prediction data; In the process of performing an m-th facial image replacement on the first image and the second image using the trained facial image replacement model, taking the first image and the second image as image guidance information, and obtaining an m-th predicted noise data based on the m-1-th noise reduction predicted data, where m is a positive integer greater than 1; In response to the m-th predicted noise data reaching a preset replacement condition, the first facial region on the first image is replaced with the second facial region on the second image to obtain the facial replacement image.

14. A facial image replacement device, comprising: An image acquisition module, used to acquire a first sample image, a second sample image, and a sample replacement image, wherein the sample replacement image is an image obtained by replacing a first facial area in the first sample image with a second facial area in the second sample image; An image denoising module, used to denoise the sample replacement image n times in a time dimension using sample noise data to obtain a sample noisy image, where n is a positive integer; A noise prediction module, configured to obtain predicted noise data based on the sample noisy image prediction in the process of performing facial region replacement on the first sample image and the second sample image by using a facial image replacement model; A model training module is used to train the facial image replacement model with the difference between the sample noise data and the predicted noise data to obtain a trained facial image replacement model, wherein the trained facial image replacement model is used to replace a first facial area in a first image with a second facial area in a second image.

15. A facial image replacement device, comprising: An image acquisition module, used to acquire a first image and a second image; A noise acquisition module, used to acquire a noise map, wherein the noise map is a randomly acquired or pre-set map representing noise data; A denoising processing module is used to replace the first facial region of the first image with the second facial region of the second image by performing denoising on the noise map during the process of performing facial image replacement on the first image and the second image using the trained facial image replacement model, taking the first image and the second image as image guidance information, and obtaining the facial replacement image by replacing the first facial region of the first image with the second facial region of the second image.

16. A computer device comprising a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the facial image replacement method according to any one of claims 1 to 13.

17. A computer-readable storage medium, wherein at least one program is stored in the storage medium, and the at least one program is loaded and executed by a processor to implement the facial image replacement method according to any one of claims 1 to 13.

18. A computer program product or a computer program, wherein the computer program product or the computer program comprises computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium, and wherein the processor executes the computer instructions, so that the computer device executes the facial image replacement method as described in any one of claims 1 to 13.

Images