Facial expression transfer method and device based on diffusion model, equipment and medium

Abstract

This invention discloses an expression transfer method based on a diffusion model. The method includes acquiring an image to be transferred and an image to be segmented; inputting the image to be transferred and the image to be segmented into a preset diffusion model to obtain expression features corresponding to the image to be transferred and contour features corresponding to the image to be segmented; inputting the expression features and contour features into a preset labeling model to obtain expression semantic labels corresponding to the expression features and contour semantic labels corresponding to the contour features; and performing image fusion based on the expression semantic labels and contour semantic labels to obtain a target face image. This invention extracts expression features and contour features based on a diffusion model, improving the sampling quality of image features and increasing the diversity of captured expression features. Image fusion using expression semantic labels and contour semantic labels reduces the complexity of image fusion, enhances the details of the target face image, and makes the target face image more natural.

Description

Technical Field

[0001] This invention relates to the field of biometrics, and in particular to a method, apparatus, device, and medium for facial expression transfer based on a diffusion model. Background Technology

[0002] Facial expression transfer technology refers to transferring the facial expressions of an input face to a target face through a mapping relationship. This technology not only allows users to control facial expressions in target images or videos by inputting their faces, but also provides data augmentation services for facial recognition tasks.

[0003] Current technologies often use Generative Adversarial Networks (GANs) for semantic image segmentation, primarily through binary segmentation of images using the latent space of the GAN. However, GAN segmentation requires an additional encoder to map the image into the latent space. Furthermore, GANs cannot learn the overall data distribution of the image, resulting in poor feature extraction quality and insufficient diversity in feature capture. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for facial expression transfer based on a diffusion model, in order to solve the problems of poor feature sampling quality and insufficient diversity of captured features in the prior art.

[0005] A facial expression transfer method based on a diffusion model includes:

[0006] Obtain the image to be migrated and the image to be segmented; the image to be migrated and the image to be segmented are different images.

[0007] The image to be transferred and the image to be segmented are respectively input into a preset diffusion model to obtain the facial expression features corresponding to the image to be transferred and the contour features corresponding to the image to be segmented.

[0008] The facial expression features and the contour features are respectively input into a preset label model to obtain the facial expression semantic labels corresponding to the facial expression features and the contour semantic labels corresponding to the contour features;

[0009] Based on the facial expression semantic label and the contour semantic label, the facial expression features and the contour features are fused to obtain the target face image.

[0010] An expression transfer device based on a diffusion model, comprising:

[0011] An image acquisition module is used to acquire the image to be migrated and the image to be segmented; the image to be migrated and the image to be segmented are different images.

[0012] The feature extraction module is used to input the image to be transferred and the image to be segmented into a preset diffusion model to obtain the facial expression features corresponding to the image to be transferred and the contour features corresponding to the image to be segmented.

[0013] The tag acquisition module is used to input the facial expression features and the contour features into a preset tag model respectively, and obtain the facial expression semantic tags corresponding to the facial expression features and the contour semantic tags corresponding to the contour features;

[0014] The expression transfer module is used to perform image fusion on the expression features and the contour features based on the expression semantic labels and the contour semantic labels to obtain a target face image.

[0015] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the above-described expression transfer method based on a diffusion model.

[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described expression transfer method based on a diffusion model.

[0017] This invention provides a method, apparatus, device, and medium for facial expression transfer based on a diffusion model. The method extracts features from the image to be transferred and the image to be segmented using a preset diffusion model, achieving the extraction of facial expression features and contour features. This allows the system to learn the data distribution of the entire image, thereby improving the sampling quality of facial expression and contour features and enriching the diversity of captured facial expression features. A preset labeling model is used to predict labels for the facial expression and contour features, thus assigning semantic labels to them. Based on the facial expression semantic labels and contour semantic labels, the facial expression features are fused into the contour features, achieving facial expression feature transfer and reducing the complexity of image fusion. During the image fusion process, median filtering is used to enhance the details of the target face image, making the target face image more natural. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the application environment of the expression transfer method based on the diffusion model in one embodiment of the present invention;

[0020] Figure 2 This is a flowchart of an expression transfer method based on a diffusion model in one embodiment of the present invention;

[0021] Figure 3 This is a flowchart of step S20 in the expression transfer method based on the diffusion model in one embodiment of the present invention;

[0022] Figure 4 This is a flowchart of step S40 in the expression transfer method based on the diffusion model in one embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of an expression transfer device based on a diffusion model in one embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The expression transfer method based on a diffusion model provided in this invention can be applied to, for example... Figure 1 The application environment is shown. Specifically, this diffusion model-based expression transfer method is applied in a diffusion model-based expression transfer device, which includes, as shown in the example... Figure 1 The client and server shown communicate over a network to address the problems of poor feature sampling quality and insufficient diversity of captured features in existing technologies. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The client, also known as the user terminal, refers to the program that provides local services to the client, corresponding to the server. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0027] In one embodiment, such as Figure 2 As shown, a facial expression transfer method based on a diffusion model is provided, which is then applied to... Figure 1 Taking the server in the example, the following steps are included:

[0028] S10: Obtain the image to be migrated and the image to be segmented; the image to be migrated and the image to be segmented are different images.

[0029] Understandably, the image to be transferred is the image on which the facial expression needs to be transferred. The image to be segmented is the image on which the facial contour needs to be segmented, that is, the facial expression needs to be transferred to this image. The image to be transferred and the image to be segmented can be facial images of two completely different people, i.e., images of two different people, or facial images of the same person with different facial expressions. Furthermore, the images to be transferred and the images to be segmented can be obtained through web scraping technology from different websites or clients, or from different databases, or they can be sent by the user from the client to the server.

[0030] S20: Input the image to be transferred and the image to be segmented into a preset diffusion model to obtain the facial expression features corresponding to the image to be transferred and the contour features corresponding to the image to be segmented.

[0031] Understandably, the preset diffusion model is a pre-trained diffusion model, which is trained on a large amount of image training data using the UNet network as the architecture. Expression features are organ features, texture regions, or predefined feature points in facial images, such as eyes, mouth, nose, and eyebrows. Contour features are used to represent a person's facial contours, which differ from person to person.

[0032] Specifically, after obtaining the images to be transferred and segmented, a preset diffusion model is retrieved from the server. The images to be transferred and segmented are input into the preset diffusion model, which extracts features from them. This involves encoding through a series of convolutional and down-pooling layers, progressively downsampling the images to be transferred and segmented down to the bottom convolutional layer. At the bottom layer, convolution processing yields downsampled features, which are then up-pooled. Skip transfer is used to concatenate the downsampled and upsampled features from each layer. Then, a series of transposed convolutional and up-pooling layers decode the features, progressively upsampling them to obtain pixel-level features of the images to be transferred and segmented. This yields the facial expression features corresponding to the image to be transferred and the contour features corresponding to the image to be segmented. In this way, all facial expression features corresponding to the image to be transferred and contour features corresponding to all images to be segmented can be obtained.

[0033] S30: Input the facial expression features and the contour features into a preset label model to obtain the facial expression semantic label corresponding to the facial expression features and the contour semantic label corresponding to the contour features.

[0034] Understandably, the pre-trained label model is a pre-trained model used to predict labels for facial expression feature categories. This label model can be a multilayer perceptron model. Facial expression semantic labels are labels used to represent facial expression features, such as the eyes in the image to be transferred. The pre-trained label model includes a predetermined number of hidden layers (e.g., 2 or 3), and each hidden layer has a predetermined number of hidden units (e.g., 64 or 128). Contour semantic labels are used to represent the contour features corresponding to the image to be segmented, such as the contour feature being for user 2.

[0035] Specifically, after obtaining all facial expression features and contour features, these features are input into a pre-defined labeling model. This model predicts category labels for all facial expression and contour features, specifically using a multilayer perceptron model. The first hidden layer of the multilayer perceptron model calculates the predictions for all facial expression and contour features using different weight values. These predictions are then passed to the next hidden layer, where different weight values ​​are used for label prediction. After passing through a predetermined number of hidden layers with different weight values, the facial expression semantic label and the contour semantic label corresponding to the contour feature are obtained. In this way, facial expression semantic labels are assigned to all facial expression features. Finally, the contour semantic labels corresponding to the contour features of all images to be segmented are determined using the above method.

[0036] The process involves acquiring a large number of facial images from websites or clients using web scraping technology. A diffusion model is then used to segment the speech in all facial images, resulting in local images of each face. Each local image is assigned a corresponding facial image number, i.e., an actual label. These local images are then fed into a multilayer perceptron (MLP) model for training. The MLP model predicts labels for these local images. The predicted and actual labels for the same local image of the same face are compared. If the labels differ, both the local image and the actual label are fed back into the MLP model for further training. This process continuously improves the accuracy of the MLP model's output, until the prediction loss values ​​for all local images of the face reach a predetermined convergence condition. The converged MLP model is then recorded as the model with the predetermined label.

[0037] S40: Based on the facial expression semantic label and the contour semantic label, perform image fusion on the facial expression features and the contour features to obtain the target face image.

[0038] Understandably, the target face image is obtained by transferring the facial expression features of the image to be transferred to the image to be segmented.

[0039] Specifically, after obtaining all expression semantic labels and contour semantic labels, the expression features are concatenated to the contour features corresponding to the contour semantic labels, i.e., the expressions on the image to be transferred are transferred to the image to be segmented. For example, the nose in the expression features of the image to be transferred is concatenated to the nose position in the image to be segmented, and the eyes in the expression features of the image to be transferred are concatenated to the eye position in the image to be segmented. In this way, all expression features are fused to the position in the image to be segmented based on all expression semantic labels, thus obtaining a fused face image. The concatenated face image is then processed for detail based on the expression features and contour features to make the expression features more consistent with the contour features. Finally, the fused face image is filtered using median filtering, i.e., the coded median value of the fused face image is used to filter the fused face image to obtain the target face image, making the target face image look more natural.

[0040] This invention discloses an expression transfer method based on a diffusion model. This method extracts features from the image to be transferred and the image to be segmented using a preset diffusion model, thereby extracting expression features and contour features. This allows the method to learn the data distribution of the entire image, improving the sampling quality of expression and contour features and increasing the diversity of captured expression features. A preset labeling model is used to predict expression and contour features, thus assigning semantic labels to them. Based on the expression and contour semantic labels, the expression features are fused into the contour features, achieving expression feature transfer and reducing the complexity of image fusion. During image fusion, median filtering is used to enhance the details of the target face image, making it more natural.

[0041] In one embodiment, step S10, i.e., before acquiring the image to be migrated, further includes:

[0042] S101, acquire an initial image and perform noise diffusion processing on the initial image to obtain a noisy initial image.

[0043] Understandably, the initial image can be obtained through web scraping techniques from different websites or clients, or from different databases, or it can be sent by the user from the client to the server. The noisy initial image is obtained by adding Gaussian noise to the initial image and gradually spreading it.

[0044] Specifically, before acquiring the image to be transferred, an initial image is acquired and preprocessed. Specifically, Gaussian noise is added to the initial image in the first time step, and according to the feature diffusion step, the added Gaussian noise is diffused throughout the entire initial image over T-1 time steps. That is, the noise addition process continues T times, generating a series of noisy images x1, x2, ..., x... t In the case of x t-1 Add noise to x t During the process, the standard deviation / variance of the noise is a fixed value β within the interval (0, 1). t The mean is determined by a fixed value β. t And the image data at the current moment x t-1 This is determined by [the formula]. That is, the formula for this diffusion step is [formula]. q(x t |x t-1 ) is Gaussian noise from x t-1 To x t During the process, Gaussian noise follows a distribution. (referring to) β is the mean. t (assuming a Gaussian distribution with variance), thus, it is only necessary to determine a fixed value β for each step based on the initial image. t This allows us to obtain noise images at each time step until diffusion is complete at the last time step, at which point we obtain the initial noise image.

[0045] S102, perform denoising prediction processing on the initial noisy image to obtain the image to be migrated.

[0046] Specifically, after obtaining the initial noise image, a reverse diffusion process is performed on the initial noise image to denoise it. This means that the initial noise image is predicted based on the initial image; in other words, it simply involves reversing the process in step S101 to obtain the initial noise image from q(x). t-1 |x t Sampling from a noisy initial image (N(0,I)) allows us to reconstruct a true initial image from a random Gaussian distribution N(0,I), meaning we can obtain a true initial image from a completely chaotic noisy initial image. Since we need to find the data distribution from the noisy initial image, we require a model p. θ This approximates the conditional probability, thus performing a reverse diffusion process on the initial noisy image. The formula for this reverse diffusion is p. θ (x t-1 |x t ):=N(x t-1 μ θ (x t,t),∑ θ (x t ,t)), where, The predicted average value is parameterized. When performing denoising prediction, it is necessary to adjust x based on the initial image (x0). t To x t-1 The prediction (q(x) t-1 |x t The conditional probability is calculated using x0) to obtain the image to be transferred.

[0047] This invention achieves the acquisition of a noisy initial image by adding Gaussian noise to the initial image and then spreading the Gaussian noise throughout the entire initial image using a feature diffusion step. By performing denoising processing on the noisy initial image based on the initial image, noise is removed from the initial image, thereby enabling the acquisition of the image to be transferred. This improves the accuracy and quality of subsequent facial expression feature extraction.

[0048] In one embodiment, before step S20, that is, before the image to be migrated and the image to be segmented are respectively input into the preset diffusion model, the following steps are included:

[0049] S103, Obtain the image training dataset; the image training dataset includes at least one image training data; one image training data is associated with one sample feature.

[0050] Understandably, image training data can be collected from different websites or databases using web scraping techniques, or it can be sent by the user from the client to the server. An image training dataset is then constructed based on all the acquired image training data. A sample feature is assigned to each image training data point; this sample feature can be an expression feature or a contour feature.

[0051] S104, Obtain a preset training model, and perform prediction processing on the image training data using the preset training model to obtain prediction features.

[0052] Understandably, the preset training model is a model based on the Unet network architecture. The predicted features are obtained by predicting image training data through the preset training model, and these predicted features can be facial expression features or contour features.

[0053] Specifically, a pre-set training model is retrieved from the server, and all acquired image training data is input into the pre-set training model. The pre-set training model first performs downsampling processing on the image training data using at least one set of convolutional layers and upper pooling layers to obtain downsampled features. Then, dimensionality restoration is performed on the downsampled features, i.e., upsampling processing is performed on the downsampled features using at least one set of upper pooling layers and transposed convolutional layers to obtain upsampled features of the image training data. Specifically, during dimensionality restoration of the downsampled features, the features after upper pooling and the convolutional features from the downsampling process are concatenated, and the concatenated features are then subjected to transposed convolution processing, resulting in at least one set of pooling convolution processing to obtain upsampled features. Finally, all upsampled features are subjected to one-dimensional convolution processing to obtain the predicted features.

[0054] S105, determine the prediction loss value of the image training data based on the sample features and the prediction features corresponding to the same image training data.

[0055] Understandably, the prediction loss is generated during the prediction of the predictive features of the image training data, and this prediction loss is used to characterize the difference between the sample features and the predicted features.

[0056] Specifically, after obtaining the predicted features, all sample features corresponding to the image training data are arranged according to the order of the image training data in the image training dataset. Then, the predicted features associated with the image training data are compared with the sample features of the image training data in the same sequence. That is, according to the sorting of the image training data, the sample features corresponding to the first image training data are compared with the predicted features corresponding to the first image training data, and the loss value between the sample features and the predicted features is determined by the loss function. Then, the sample features corresponding to the second image training data are compared with the predicted features corresponding to the second image training data, until all sample features and predicted features have been compared, and the predicted loss value of the image training data can be determined.

[0057] S106, when the predicted loss value does not reach the preset convergence condition, iteratively update the initial parameters in the preset training model until the predicted loss value reaches the convergence condition, and record the converged preset training model as the preset diffusion model.

[0058] Understandably, the convergence condition can be the condition that the predicted loss value is less than a set threshold, that is, when the predicted loss value is less than the set threshold, training stops; the convergence condition can also be the condition that the predicted loss value is very small after 500 calculations and will not decrease further, that is, when the predicted loss value is very small after 500 calculations and will not decrease further, training stops, and the preset training model after convergence is recorded as the preset diffusion model.

[0059] Furthermore, after determining the predicted loss value of the image training data, if the predicted loss value does not reach the preset convergence condition, the initial parameters of the preset training model are adjusted according to the predicted loss value, and all image training data are re-input into the preset training model after adjusting the initial parameters. When the predicted loss value of the image training data reaches the preset convergence condition, other image training data in the image training dataset are selected, and the above steps S103 to S105 are executed to obtain the predicted loss value corresponding to the preset training model with adjusted initial parameters. If the predicted loss value does not reach the preset convergence condition, the initial parameters of the preset training model are adjusted again according to the predicted loss value, so that the predicted loss value of the preset training model with adjusted initial parameters reaches the preset convergence condition.

[0060] Thus, after training the preset training model with all image training data, the output of the preset training model can continuously approach the accurate result, making the prediction accuracy higher and higher, until the prediction loss value of all image training data reaches the preset convergence condition, and the converged preset training model is recorded as the preset diffusion model.

[0061] This invention trains a pre-defined training model using a large amount of image training data and determines the prediction loss value between the predicted features and sample features using a pre-defined loss function. The initial parameters of the pre-defined training model are adjusted based on the prediction loss value until the model converges, thereby obtaining the pre-defined diffusion model and ensuring that the pre-defined diffusion model has a high prediction accuracy.

[0062] In one embodiment, such as Figure 3 As shown, step S20, which involves obtaining the facial expression features corresponding to the image to be transferred, includes:

[0063] S201: The image to be migrated is downsampled using the downsampling module in the preset diffusion model to obtain downsampled features.

[0064] Understandably, the downsampling module includes at least one set of convolutional layers and downpooling layers, which is used to encode the image to be transferred. The downsampling features are obtained by feature extraction from the image to be transferred through the downsampling module.

[0065] Specifically, after obtaining the image to be transferred and the image to be segmented, a preset diffusion model is retrieved from the server. The image to be transferred and the image to be segmented are input into the preset diffusion model sequentially. The downsampling module in the preset diffusion model extracts facial expression features from the image to be transferred. That is, the image to be transferred is downsampled through at least one set of convolutional layers and downpooling layers, which encodes the image to be transferred through at least one set of convolutional layers and downpooling layers, and downsampling is gradually performed downsampled to the lowest layer. At the lowest layer, a 3*3 convolutional kernel is used to output a convolutional matrix with 1024 channels to perform convolution processing on the previous pooling result, thereby obtaining the downsampled features corresponding to the image to be transferred. In this process, each convolutional layer inputs two convolutional features. One convolutional feature is used for pooling processing in the downpooling layer, and the other convolutional feature is used for skip propagation to the upsampling process and is used to connect with the upsampled features.

[0066] S202: The downsampled features are upsampled using the upsampling module in the preset diffusion model to obtain facial expression features.

[0067] Understandably, the upsampling module includes at least one set of transposed convolutional layers and up-pooling layers, which are used to decode the image to be migrated.

[0068] Specifically, after obtaining the downsampled features, the upsampling module in the preset diffusion model performs feature dimension restoration processing on the downsampled features, that is, restores the downsampled features to their original dimensions. The downsampled features are then upsampled using at least one set of transposed convolutional layers and upper pooling layers, i.e., decoded using at least one set of transposed convolutional layers and upper pooling layers. Upper pooling is then applied to the downsampled features, which doubles the number of output channels. The final convolutional feature and the first upper pooling feature are concatenated, and the concatenated fused feature is then convolved. This process, involving at least one step, yields the upsampled features. A one-dimensional convolution is then applied to the upsampled features to obtain the facial expression features. Finally, by upsampling all the downsampled features, all facial expression features are obtained.

[0069] In another embodiment, the contour features corresponding to the image to be segmented can also be obtained by using the above steps S201 to S202. The specific process will not be repeated here.

[0070] This invention achieves semantic segmentation of the image to be transferred by downsampling, thereby acquiring downsampled features. By upsampling the downsampled features, facial expression features are acquired, improving the quality of feature extraction.

[0071] In one embodiment, step S201, namely, downsampling the image to be migrated using the downsampling module in the preset diffusion model to obtain downsampled features, includes:

[0072] S2011, The image to be transferred is processed by convolution through the convolutional layer in the preset diffusion model to obtain convolutional features.

[0073] Understandably, convolutional features are obtained by performing convolution processing on the image to be transferred.

[0074] Specifically, after obtaining the image to be transferred, convolution processing is performed on it. First, a 3x3 convolution kernel is used to output a convolution matrix with 64 channels, resulting in the first convolution result. This first convolution result is then normalized using a Batch Normalization (BN) layer and activated using a ReLU nonlinear function to obtain the first convolutional feature. Next, a second convolution is performed using the first convolutional feature, again with a 3x3 kernel and 64 channels, resulting in the second convolution result. This second convolution result is then normalized using a BN layer and activated using a ReLU nonlinear function to obtain the second convolutional feature. One of these two convolutional features is used for pooling, and the other is used for skip transport and upsampling feature concatenation. For example, the initial image has a grayscale value of 572*572*1. A 3*3*64 convolution matrix is ​​used to perform a first convolution process on the initial image, resulting in a convolution result of 570*570*64. A second convolution process is then performed on the convolution result using a 3*3*64 convolution matrix, resulting in a convolution feature of 568*568*64.

[0075] S2012, the convolutional features are pooled through the lower pooling layer in the preset diffusion model to obtain downsampled features.

[0076] Specifically, after obtaining the convolutional features, max pooling is performed on them. This involves using a 2x2 pooling matrix to pool the features, where the maximum value in each pooling matrix is ​​used as the pooling result. For example, pooling a 568x568x64 convolutional feature using a 2x2 matrix yields a 248x248x64 pooled feature. This process is repeated at least once, i.e., (3x3 convolution + 2x2 pooling) * N times (N can be 1, 2, 3, etc.). In the first 3x3 convolution operation after each pooling operation, the number of 3x3 convolution kernels increases exponentially. At the bottom layer, after the final max pooling, the image becomes 32x32x512. Then, two more 3x3x1024 convolution operations are performed, resulting in a final image size of 28x28x1024, which is then used as the downsampled feature. In this way, all downsampled features in the image to be transferred are obtained.

[0077] This invention employs a convolutional layer to perform two convolutional processes on the image to be transferred, thereby acquiring convolutional features. Then, a lower pooling layer is used to pool these convolutional features, thereby acquiring pooling features. Through at least one set of convolutional pooling processes, downsampled features in the image to be transferred are extracted.

[0078] In one embodiment, step S202, namely, upsampling the downsampled features using the upsampling module in the preset diffusion model to obtain facial expression features, includes:

[0079] S2021, The downsampled features are pooled through the upper pooling layer in the preset diffusion model to obtain the upper pooled features.

[0080] Understandably, the up-pooling feature is obtained by pooling the downsampled feature using a 2*2 pooling matrix.

[0081] Specifically, after obtaining all downsampled features, up-pooling is performed on all downsampled features. That is, a 2*2 pooling matrix is ​​used to pool the downsampled features, and the maximum value in each pooling matrix is ​​used as the pooling result, thereby restoring the dimensionality of the downsampled features. In other words, a 2*2 pooling matrix is ​​used to pool the 28*28*1024 downsampled features to obtain 56*56*512 up-pooled features.

[0082] S2022, The up-pooling features are processed by transposed convolution of the transposed convolution layer in the preset diffusion model to obtain upsampled features.

[0083] S2023, perform one-dimensional convolution processing on all the upsampled features to obtain all the expression features.

[0084] Understandably, the transposed convolutional layer is a convolutional layer in the upsampling process, used to restore the downsampled features to their original dimensions. The transposed convolutional features are obtained by performing a transposed convolution on the downsampled features. The downsampled features are obtained by restoring the upsampled features to their original dimensions.

[0085] Specifically, after obtaining all pooling features, the fourth convolutional features obtained through skip propagation are concatenated with the first pooling features to obtain a concatenated feature of 56*56*1024. This concatenated feature is then subjected to transposed convolution processing; that is, a 3*3*512 convolution matrix is ​​first used to perform a convolution processing on the concatenated feature, resulting in a 54*54*512 transposed convolution result. The first transposed convolution result is then normalized using a BN layer and activated using a ReLU function to obtain the first transposed convolution feature. Next, a 3*3*512 convolution matrix is ​​used to perform a second convolution processing on the first transposed convolution matrix, resulting in a 52*52*256 transposed convolution result. The second transposed convolution result is then normalized using a BN layer and activated using a ReLU function to obtain the second transposed convolution feature.

[0086] Thus, using the same 3x3 convolution kernel, convolution matrices with different numbers of output channels, and pooling matrices with the same 2x2 kernel, the process is repeated at least once, i.e., (3x3 convolution + 2x2 pooling) * N times (N can be 1, 2, 3, etc.). After each up-pooling, the convolutional features transferred via skipping are concatenated with the up-pooled features, and then two more 3x3 convolution operations are performed, significantly reducing the number of 3x3 kernels. After reaching the last layer, two transposed convolutions are performed to obtain a 388*388*54 image, which is then identified as the upsampled feature. Further, all upsampled features are subjected to one-dimensional convolution processing, i.e., a 1x1*2 one-dimensional convolution matrix is ​​used to predict all upsampled features, thereby obtaining the facial expression features corresponding to the image to be transferred.

[0087] In another embodiment, bilinear interpolation is used to restore the dimensionality of the downsampled features during the upsampling process. That is, four points Q11(x1, y1), Q12(x1, y2), Q21(x2, y1) and Q22(x2, y2) are selected first, and interpolation is performed in the X direction to obtain R1(x, y1) and R2(x, y2). The coordinates of R1 are R1(x, y1) = (x2-x) / (x2-x1)Q11 + (x-x1) / (x2-x1)Q21, and the coordinates of R2 are R2(x, y2) = (x2-x) / (x2-x1)Q12 + (x-x1) / (x2-x1)Q22. Interpolating in the Y direction yields P(x, y), whose coordinates are P(x, y) = (y2-y) / (y2-y1)R1 + (y-y1) / (y2-y1)R2. Thus, by restoring the downsampled features to their original dimensions using the above method, the facial expression features can be obtained.

[0088] This invention restores downsampled features to their original dimensions by performing up-pooling on them. The downsampled features are enhanced by fusing the up-pooled and convolutional features. Upsampled features are obtained by performing at least one set of convolutional pooling on the fused features, and facial expression features are extracted through one-dimensional convolution.

[0089] In one embodiment, such as Figure 4 As shown, step S40, which involves image fusion of the facial expression features and the contour features to obtain the target face image, includes:

[0090] S401, based on all the facial expression semantic tags, fuse all the facial expression features into the contour features to obtain the image to be filtered.

[0091] Understandably, the image to be filtered is obtained by fusing facial expression features and contour features.

[0092] Specifically, after obtaining the contour semantic labels, the facial expression features are transferred to the contour features based on all facial expression semantic labels and contour semantic labels, thus achieving the fusion of facial expression features and contour features. The facial expression semantic labels determine the position of the facial expression features in the image to be transferred, and the contour semantic labels determine the position of the contour features in the image to be segmented. Therefore, the extracted facial expression features are filled into the positions of the contour features using all facial expression semantic labels and contour semantic labels. In this way, by filling all facial expression features into the contour features, that is, transferring the facial expression features from the image to be transferred to the image to be segmented, the image to be filtered is obtained.

[0093] S402, the image to be filtered is processed by median filtering to obtain the target face image.

[0094] Understandably, the target face image is obtained by median filtering of the image to be filtered.

[0095] Specifically, after obtaining the image to be filtered, a sampling window is selected, and N (an odd number, set according to the image size) consecutive sampling points are selected on the image to be filtered. Sampling is performed on the N sampling points through the sampling window to obtain N sample values. The minimum and maximum values ​​of all sample values ​​are then removed, and the median value of the remaining sample values ​​is used as the filter value. The image to be filtered is then processed based on this filter value to obtain the target face image.

[0096] This invention achieves expression transfer and reduces the complexity of image fusion by fusing all expression and contour features using all facial expression semantic tags and contour semantic tags. Median filtering is then applied to the image to be filtered, thereby acquiring the target face image and making it more natural.

[0097] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0098] In one embodiment, a diffusion-based expression transfer device is provided, which corresponds one-to-one with the diffusion-based expression transfer method described in the above embodiments. For example... Figure 5 As shown, the expression transfer device based on the diffusion model includes an image acquisition module 11, a feature extraction module 12, a label acquisition module 13, and an expression transfer module 14. Detailed descriptions of each functional module are as follows:

[0099] Image acquisition module 11 is used to acquire the image to be migrated and the image to be segmented; the image to be migrated and the image to be segmented are different images;

[0100] The feature extraction module 12 is used to input the image to be transferred and the image to be segmented into a preset diffusion model to obtain the facial expression features corresponding to the image to be transferred and the contour features corresponding to the image to be segmented.

[0101] The tag acquisition module 13 is used to input the facial expression features and the contour features into a preset tag model respectively, and to obtain the facial expression semantic tags corresponding to the facial expression features and the contour semantic tags corresponding to the contour features.

[0102] The expression transfer module 14 is used to perform image fusion on the expression features and the contour features based on the expression semantic label and the contour semantic label to obtain a target face image.

[0103] In one embodiment, the image acquisition module 11 includes:

[0104] A noise diffusion unit is used to acquire an initial image and perform noise diffusion processing on the initial image to obtain a noise-initiated image;

[0105] The denoising prediction unit is used to perform denoising prediction processing on the initial noisy image to obtain the image to be migrated.

[0106] In one embodiment, the feature extraction module 12 includes:

[0107] The downsampling feature unit is used to downsample the image to be migrated through the downsampling module in the preset diffusion model to obtain downsampling features;

[0108] The upsampling feature unit is used to upsample the downsampling features through the upsampling module in the preset diffusion model to obtain facial expression features.

[0109] In one embodiment, the downsampling feature unit includes:

[0110] The convolutional feature subunit is used to perform convolution processing on the image to be transferred through the convolutional layer in the preset diffusion model to obtain convolutional features;

[0111] The lower pooling subunit is used to perform pooling processing on the convolutional features through the lower pooling layer in the preset diffusion model to obtain downsampled features.

[0112] In one embodiment, the upsampling feature unit includes:

[0113] The upper pooling subunit is used to perform pooling processing on the downsampled features through the upper pooling layer in the preset diffusion model to obtain upper pooled features;

[0114] The transposed convolutional subunit is used to perform transposed convolution processing on the uppooling features through the transposed convolutional layer in the preset diffusion model to obtain upsampled features;

[0115] A one-dimensional convolutional subunit is used to perform one-dimensional convolution processing on all the upsampled features to obtain all the expression features.

[0116] In one embodiment, the expression transfer module 14 includes:

[0117] An image fusion unit is used to fuse all the facial expression features into the contour features based on all the facial expression semantic tags to obtain an image to be filtered.

[0118] The image filtering unit is used to filter the image to be filtered using median filtering to obtain the target face image.

[0119] Specific limitations regarding the diffusion model-based expression transfer device can be found in the limitations of the diffusion model-based expression transfer method described above, and will not be repeated here. Each module in the aforementioned diffusion model-based expression transfer device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0120] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores the data used in the diffusion model-based expression transfer method described in the above embodiments. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a diffusion model-based expression transfer method.

[0121] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the expression transfer method based on the diffusion model described in the above embodiment.

[0122] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the expression transfer method based on the diffusion model described in the above embodiment.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0125] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for expression transfer based on a diffusion model, characterized in that, include: Obtain the image to be migrated and the image to be segmented; The image to be transferred and the image to be segmented are different images; The image to be transferred and the image to be segmented are respectively input into a preset diffusion model to obtain the facial expression features corresponding to the image to be transferred and the contour features corresponding to the image to be segmented. The facial expression features and the contour features are respectively input into a preset label model to obtain the facial expression semantic labels corresponding to the facial expression features and the contour semantic labels corresponding to the contour features; Based on the facial expression semantic label and the contour semantic label, the facial expression features and the contour features are fused to obtain a target face image; wherein, based on the facial expression semantic label corresponding to the facial expression feature, the facial expression feature is spliced ​​onto the contour feature corresponding to the contour semantic label. The step of obtaining the facial expression features corresponding to the image to be transferred includes: The image to be migrated is downsampled using the downsampling module in the preset diffusion model to obtain downsampled features. The downsampled features are upsampled using the upsampling module in the preset diffusion model to obtain facial expression features. 2.The expression transfer method based on a diffusion model according to claim 1, wherein, Before acquiring the image to be migrated, the process also includes: An initial image is acquired, and noise diffusion processing is performed on the initial image to obtain a noisy initial image; The initial noisy image is subjected to denoising prediction processing to obtain the image to be migrated. 3.The expression transfer method based on a diffusion model according to claim 1, wherein, The downsampling module includes at least one set of convolutional layers and downpooling layers; The step of downsampling the image to be migrated through the downsampling module in the preset diffusion model to obtain downsampled features includes: The image to be transferred is processed by convolutional layer in the preset diffusion model to obtain convolutional features; The convolutional features are pooled using the lower pooling layer in the preset diffusion model to obtain downsampled features.

4. The expression transfer method based on the diffusion model as described in claim 1, characterized in that, The upsampling module includes at least one set of transposed convolutional layers and an up pooling layer; The step of upsampling the downsampled features using the upsampling module in the preset diffusion model to obtain facial expression features includes: The downsampled features are pooled using the upper pooling layer in the preset diffusion model to obtain the upper pooled features; The up-pooling features are obtained by transposing the transposed convolutional layer in the preset diffusion model; All the upsampled features are subjected to one-dimensional convolution to obtain all the facial expression features. 5.The expression transfer method based on a diffusion model according to claim 1, wherein, The step of fusing the facial expression features and the contour features to obtain the target face image includes: Based on all the facial expression semantic labels, all the facial expression features are fused into the contour features to obtain the image to be filtered; The target face image is obtained by filtering the image to be filtered using the median filtering method.

6. The expression transfer method based on the diffusion model as described in claim 1, characterized in that, Before inputting the image to be transferred and the image to be segmented into the preset diffusion model, the following steps are included: Obtain an image training dataset; the image training dataset includes at least one image training data; each image training data is associated with a sample feature; Obtain a preset training model, and use the preset training model to perform prediction processing on the image training data to obtain prediction features; The prediction loss value of the image training data is determined based on the sample features and the prediction features corresponding to the same image training data. When the predicted loss value does not reach the preset convergence condition, the initial parameters in the preset training model are iteratively updated until the predicted loss value reaches the convergence condition. Then, the preset training model after convergence is recorded as the preset diffusion model.

7. A facial expression transfer device based on a diffusion model, characterized in that, include: An image acquisition module is used to acquire the image to be migrated and the image to be segmented; the image to be migrated and the image to be segmented are different images. The feature extraction module is used to input the image to be transferred and the image to be segmented into a preset diffusion model to obtain the facial expression features corresponding to the image to be transferred and the contour features corresponding to the image to be segmented. The tag acquisition module is used to input the facial expression features and the contour features into a preset tag model respectively, and obtain the facial expression semantic tags corresponding to the facial expression features and the contour semantic tags corresponding to the contour features; The expression transfer module is used to perform image fusion on the expression features and the contour features according to the expression semantic label and the contour semantic label to obtain a target face image; wherein, according to the expression semantic label corresponding to the expression feature, the expression feature is spliced ​​onto the contour feature corresponding to the contour semantic label. The feature extraction module includes: The downsampling feature unit is used to downsample the image to be migrated through the downsampling module in the preset diffusion model to obtain downsampling features; The upsampling feature unit is used to upsample the downsampling features through the upsampling module in the preset diffusion model to obtain facial expression features.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the expression transfer method based on the diffusion model as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the expression transfer method based on the diffusion model as described in any one of claims 1 to 6.

Images