Virtual face generation method and system, electronic equipment and storage medium

Abstract

The invention provides a virtual face generation method and system, electronic equipment and a storage medium, and the method comprises the steps: inputting a to-be-processed face image, and audio features and face key point features corresponding to the to-be-processed face image into a coding module in a face generation model, and obtaining image coding features and lip motion coding features; inputting the image coding features and the lip motion coding features into a feature fusion module in a face generation model to obtain fusion features; inputting the fusion features into a decoding module in a face generation model to obtain a virtual face image; wherein the face generation model is obtained by training an initial image generation model, the parameter scale value of which is smaller than a preset value, based on a sample real face image, a mask face image corresponding to the sample real face image, and sample audio features and sample face key point features corresponding to the mask face image. According to the invention, the virtual face generation quality is guaranteed, and the virtual face generation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a virtual face generation method, system, electronic device and storage medium. Background Art

[0002] In the process of virtual human driving, the accurate generation of virtual human face and synchronization of voice are the key to achieving a natural interactive experience of virtual human. Therefore, how to achieve efficient and accurate virtual human face generation is an important topic that needs to be studied in the process of virtual human driving.

[0003] In order to improve the quality of virtual face generation, related technologies often use model architectures with large computational scale, such as Transformer, Cross Attention model, Diffusion model, etc. to generate virtual faces. Although these methods can perform well in terms of generation quality, the high complexity of the model leads to low efficiency in virtual face generation. Summary of the invention

[0004] The present invention provides a virtual face generation method, system, electronic device and storage medium, which are used to solve the defects of the prior art that the model complexity is high, the reasoning speed is slow, and it is difficult to meet the real-time requirements in practical applications, so as to ensure the quality of virtual face generation while improving the efficiency of virtual face generation.

[0005] The present invention provides a method for generating a virtual face, comprising: Inputting the face image to be processed, and the audio features and face key point features corresponding to the face image to be processed into the encoding module in the face generation model, and obtaining the image encoding features corresponding to the face image to be processed and the lip movement encoding features corresponding to the audio features; Inputting the image coding features and the lip movement coding features into a feature fusion module in the face generation model to obtain fusion features; Inputting the fusion feature into the decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed; the virtual face image is a face image formed by fusing the lip movement image matching the audio feature on the basis of the face image to be processed; Among them, the face generation model is obtained by training the initial image generation model based on the sample real face image, the mask face image corresponding to the sample real face image, and the sample audio features and sample face key point features corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model with a parameter scale value less than a preset value.

[0006] According to a virtual face generation method provided by the present invention, the step of inputting the fusion feature into a decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed includes: Inputting the fused feature into the first feature decoding unit of the decoding module to obtain the first image feature of the virtual face image; the first image feature is used to reconstruct the contour information of the face image to be processed; Inputting the first image feature and the fusion feature into the second feature decoding unit of the decoding module to obtain the second image feature of the virtual face image; the second image feature is used to reconstruct the edge information of the face image to be processed; The first image feature and the second image feature are input into the feature recombination unit of the decoding module to obtain the virtual face image.

[0007] According to a virtual face generation method provided by the present invention, the first feature decoding unit is constructed based on a residual block, an upsampling block and an inverted residual block; The second feature decoding unit is constructed based on an edge-guided convolution block.

[0008] According to a virtual face generation method provided by the present invention, the edge-guided convolution block is constructed based on a standard convolution network and an equivalent convolution network corresponding to a plurality of different convolution operator networks; The equivalent convolutional network is a convolutional network obtained by linearly fusing the model parameters of the multiple different convolutional operator networks.

[0009] According to a virtual face generation method provided by the present invention, the face image to be processed, and the audio features and face key point features corresponding to the face image to be processed are input into a coding module in a face generation model to obtain image coding features corresponding to the face image to be processed and lip movement coding features corresponding to the audio features, including: Inputting the face image to be processed into the image encoder of the encoding module to obtain image encoding features corresponding to the face image to be processed; The audio features and facial key point features corresponding to the facial image to be processed are input into the lip movement encoder of the encoding module to obtain the lip movement encoding features corresponding to the audio features.

[0010] According to a virtual face generation method provided by the present invention, the image encoder is constructed based on a residual block, a downsampling block and an inverted residual block; The lip movement encoder is constructed based on a time series model.

[0011] According to a virtual face generation method provided by the present invention, the step of inputting the image coding feature and the lip movement coding feature into a feature fusion module in the face generation model to obtain the fusion feature includes: Inputting the lip movement coding feature into the feature conversion layer of the feature fusion module to obtain a converted lip movement coding feature; the feature dimension of the converted lip movement coding feature is the same as the feature dimension of the image coding feature; The converted lip movement coding features and the image coding features are input into the feature fusion layer of the feature fusion module to obtain the fusion features.

[0012] According to a virtual face generation method provided by the present invention, the face generation model is trained based on the following steps: Inputting the masked face image, the sample audio features and the sample face key point features into the initial image generation model to obtain sample lip movement coding features corresponding to the sample audio features and a sample virtual face image corresponding to the masked face image; Performing three-dimensional reconstruction on the sample real face image to obtain a lip movement feature label corresponding to the sample real face image; Obtaining a target loss function according to the sample lip movement coding features, the sample virtual face image, the lip movement feature label, and the sample real face image; According to the target loss function, the initial image generation model is iteratively trained to obtain the face generation model.

[0013] According to a virtual face generation method provided by the present invention, the target loss function is obtained according to the sample lip movement coding features, the sample virtual face image, the lip movement feature label, and the sample real face image, including: Based on the discriminator, the sample virtual face image and the sample real face image are applied to perform classification performance perception to obtain a first loss function; Based on the generator, applying the sample virtual face image to generate performance perception to obtain a second loss function; Obtaining a third loss function according to a distribution error between the sample lip movement encoding feature and the lip movement feature label; Obtaining a fourth loss function according to a feature distribution error between the sample virtual face image and the sample real face image; Obtaining a fifth loss function according to a feature distribution error between the mouth area of ​​the sample virtual face image and the mouth area of ​​the sample real face image; Obtaining a sixth loss function according to a reconstruction error between the sample virtual face image and the sample real face image; The target loss function is obtained by combining the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function and the sixth loss function.

[0014] The present invention also provides a virtual face generation system, comprising: A first processing unit is used to input the face image to be processed, and the audio features and face key point features corresponding to the face image to be processed into a coding module in the face generation model, to obtain the image coding features corresponding to the face image to be processed and the lip movement coding features corresponding to the audio features; A second processing unit, configured to input the image coding feature and the lip movement coding feature into a feature fusion module in the face generation model to obtain a fusion feature; A generating unit, configured to input the fusion feature into a decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed; the virtual face image is a face image formed by fusing a lip movement image matching the audio feature on the basis of the face image to be processed; Among them, the face generation model is obtained by training the initial image generation model based on the sample real face image, the mask face image corresponding to the sample real face image, and the sample audio features and sample face key point features corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model with a parameter scale value less than a preset value.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described methods for generating a virtual face is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a virtual face as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned virtual face generation methods.

[0018] The virtual face generation method, system, electronic device and storage medium provided by the present invention generate a face generation model by training an initial image generation model whose parameter scale value is less than a preset value, so that the trained face generation model has fewer network parameters and a faster training convergence speed, thereby ensuring that when performing real-time reasoning of a virtual face image, a virtual face image with clear lips and high resolution can be generated while meeting the real-time requirements of practical applications; on the other hand, during the reasoning process, the face image to be processed and its corresponding audio features and face key point features are input into the encoding module and feature fusion module in the face generation model, so that the encoding module and the feature fusion module use a concise multimodal feature embedding method to unify the speech features and the image features into the same feature space, effectively reducing the use of complex network design, thereby reducing the calculation overhead, and then, the fused features generated by the fusion are input into the decoding module, so that the decoding fuses the lip motion image matching the audio features on the basis of the face image to be processed, so as to form a virtual face image fused with the lip motion image matching the audio features and realistic, thereby further ensuring that the reasoning speed is improved without losing the generation quality of the virtual face image, and meeting the requirements of real-time applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the virtual face generation method provided by the present invention.

[0021] Figure 2 It is a structural schematic diagram of the decoding module provided by the present invention.

[0022] Figure 3 Schematic diagram of the structure of the residual block provided by the present invention.

[0023] Figure 4 It is a structural schematic diagram of the inverted residual block provided by the present invention.

[0024] Figure 5 It is a schematic diagram of the structure of the edge-guided convolution block provided by the present invention.

[0025] Figure 6 It is a structural schematic diagram of the image encoder provided by the present invention.

[0026] Figure 7 It is a schematic diagram of the training process of the face generation model provided by the present invention.

[0027] Figure 8 It is a structural schematic diagram of the virtual face generation system provided by the present invention.

[0028] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In the process of driving a virtual human, accurate generation of virtual faces (such as accurate generation of lips in virtual faces) and synchronization of speech are the key to achieving a natural interactive experience of virtual humans. However, the implementation of virtual face generation methods in related technologies often relies on model architectures with large computational loads, such as Transformer, cross-attention model, diffusion model, etc. Although these methods can perform well in terms of generation quality, they are difficult to meet the real-time requirements in practical applications due to the high model complexity and slow inference speed.

[0031] For example, some related technologies introduce waveform-based automatic lip-reading (Wav2Lip) technology to achieve virtual face generation. This technology uses deep convolutional neural networks to process input video frames and speech features to generate realistic faces with lip movements. Its encoder and decoder use more than 20 layers of 3×3 convolutional networks to capture multi-level features. However, due to the large number of network layers, the model parameters are large, the computational complexity is high, the receptive field grows slowly, and global features cannot be captured efficiently. Although this design can extract rich feature information, it performs poorly in terms of inference speed and resource consumption, and it is difficult to meet the requirements of real-time applications.

[0032] For example, some related technologies introduce lip-reading experts to improve the accuracy of synchronization between lip movement features and speech features, and use Transformer to fuse audio and images. Although Transformer can effectively capture the fusion of long-distance dependencies and multimodal features, its computational complexity is relatively high. In addition, this method introduces the fusion features of audio and image in each layer of the encoder and decoder. Although it improves the quality of synchronization between lip movement features and speech features to a certain extent, it also significantly increases the complexity of the network, resulting in a long training and inference process, large computing resource requirements, and difficulty in meeting the requirements of real-time applications.

[0033] In this regard, in order to solve the above problems, an embodiment of the present application provides a virtual face generation method. The method, on the one hand, adopts an initial image generation model of a lightweight network architecture to obtain a face generation model, which not only has fewer network parameters and faster training convergence speed, but also can generate high-resolution clear lip face images even when real-time inference is performed on the GPU (Graphic Processing Unit) or NPU (Neural Processing Unit), thereby meeting the real-time requirements of practical applications. On the other hand, a simple multimodal feature embedding method is adopted in the face generation model to unify the audio features and the face image to be processed into the same feature space for fusion, thereby avoiding complex network design, reducing computational overhead, and significantly improving generation efficiency, thereby achieving the improvement of inference speed without losing the generation quality of the virtual face image, thereby meeting the requirements of real-time applications.

[0034] Figure 1 is a flow chart of the virtual face generation method provided by the present invention; Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0035] Step 110, inputting the face image to be processed, and the audio features and face key point features corresponding to the face image to be processed into a coding module in a face generation model, and obtaining image coding features corresponding to the face image to be processed and lip movement coding features corresponding to the audio features; Step 120, inputting the image coding feature and the lip movement coding feature into a feature fusion module in the face generation model to obtain a fusion feature; Step 130, inputting the fusion feature into a decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed; the virtual face image is a face image formed by fusing the lip movement image matching the audio feature on the basis of the face image to be processed; Among them, the face generation model is obtained by training the initial image generation model based on the sample real face image, the mask face image corresponding to the sample real face image, and the sample audio features and sample face key point features corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model with a parameter scale value less than a preset value.

[0036] It is understandable that in the related technology, the face generation model is formed based on a large-scale model framework. Since the model calculation process involves many parameters and a large amount of calculation, although it can perform well in terms of generation quality, the efficiency of virtual face generation is low due to the high model complexity and slow inference speed.

[0037] To address this problem, the embodiment of the present application establishes a face generation model with a small scale, i.e., a lightweight model structure, including an encoding module, a feature fusion module, and a decoding module, whose parameter scale value is less than a preset value. In the face generation model, since the model parameter scale value is less than the preset value, the model complexity is effectively reduced, the inference rate is improved, and the real-time performance of virtual face generation is improved; and in the face generation model, through the encoding module and the feature fusion module, the audio features and image features are unified into the same feature space for fusion, which effectively ensures the quality of image generation while avoiding complex network design, reducing computational overhead, and further significantly improving the efficiency of virtual face generation.

[0038] The setting value here is a value configured according to actual needs to limit the model scale.

[0039] Optionally, before executing step 110, a face generation model may be first obtained through training. Specifically, an initial image generation model may be constructed first, in which the model parameter scale is smaller than the set value, that is, a lightweight model structure. The initial image generation model may include an initial encoding module, an initial feature fusion module, and an initial decoding module. The initial encoding module may be a model prepared for image feature encoding and lip movement feature encoding after parameter initialization, or a pre-trained model with image feature encoding and lip movement feature encoding functions; the initial feature fusion module may be a model prepared for feature fusion after parameter initialization, or a pre-trained model with feature fusion functions; the initial decoding module may be a model prepared for face image reconstruction after parameter initialization, or a pre-trained model with face image reconstruction functions, and this embodiment does not specifically limit this.

[0040] In addition, the sample real face image, the mask face image corresponding to the sample real face image, and the sample audio features and sample face key point features corresponding to the mask face image can also be collected. The mask face image here is a face image with occlusion; it can be obtained by cropping the face area and specifying the occlusion position (that is, the lip area). The sample audio features here can be pre-processed by MFCC (Mel-Frequency Cepstral Coefficients) or other audio feature extraction methods; the sample face key points here can be obtained by key point network (landmark) detection.

[0041] Subsequently, the mask face image corresponding to the sample real face image, the sample audio features and the sample face key point features corresponding to the mask face image can be used as the input of the initial image generation model, and the initial encoding module in the initial image generation model performs feature encoding on the mask face image corresponding to the sample real face image, the sample audio features and the sample face key point features corresponding to the mask face image, and obtains the sample image encoding features of the mask face image and the sample lip movement encoding features corresponding to the sample audio features, and then the initial feature fusion module in the initial image generation model performs feature fusion to obtain the sample fusion features, and then the decoding module in the initial image generation model applies the sample fusion features to reconstruct the mask face image, and obtains the sample virtual face image of the lip movement image that is fused and matched with the sample audio features. After obtaining the sample virtual face image, the loss function can be calculated based on the feature differences, local key point differences, and multiple items of feature encoding differences between the sample virtual face image and the sample real face image, and the initial image generation model is iterated on the parameters as a whole based on the loss function, so as to obtain the face generation model.

[0042] The face generation model thus obtained can be used in practical applications to perform feature encoding on the face image to be processed, as well as the audio features and facial key point features corresponding to the face image to be processed, through the encoding module in the face generation model. In this process, the encoding module specifically performs feature encoding on the face image to be processed to obtain image encoding features corresponding to the face image to be processed; and performs feature encoding on the audio features and facial key point features corresponding to the face image to be processed to achieve mapping between audio features and lip movement features, thereby obtaining lip movement encoding features corresponding to the audio features.

[0043] The face image to be processed here may be a face image with lips occluded, or may be a face template image with fixed lip movements, which is not specifically limited in this embodiment.

[0044] The encoding module here can be constructed based on an integrated encoding model with both image encoding and lip movement encoding, or it can be constructed based on an image encoder with image encoding and a lip movement encoder with lip movement encoding, and this embodiment does not make any specific limitations on this.

[0045] The image coding features here are used to characterize the facial image feature representation of the facial image to be processed, including but not limited to edge information, texture information and semantic information; the lip movement coding features are used to characterize the lip movement coefficients.

[0046] After obtaining the image coding features corresponding to the facial image to be processed and the lip movement coding features corresponding to the audio features through the coding module, the image coding features and the lip movement coding features can be fused through the feature fusion module in the face generation model. During the feature fusion process, the feature fusion module can first perform feature dimension conversion on the image coding features and / or the lip movement coding features to unify the feature dimensions of the image coding features and the lip movement coding features to the same feature dimension, and then perform feature fusion on the lip movement coding features and the image coding features to obtain a fusion feature of the image coding features and the lip movement coding features.

[0047] After the fusion features are obtained through the feature fusion module, the fusion features can be applied to reconstruct the face image to be processed through the decoding module in the face generation model. During the reconstruction process, the decoding module can specifically perform multi-level reconstruction of the face image to be processed by applying the fusion features, so as to generate a virtual face image that can show accurate lip movements based on the multi-level reconstruction results, so that the reconstructed virtual face image is fused with a realistic lip movement image that matches the audio features on the basis of the face image to be processed, thereby ensuring the quality of the virtual face image generation. In addition, all the generated virtual face image frames can be spliced ​​together in sequence to form a coherent and realistic virtual human video. The multi-level reconstruction here includes but is not limited to the reconstruction of low-frequency contour information and high-frequency edge information, which is not specifically limited in this embodiment.

[0048] In the embodiment of the present invention, a lightweight network is used, which can run on the GPU side and mobile devices, effectively ensuring the fluency and real-time nature of real-time conversations with people. The virtual face image generated in this way not only makes the virtual human technology more natural visually, but also can be flexibly deployed on different devices to meet a wide range of real-time application needs.

[0049] The method provided in the embodiment of the present application, on the one hand, generates a face generation model based on the training of an initial image generation model whose parameter scale value is less than a preset value, so that the trained face generation model has fewer network parameters and a faster training convergence speed, thereby ensuring that when performing real-time reasoning of a virtual face image, a virtual face image with clear lips and high resolution can be generated while meeting the real-time requirements of practical applications; on the other hand, during the reasoning process, the face image to be processed and its corresponding audio features and face key point features are input into the encoding module and feature fusion module in the face generation model, so that the encoding module and the feature fusion module use a simple multimodal feature embedding method to unify the speech features and the image features into the same feature space, effectively reducing the use of complex network design, thereby reducing the computational overhead, and then, the fused features generated by the fusion are input into the decoding module, so that the decoding, based on the face image to be processed, fuses the lip motion image matching the audio features to form a virtual face image fused with the lip motion image matching the audio features and realistic, thereby further ensuring that the reasoning speed is improved without losing the generation quality of the virtual face image, and meeting the requirements of real-time applications.

[0050] In some embodiments, step 130 specifically includes: Inputting the fused feature into the first feature decoding unit of the decoding module to obtain the first image feature of the virtual face image; the first image feature is used to reconstruct the contour information of the face image to be processed; Inputting the first image feature and the fusion feature into the second feature decoding unit of the decoding module to obtain the second image feature of the virtual face image; the second image feature is used to reconstruct the edge information of the face image to be processed; The first image feature and the second image feature are input into the feature recombination unit of the decoding module to obtain the virtual face image.

[0051] It should be noted that in order to better extract multi-level features, especially to capture high-frequency information, and ensure that the generated image is more realistic in details such as edges, textures, and lips, the decoding module here adopts a dual-stream structure, specifically including a first feature decoding unit (also called a low-frequency feature decoding unit), a second feature decoding unit (also called a high-frequency feature decoding unit), and a feature reorganization unit. Among them, the first feature decoding unit is used to reconstruct the low-frequency contour information of the face image to be processed, the second feature decoding unit is used to reconstruct the high-frequency edge information of the face image to be processed, and the feature reorganization unit is used to perform feature reorganization to obtain a high-quality virtual face image.

[0052] Here, the model structure of the first feature decoding unit and the model structure of the second feature decoding unit may be the same or different.

[0053] Exemplarily, in some embodiments, the first feature decoding unit is constructed based on a residual block, an upsampling block and an inverted residual block; and the second feature decoding unit is constructed based on an edge-guided convolution block.

[0054] Figure 2 is a schematic diagram of the structure of the decoding module provided by the present invention; Figure 2 As shown, in some implementations, the first feature decoding unit can be constructed by multiple layers of residual blocks (or resnet), multiple layers of upsampling blocks (or upsample) and inverted residual blocks (or InvertedResidual); exemplarily, the first feature decoding unit includes two layers of residual blocks, three layers of upsampling blocks and one layer of inverted residual blocks, wherein the first layer of residual blocks in the first feature decoding unit are connected to the first layer of upsampling blocks, the first layer of upsampling blocks are connected to the inverted residual blocks, the inverted residual blocks are connected to the second layer of upsampling blocks, the second layer of upsampling blocks are connected to the second layer of residual blocks, and the second layer of residual blocks are connected to the third layer of upsampling blocks.

[0055] Figure 3 is a schematic diagram of the structure of the residual block provided by the present invention; Figure 3 As shown, in some embodiments, the residual block can be formed by residual connection based on a convolutional network layer (or conv) with a multi-layer 3×3 convolution kernel, a convolutional network layer with a 1×1 convolution kernel, a multi-layer batch normalization layer (or batch norm), and a multi-layer activation function layer. The activation function used in the activation function layer can be configured according to actual needs, such as a rectified linear unit activation function (Rectified LinearUnit, ReLU), etc. This embodiment does not specifically limit this.

[0056] Among them, the inverted residual block uses depthwise separable convolution, which can decompose the operations in the spatial and channel dimensions of traditional convolution into two steps: first, depthwise separable convolution is performed on each channel, and then point-by-point convolution is performed through 1x1 convolution to integrate information, which significantly reduces the amount of calculation while retaining the ability to capture the diverse features of the input image, enabling the decoder to generate richer and more recognizable feature representations. Figure 4 is a schematic diagram of the structure of the inverted residual block provided by the present invention; Figure 4As shown, in some embodiments, the inverted residual block can be constructed based on a convolution network layer of a multi-layer 1×1 convolution kernel, a multi-layer activation function layer, and a depthwise separable convolution (dwise) of a 1×1 convolution kernel. The activation function used in the activation function layer here can be configured according to actual needs, such as the ReLU6 function obtained from the ReLU variant, and the linear activation function (or linear), etc. This embodiment does not specifically limit this.

[0057] The second feature decoding unit may be constructed by an edge-oriented convolution block (ECB).

[0058] Figure 5 is a schematic diagram of the structure of the edge-guided convolution block provided by the present invention; Figure 5 As shown in the figure, the edge-guided convolution block can be constructed by a standard convolution network and multiple different convolution operator networks, such as an edge-guided convolution block can be constructed by a standard convolution network based on a 3×3 convolution kernel, a Sobel operator convolution network in the X direction of a 1×1 convolution kernel (or conv1×1SobelX), a Sobel operator convolution network in the Y direction of a 1×1 convolution kernel (or conv1×1SobelY), and a Laplacian operator convolution network of a 1×1 convolution kernel (or conv1×1Laplacian). In the model application process, the edge-guided convolution block first uses the conv3x3 standard convolution to capture the basic edges and local features of the face image, and then performs multi-directional convolution operations, using conv1x1SobelX to extract the horizontal edge of the face image, conv3x3SobelY to extract the vertical edge of the face image, and conv1x1Laplacian to extract the high-frequency detail information of the face image, so as to enhance the network's response to features in different directions.

[0059] It should be noted that in order to simplify the amount of computation in the model reasoning process and accelerate the efficiency of model reasoning, during the model reasoning process, the embodiment of the present application performs equivalent replacements for multiple different convolution operator networks in the edge-guided convolution block.

[0060] Exemplarily, in some embodiments, the edge-guided convolution block is constructed based on an equivalent convolutional network corresponding to a standard convolutional network and multiple different convolutional operator networks; the equivalent convolutional network is a convolutional network obtained by linearly fusing model parameters of the multiple different convolutional operator networks.

[0061] Optionally, during the model inference process, the model parameters of multiple different convolution operator networks can be fused to obtain an equivalent convolution network. For example, during the fusion process, the weights and biases of conv1x1SobelX, conv3x3SobelY and conv1x1Laplacian can be linearly combined to form an equivalent 3 3 standard convolution kernels, as follows Figure 5 As shown, the second feature decoding unit extracts the second image feature based on the standard convolutional network and the equivalent convolutional network, thereby simplifying the calculation and accelerating the reasoning speed, thereby achieving improved efficiency of virtual face generation while ensuring the quality of virtual face generation.

[0062] In summary, compared with the traditional convolutional network, which loses some high-frequency information during the feature extraction process and has the defect of blurred edges of the generated face image, the second feature decoding unit in the embodiment of the present application introduces ECB, so that ECB retains the high-frequency details in the face image through multiple edge detection operations, thereby improving the clarity of the generated image. Although ECB introduces multiple convolution operations, since multiple operators are fused into an equivalent convolution during its reasoning process, the computational complexity is still low, which can effectively improve the efficiency of virtual face generation while ensuring the quality of virtual face generation, and can be effectively suitable for resource-constrained application scenarios, such as real-time face generation tasks on mobile devices.

[0063] Optionally, the step of generating a virtual face image in step 130 specifically includes: After the fusion feature is obtained by the feature fusion module, the fusion feature can be used to extract low-frequency information through the first feature decoding unit of the decoding module to obtain the first image feature for reconstructing the contour information of the face image to be processed.

[0064] After obtaining the first image feature output by the first feature decoding unit, the second feature decoding unit of the decoding module can apply the fusion feature and the first image feature to extract high-frequency information to obtain the second image feature for reconstructing the edge detail information of the processed face image.

[0065] After obtaining the second image feature output by the second feature decoding unit, the first image feature and the second image feature can be recombined through the feature recombining unit (or pixel shuffle) of the decoding module to reconstruct a virtual face image that matches the audio feature and contains a vivid and realistic lip movement image based on the face image to be processed.

[0066] It should be noted that, during the training of the decoding module, the first feature decoding unit and the second feature decoding unit can be trained separately, and in the inference stage, the first feature decoding unit and the second feature decoding unit are fused into one network. For example, during the model training process, the first feature decoding unit and the second feature decoding unit are trained separately as two parallel branches, and in the model inference process, the first feature decoding unit and the second feature decoding unit are connected in series to form the above-mentioned encoding module, thereby achieving inference acceleration without increasing parameters, thereby improving the image generation quality while improving the image generation efficiency.

[0067] In summary, the method provided in this embodiment, by sequentially inputting the fused features into different feature decoding units of the decoding module, obtains the first image features and the second image features for reconstructing the contour and the edge, and finally recombines these features to generate a virtual face image. It not only improves the quality of image generation through multi-level feature extraction, but also effectively improves the efficiency of image generation.

[0068] In some embodiments, step 110 specifically includes: Inputting the face image to be processed into the image encoder of the encoding module to obtain image encoding features corresponding to the face image to be processed; The audio features and facial key point features corresponding to the facial image to be processed are input into the lip movement encoder of the encoding module to obtain the lip movement encoding features corresponding to the audio features.

[0069] Among them, the encoding module includes an image encoder and a lip movement encoder; wherein the image encoder is used to perform image encoding to extract corresponding image encoding features; the lip movement encoder is used to map audio features to encode and generate lip movement encoding features that match the audio features.

[0070] The structures of the image encoder and the lip movement encoder here can be the same or different.

[0071] Exemplarily, in some embodiments, the image encoder is constructed based on a residual block, a downsampling block and an inverse residual block; and the lip movement encoder is constructed based on a time series model.

[0072] For image encoders, compared with the prior art of image encoding through deep convolutional networks, some key details may be lost when processing high-frequency information, especially the reconstruction of facial details such as lips, and the generated facial image is overall blurred and the lip movements are not accurate enough. The embodiment of the present application introduces residual blocks in the image encoder for feature extraction, so that the image encoder can provide high-quality facial image feature representation. The specific residual block can extract multi-level features of the input image layer by layer, including low-level edge and texture information, as well as high-level abstract semantic information. These features are crucial for generating clear and realistic facial images. In addition, through the residual connection, the image encoder can retain the original information transmission, avoid the loss of important details in the feature extraction process, and thus enhance the accuracy of the generated image and the ability to restore details.

[0073] In addition, compared with the prior art, the image encoder in the embodiment of the present application avoids using complex attention mechanisms or a large number of ordinary convolutions that will greatly increase the computational overhead, resulting in higher model complexity and reduced reasoning speed. On the contrary, a residual block is introduced in the image encoder. The lightweight design of the residual block enables it to effectively alleviate the gradient vanishing problem in the deep network while reducing the computational burden, and ensure the stability of network training. Moreover, due to the modular structure of the residual block, it can be easily combined with other network components to form a flexible encoder architecture to adapt to different needs in the generation task.

[0074] In addition, in order to further reduce the amount of calculation and improve the efficiency of feature representation, this embodiment also adds a layer of inverted residual blocks at the end of the image encoder. The inverted residual block uses depth-separable convolution to decompose the operations of spatial and channel dimensions in traditional convolution into two steps: first, spatial convolution is performed on each channel separately, and then the information is integrated point by point through 1x1 convolution. In this way, the amount of calculation is significantly reduced, while retaining the ability to capture the diverse features of the input image, so that the encoder can generate richer and more recognizable feature representations.

[0075] In summary, the residual network plays a key role in the process of generating facial images. It can extract detailed features in facial images layer by layer, and retain important facial information, so that the generated image is more realistic in terms of edges, textures, and high-level semantic information. In addition, the use of inverted residual blocks can capture the fine structure of facial features while reducing computational complexity, thereby enhancing the clarity and authenticity of the generated image. Therefore, the residual network and inverted residual blocks are introduced into the image encoder provided by this embodiment, which can better extract multi-level features, especially the capture of high-frequency information, ensure that the generated image is more realistic in details such as edges, textures, and lips, and ensure that the output facial image not only has high-definition resolution, but also can show natural facial features, so that the generated result is more in line with the requirements of visual perception, with high detail fidelity, excellent generation quality, and can effectively improve the efficiency of model reasoning.

[0076] The structure of the image encoder is described in detail below.

[0077] Figure 6 Schematic diagram of the structure of the image encoder provided by the present invention. Figure 6 As shown, the image encoder can be formed by multiple layers of residual blocks, multiple layers of downsampling blocks and residual blocks. Exemplarily, the image encoder includes three layers of residual blocks (or resnet), four layers of downsampling blocks (or downsample) and one layer of inverted residual blocks (or inverted residual), wherein the first layer of residual blocks in the image encoder are connected to the first layer of downsampling blocks, the first layer of downsampling blocks are connected to the second layer of residual blocks, the second layer of residual blocks are connected to the second layer of downsampling blocks, the second layer of downsampling blocks are connected to the third layer of residual blocks, the third layer of residual blocks are connected to the third layer of downsampling blocks, the third layer of downsampling blocks are connected to the inverted residual blocks, and the inverted residual blocks are connected to the fourth layer of downsampling blocks.

[0078] The specific structure of the residual block can be found in Figure 3 The specific structure of the inverted residual block can be found in Figure 4 , which will not be described in detail here.

[0079] For the lip movement encoder, it applies audio feature fitting to obtain the corresponding lip movement coding features, also known as lip movement coefficients. Since audio features and lip movement features are both time series data, they are time-dependent. The Long Short-Term Memory (LSTM) network can well capture and learn the complex nonlinear relationships in these time series. At the same time, the gating structure of the LSTM network can selectively remember or forget information, effectively retain key information in the speech, and filter out noise, so that the model can better understand the dynamic changes between audio features and corresponding lip movement features, thereby making the lip movement images generated by the model more natural and realistic. Furthermore, the LSTM model generates high-quality lip movements within a shorter historical window, which can meet subsequent real-time applications. Therefore, the embodiment of the present application preferably constructs a lip movement encoder using LSTM to generate corresponding lip movement coding features by mapping audio features through LSTM, while ensuring the synchronization and naturalness of the lip output speech, and improving the real-time generation of virtual faces.

[0080] Optionally, the encoding step of the encoding module in step 110 includes: The face image to be processed is input into the image encoder of the encoding module, and the image encoder uses the face image to be processed to perform multi-level extraction of edge information, texture information, high-level semantic features, etc., so as to obtain image coding features that can finely characterize the facial features of the face image to be processed.

[0081] In addition, the audio features and facial key point features corresponding to the facial image to be processed are input into the lip movement encoder of the encoding module, so that the lip movement encoder can use the audio features and facial key point features corresponding to the facial image to be processed to capture lip movement coding features that can be synchronized with the audio and have a higher degree of naturalness.

[0082] In summary, the method provided in this embodiment, through the image encoder and the lip movement encoder, collaboratively performs image coding feature extraction and lip movement coding feature extraction, and unifies the audio features and image features into the same feature space through a multimodal feature embedding method. It can not only ensure the quality of image generation, but also avoid complex network design and reduce computational overhead through efficient feature extraction and encoding processes, and significantly improve the efficiency of image generation, thereby achieving a double improvement in image generation quality and efficiency, and achieving a better quality image generation effect.

[0083] In some embodiments, step 120 specifically includes: Inputting the lip movement coding feature into the feature conversion layer of the feature fusion module to obtain a converted lip movement coding feature; the feature dimension of the converted lip movement coding feature is the same as the feature dimension of the image coding feature; The converted lip movement coding features and the image coding features are input into the feature fusion layer of the feature fusion module to obtain the fusion features.

[0084] Optionally, in the process of generating a virtual face image, the feature fusion stage plays a key role. In the feature fusion stage, the feature dimension of the lip movement coding feature is first transformed by the feature conversion layer of the feature fusion module to transform its feature dimension into the same feature dimension as the image coding feature. The feature conversion layer here can be constructed based on a network layer that can convert feature dimensions, such as a fully connected layer, etc., which is not specifically limited in this embodiment.

[0085] Subsequently, the converted lip motion coding features and image coding features are added and fused through the feature fusion layer of the feature fusion module to obtain a fused feature representation, namely the fused feature. This fusion method combines image and speech information, so that the generated image can not only restore the overall structure of the face, but also capture the lip details related to the audio, ensuring that the generated result is more natural and consistent.

[0086] Compared with the prior art in which audio latent features are injected into each layer of the image encoder and image decoder respectively, and complex model architectures such as cross-attention mechanism and convolution-enhanced Transformer model are used to achieve multimodal fusion, this technology has the defects of increased model complexity and difficult network convergence. The method provided in this application uses a multimodal feature embedding method to unify audio features and image features into the same feature space for fusion, which significantly simplifies the feature fusion process and reduces the complexity and parameter amount of the network, which not only reduces the computational overhead, but also avoids the impact of repeated operations on the stability of network training.

[0087] In addition, the results of ablation experiments in the embodiments of the present application further verify the advantages of the method provided by the present application, that is, adding speech features to each layer of the image encoder and decoder does not significantly improve the lip detail performance of the generated image. The method provided by the present application adopts a direct addition strategy after fully connected layer processing in the feature fusion stage, which can not only efficiently fuse speech and image information, but also ensure that the generated face image has high-quality facial details, especially in the lip area to achieve natural transition and realistic performance, thereby improving the image generation quality while reducing computational overhead and improving the real-time performance of image generation.

[0088] In some embodiments, the face generation model is trained based on the following steps: Inputting the masked face image, the sample audio features and the sample face key point features into the initial image generation model to obtain sample lip movement coding features corresponding to the sample audio features and a sample virtual face image corresponding to the masked face image; Performing three-dimensional reconstruction on the sample real face image to obtain a lip movement feature label corresponding to the sample real face image; Obtaining a target loss function according to the sample lip movement coding features, the sample virtual face image, the lip movement feature label, and the sample real face image; According to the target loss function, the initial image generation model is iteratively trained to obtain the face generation model.

[0089] Figure 7 It is a schematic diagram of the training process of the face generation model provided by the present invention.

[0090] like Figure 7 As shown in Figure 1, the training of the face generation model specifically includes: First, the input information of the initial image generation model is obtained, and the lip area of ​​the sample real face image is masked to obtain a masked face image. The voice information input to the masked face image is extracted to obtain the sample audio features, and the key points of the masked face image are extracted to obtain the key point features of the sample face.

[0091] Subsequently, the masked face image, the sample audio features and the sample face key point features are input into the initial image generation model, so that the initial encoding module in the initial image generation model encodes the masked face image to obtain the sample image encoding features, and encodes the sample audio features and the sample face key point features to obtain the sample lip movement encoding features; the initial feature fusion module in the initial image generation model fuses the sample image encoding features and the sample lip movement encoding features to obtain the sample fusion features; the initial first feature decoding unit in the initial image generation model applies the fusion features to extract the contour information of the reconstructed masked face image to obtain the first sample image features; the initial second feature decoding unit in the initial image generation model applies the fusion features to extract the edge information of the reconstructed masked face image to obtain the second sample image features; the initial feature recombination unit in the initial image generation model recombines the first sample image features and the second sample image features to obtain the sample virtual face image.

[0092] In addition, the sample real face image is three-dimensionally reconstructed. In the three-dimensional reconstruction, a reconstruction model such as a 3D Morphable Face Model (3DMM) is used to reconstruct the sample real face image in three dimensions to obtain the lip movement feature label corresponding to the sample real face image, that is, the lip coefficient containing the expression.

[0093] Subsequently, at least one loss function is obtained by combining the sample lip movement encoding features, the sample virtual face image, the lip movement feature label, and the sample real face image, and at least one loss function is combined to construct a target loss function, thereby improving the quality and detail expression of the generated face image.

[0094] The at least one loss function here includes but is not limited to a loss function for evaluating the difference between a real image and a generated image, a loss function for evaluating the difference in feature distribution between a real image and a generated image, and a loss function for evaluating the difference in feature distribution between a local key area of ​​a real image and a local key area of ​​a generated image. This embodiment does not specifically limit this.

[0095] Exemplarily, in some embodiments, the specific steps of obtaining the target loss function include: Based on the discriminator, the sample virtual face image and the sample real face image are applied to perform classification performance perception to obtain a first loss function; Based on the generator, applying the sample virtual face image to generate performance perception to obtain a second loss function; Obtaining a third loss function according to a distribution error between the sample lip movement encoding feature and the lip movement feature label; Obtaining a fourth loss function according to a feature distribution error between the sample virtual face image and the sample real face image; Obtaining a fifth loss function according to a feature distribution error between the mouth area of ​​the sample virtual face image and the mouth area of ​​the sample real face image; Obtaining a sixth loss function according to a reconstruction error between the sample virtual face image and the sample real face image; The target loss function is obtained by combining the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function and the sixth loss function.

[0096] Among them, the first loss function It is the loss function of the discriminator, which is used to measure the ability of the discriminator to correctly classify the sample real face image and the generated sample virtual face image. Specifically, it is based on the discriminator, using the sample virtual face image and the sample real face image to perform classification performance perception; the first loss function The specific calculation formula is: ; in, is the first loss function; is the discriminator for the sample real face image The output probability of is a sample real face image Expectation of distribution; represents the generated sample virtual face image (that is, the sample virtual face image output by the initial image generation model), is a masked face image with occlusion. is the mask face image Expectation of the distribution.

[0097] The second loss function It is the loss function of the generator, which is used to measure the ability of the generated sample virtual face image to deceive the discriminator, so that the generated virtual face image is closer to the real face image. It is based on the generator and the sample virtual face image is used to generate performance perception. The second loss function The specific calculation formula is as follows: ; in, is the second loss function.

[0098] The third loss function It is the audio feature fitting lip coefficient loss function, which is determined based on the distribution error between the sample lip movement encoding features and the lip movement feature labels. The third loss function The specific calculation formula is as follows: ; in, For the Sample lip movement encoding features corresponding to sample audio features; For the The lip movement feature labels corresponding to the sample audio features; is the number of samples.

[0099] The fourth loss function is a feature-aware loss function, which can be determined based on the feature distribution error between the sample virtual face image and the sample real face image. The feature distribution here can be the feature extraction output of the sample virtual face image and the sample real face image using a pre-trained feature extraction model, such as the last five convolutional layers of the VGG19 network, that is, the feature representation of the real sample virtual face image and the sample real face image is extracted from conv4_1 to conv5_4 of the VGG19 network to calculate the loss. The fourth loss function The specific calculation formula is as follows: ; in, is the fourth loss function, For feature extraction models (such as VGG19 network) The feature extraction function of the layer, For the feature extraction model The feature representation of the sample virtual face image extracted by the layer, For the feature extraction model Feature representation of sample virtual face images extracted by layers.

[0100] Fifth loss function It is the mouth loss function, which is obtained based on the feature distribution error between the mouth area of ​​the sample virtual face image and the mouth area of ​​the sample real face image; here, the corresponding loss value can also be calculated by extracting the feature representation of the mouth area of ​​the sample virtual face image and the mouth area of ​​the sample real face image through a pre-trained feature extraction model, such as the VGG19 network. The fifth loss function The specific calculation formula is as follows: ; in, is the fifth loss function; is a sample real face image The mouth area; is a sample virtual face image The mouth area; For the feature extraction model Feature representation of the mouth area of ​​the sample virtual face image extracted by layer; For the feature extraction model Feature representation of the mouth area of ​​the sample real face image extracted by layer.

[0101] Sixth loss function It is a reconstruction loss function, which is used to measure the difference between the sample real face image and the sample virtual face image obtained by reconstructing the noisy face image (that is, the masked face image) through the network. The sixth loss function The calculation formula is as follows: ; in, is the sixth loss function; For the Sample real face images; For the generated The sample virtual face images corresponding to the sample real face images.

[0102] After obtaining the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function and the sixth loss function based on the above formula, the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function and the sixth loss function can be fused to obtain the target loss function.

[0103] The fusion here can be achieved by weighted addition or through a fusion model.

[0104] Exemplarily, the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function and the sixth loss function are weightedly fused to obtain the calculation formula of the target loss function as follows: ; in, , , , , and is a hyperparameter, that is, the weight coefficient of each loss function. The weight coefficient of each loss can be adjusted according to actual needs to ensure the balanced optimization of the network in maintaining facial structure consistency, improving image detail fidelity and lip synchronization, etc. For example, you can set , , , , , .

[0105] After obtaining the target loss function, the initial image generation model can be iteratively trained based on the target loss function to obtain a virtual face image that can accurately reconstruct a realistic lip movement image that matches the audio features.

[0106] It should be noted that during the model training process, hyperparameters can be set according to actual needs to improve the convergence efficiency of model training. For example, the optimizer can be set to the Adaptive Moment Estimation (Adam) optimizer, and the learning rate can be set to , the exponential decay rate of the first-order moment estimate is set to , the exponential decay rate of the second-order moment estimate is set to Etc., and this embodiment does not make any specific limitation to this.

[0107] In summary, the method provided in this embodiment, by inputting the masked face image, sample audio features and sample face key point features into the initial image generation model, combining the lip movement feature labels obtained by three-dimensional reconstruction, and utilizing multiple loss functions, including classification performance, generation performance, distribution error and reconstruction error, etc., to jointly train and iteratively optimize the face generation model, can significantly improve the lip movement realism and overall image quality of the virtual face image.

[0108] The virtual face generation system provided by the present invention is described below. The virtual face generation system described below and the virtual face generation method described above can be referenced to each other.

[0109] Figure 8 Schematic diagram of the structure of the virtual face generation system provided by the present invention; Figure 8 As shown, the system includes: a first processing unit 810, a second processing unit 820 and a generating unit 830.

[0110] The first processing unit 810 is used to input the face image to be processed, and the audio features and face key point features corresponding to the face image to be processed into the encoding module in the face generation model, and obtain the image encoding features corresponding to the face image to be processed and the lip movement encoding features corresponding to the audio features; The second processing unit 820 is used to input the image coding feature and the lip movement coding feature into the feature fusion module in the face generation model to obtain a fusion feature; The generating unit 830 is used to input the fusion feature into the decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed; the virtual face image is a face image formed by fusing the lip movement image matching the audio feature on the basis of the face image to be processed; Among them, the face generation model is obtained by training the initial image generation model based on the sample real face image, the mask face image corresponding to the sample real face image, and the sample audio features and sample face key point features corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model with a parameter scale value less than a preset value.

[0111] The system provided in this embodiment generates a face generation model by, on the one hand, training a face generation model based on an initial image generation model whose parameter scale value is less than a preset value, so that the trained face generation model has fewer network parameters and a faster training convergence speed, thereby ensuring that when performing real-time reasoning on a virtual face image, a virtual face image with high resolution and clear lips can be generated while meeting the real-time requirements of practical applications; on the other hand, during the reasoning process, the face image to be processed and its corresponding audio features and face key point features are input into the encoding module and feature fusion module in the face generation model, so that the encoding module and the feature fusion module use a simple multimodal feature embedding method to unify the speech features and the image features into the same feature space, effectively reducing the use of complex network design, thereby reducing the computational overhead, and then, the fused features generated by the fusion are input into the decoding module, so that the decoding module fuses the lip motion image matching the audio features on the basis of the face image to be processed, so as to form a virtual face image fused with the lip motion image matching the audio features and realistic, thereby further ensuring that the reasoning speed is improved without losing the generation quality of the virtual face image, and meeting the requirements of real-time applications.

[0112] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0113] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the virtual face generation method, which includes: inputting the face image to be processed, and the audio features and face key point features corresponding to the face image to be processed into the encoding module in the face generation model to obtain the image encoding features corresponding to the face image to be processed and the lip movement encoding features corresponding to the audio features; inputting the image encoding features and the lip movement encoding features into the feature fusion module in the face generation model to obtain the fusion features; inputting the fusion features into the decoding module in the face generation model to obtain the face to be processed. A virtual face image corresponding to the image; the virtual face image is a face image formed by fusing the lip movement image matching the audio feature on the basis of the face image to be processed; wherein the face generation model is obtained by training the initial image generation model based on the sample real face image, the mask face image corresponding to the sample real face image, and the sample audio features and sample face key point features corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model with a parameter scale value less than a preset value.

[0114] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0115] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the virtual face generation method provided by the above methods, which includes: inputting a face image to be processed, and audio features and face key point features corresponding to the face image to be processed into a coding module in a face generation model to obtain image coding features corresponding to the face image to be processed and lip movement coding features corresponding to the audio features; inputting the image coding features and the lip movement coding features into a feature fusion module in the face generation model to obtain fusion features; The fused features are input into the decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed; the virtual face image is a face image formed by fusing the lip movement image matching the audio features on the basis of the face image to be processed; wherein, the face generation model is obtained by training an initial image generation model based on a sample real face image, a mask face image corresponding to the sample real face image, and sample audio features and sample face key point features corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model whose parameter scale value is less than a preset value.

[0116] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the virtual face generation method provided by the above-mentioned methods, the method comprising: inputting a face image to be processed, and audio features and face key point features corresponding to the face image to be processed into a coding module in a face generation model to obtain image coding features corresponding to the face image to be processed and lip movement coding features corresponding to the audio features; inputting the image coding features and the lip movement coding features into a feature fusion module in the face generation model to obtain fused features; inputting the fused features into the face generation model. A decoding module in the model is used to obtain a virtual face image corresponding to the face image to be processed; the virtual face image is a face image formed by fusing the lip movement image matching the audio feature on the basis of the face image to be processed; wherein the face generation model is obtained by training an initial image generation model based on a sample real face image, a mask face image corresponding to the sample real face image, and a sample audio feature and a sample face key point feature corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model whose parameter scale value is less than a preset value.

[0117] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a virtual face, characterized in that: include: Inputting the face image to be processed, and the audio features and face key point features corresponding to the face image to be processed into the encoding module in the face generation model, and obtaining the image encoding features corresponding to the face image to be processed and the lip movement encoding features corresponding to the audio features; Inputting the image coding features and the lip movement coding features into a feature fusion module in the face generation model to obtain fusion features; Inputting the fusion feature into the decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed; the virtual face image is a face image formed by fusing the lip movement image matching the audio feature on the basis of the face image to be processed; Among them, the face generation model is obtained by training the initial image generation model based on the sample real face image, the mask face image corresponding to the sample real face image, and the sample audio features and sample face key point features corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model with a parameter scale value less than a preset value.

2. The method for generating a virtual face according to claim 1, characterized in that: The step of inputting the fusion feature into a decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed includes: Inputting the fused feature into the first feature decoding unit of the decoding module to obtain the first image feature of the virtual face image; the first image feature is used to reconstruct the contour information of the face image to be processed; Inputting the first image feature and the fusion feature into the second feature decoding unit of the decoding module to obtain the second image feature of the virtual face image; the second image feature is used to reconstruct the edge information of the face image to be processed; The first image feature and the second image feature are input into the feature recombination unit of the decoding module to obtain the virtual face image.

3. The method for generating a virtual face according to claim 2, characterized in that: The first feature decoding unit is constructed based on a residual block, an upsampling block and an inverted residual block; The second feature decoding unit is constructed based on an edge-guided convolution block.

4. The method for generating a virtual face according to claim 3, characterized in that: The edge-guided convolution block is constructed based on a standard convolutional network and an equivalent convolutional network corresponding to a plurality of different convolutional operator networks; The equivalent convolutional network is a convolutional network obtained by linearly fusing the model parameters of the multiple different convolutional operator networks.

5. The method for generating a virtual face according to any one of claims 1 to 4, characterized in that: The step of inputting the face image to be processed, and the audio features and face key point features corresponding to the face image to be processed into a coding module in a face generation model to obtain image coding features corresponding to the face image to be processed and lip movement coding features corresponding to the audio features includes: Inputting the face image to be processed into the image encoder of the encoding module to obtain image encoding features corresponding to the face image to be processed; The audio features and facial key point features corresponding to the facial image to be processed are input into the lip movement encoder of the encoding module to obtain the lip movement encoding features corresponding to the audio features.

6. The method for generating a virtual face according to claim 5, characterized in that: The image encoder is constructed based on a residual block, a downsampling block and an inverted residual block; The lip movement encoder is constructed based on a time series model.

7. The method for generating a virtual face according to any one of claims 1 to 4, characterized in that: The step of inputting the image coding feature and the lip movement coding feature into a feature fusion module in the face generation model to obtain a fusion feature comprises: Inputting the lip movement coding feature into the feature conversion layer of the feature fusion module to obtain a converted lip movement coding feature; the feature dimension of the converted lip movement coding feature is the same as the feature dimension of the image coding feature; The converted lip movement coding features and the image coding features are input into the feature fusion layer of the feature fusion module to obtain the fusion features.

8. The method for generating a virtual face according to any one of claims 1 to 4, characterized in that: The face generation model is trained based on the following steps: Inputting the masked face image, the sample audio features and the sample face key point features into the initial image generation model to obtain sample lip movement coding features corresponding to the sample audio features and a sample virtual face image corresponding to the masked face image; Performing three-dimensional reconstruction on the sample real face image to obtain a lip movement feature label corresponding to the sample real face image; Obtaining a target loss function according to the sample lip movement coding features, the sample virtual face image, the lip movement feature label, and the sample real face image; According to the target loss function, the initial image generation model is iteratively trained to obtain the face generation model.

9. The method for generating a virtual face according to claim 8, characterized in that: The step of obtaining a target loss function according to the sample lip movement coding features, the sample virtual face image, the lip movement feature label, and the sample real face image includes: Based on the discriminator, the sample virtual face image and the sample real face image are applied to perform classification performance perception to obtain a first loss function; Based on the generator, applying the sample virtual face image to generate performance perception to obtain a second loss function; Obtaining a third loss function according to a distribution error between the sample lip movement encoding feature and the lip movement feature label; Obtaining a fourth loss function according to a feature distribution error between the sample virtual face image and the sample real face image; Obtaining a fifth loss function according to a feature distribution error between the mouth area of ​​the sample virtual face image and the mouth area of ​​the sample real face image; Obtaining a sixth loss function according to a reconstruction error between the sample virtual face image and the sample real face image; The target loss function is obtained by combining the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function and the sixth loss function.

10. A virtual face generation system, characterized in that: include: A first processing unit is used to input a facial image to be processed, and audio features and facial key point features corresponding to the facial image to be processed into a coding module in a face generation model, to obtain image coding features corresponding to the facial image to be processed and lip movement coding features corresponding to the audio features; A second processing unit, configured to input the image coding feature and the lip movement coding feature into a feature fusion module in the face generation model to obtain a fusion feature; A generating unit, configured to input the fusion feature into a decoding module in the face generation model to obtain a virtual face image corresponding to the face image to be processed; the virtual face image is a face image formed by fusing a lip movement image matching the audio feature on the basis of the face image to be processed; Among them, the face generation model is obtained by training the initial image generation model based on the sample real face image, the mask face image corresponding to the sample real face image, and the sample audio features and sample face key point features corresponding to the mask face image; the mask face image is obtained by masking the lip area of ​​the sample real face image; the initial image generation model is a model with a parameter scale value less than a preset value.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the virtual face generation method as described in any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the virtual face generation method according to any one of claims 1 to 9 is implemented.

Images