Face image generation method and device, computer readable storage medium, terminal

Abstract

A method and device for generating a facial image, a computer-readable storage medium, and a terminal, the method comprising: determining a facial image generation model, the facial image generation model comprising an audio content feature extraction sub-model, an audio emotion feature extraction sub-model, and a diffusion sub-model; inputting the driving audio into the audio content feature extraction sub-model and the audio emotion feature extraction sub-model for feature extraction, thereby obtaining audio content features and audio emotion features; performing splicing based at least on the audio content features and the audio emotion features to obtain audio fusion features; inputting the audio fusion features and the features of a reference facial image with noise into the diffusion sub-model for denoising to obtain target complete facial features; decoding the target complete facial features to obtain a complete facial generated image. The above scheme helps to generate a facial generated image that can accurately match the lip shape in the driving audio and accurately express the emotions contained in the driving audio.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human generation, and in particular to a method and device for generating a face image, a computer-readable storage medium, and a terminal. Background Art

[0002] With the development of artificial intelligence technology, artificial intelligence generated content (AIGC) has become the hottest research topic. AIGC technology is also widely used in the field of digital speaker generation. The essence of the digital speaker generation task is to input audio and character images, and then generate facial images corresponding to the audio content in turn. How to use AIGC technology to achieve audio-driven generation of facial images that contain accurate lip shapes and accurately express the emotions contained in the audio is of great research value.

[0003] At present, the mainstream technology for audio-driven face image generation tasks mainly uses deep models to process audio features and image features. Specifically, the audio coding features and face image coding features are obtained by encoding the driving audio and face image respectively; then the audio coding features and face image coding features are directly input into the pre-trained face image generation model to obtain the face generation image.

[0004] However, in the above schemes, the audio features used to drive the generation of facial images usually only contain feature information of a single dimension (for example, feature information at the content or semantic level or dimension, but not feature information at the emotion or mood dimension), or confuse the content and emotion expressed by the audio. This prevents the network from clearly mining the correspondence between the two dimensions of the lip shape and emotion expressed by the audio and the generated facial image, resulting in the generated facial image or lip shape not matching well, or the emotional expression (such as facial expression) not being accurate enough. Summary of the invention

[0005] The technical problem solved by the embodiments of the present invention is how to generate a face generated image that can accurately match the lip shape in the driving audio and accurately express the emotions contained in the driving audio.

[0006] To solve the above technical problems, an embodiment of the present invention provides a facial image generation method, comprising the following steps: determining a facial image generation model, the facial image generation model comprising an audio content feature extraction sub-model, an audio emotion feature extraction sub-model, and a diffusion sub-model; inputting a driving audio into the audio content feature extraction sub-model for feature extraction to obtain an audio content feature, and inputting the driving audio into the audio emotion feature extraction sub-model for feature extraction to obtain an audio emotion feature; performing splicing based at least on the audio content feature and the audio emotion feature to obtain an audio fusion feature; inputting the audio fusion feature and the noisy reference facial image feature into the diffusion sub-model for denoising to obtain a target complete facial feature, wherein the noisy reference facial image feature is obtained by splicing the image feature of the reference facial image with a noise matrix; decoding the target complete facial feature to obtain a generated complete facial image.

[0007] Optionally, the facial image generation model also includes a key point feature extraction sub-model; the splicing based at least on the audio content features and the audio emotion features includes: splicing the audio content features, the audio emotion features and the facial key point features; wherein the facial key point features are obtained by extracting key points from the facial image with the lower half occluded, and inputting the extracted key points into the key point feature extraction sub-model for feature extraction.

[0008] Optionally, before performing splicing based at least on the audio content features and the audio emotion features, the method further includes: determining a first number of audios that are temporally before the driving audio, and a second number of audios that are temporally after the driving audio, which are respectively recorded as first audio and second audio; inputting each first audio and second audio into the audio content feature extraction submodel for respectively performing feature extraction to obtain corresponding multiple first audio content features and multiple second audio content features; performing weighted operations on the multiple first audio content features, the multiple second audio content features and the audio content features to obtain a fused audio content feature; and using the fused audio content feature to update the audio content feature.

[0009] Optionally, the reference facial image includes a complete facial image and a facial image with the lower half obscured from the same speaker and the same emotion; before inputting the audio fusion features and the noisy reference facial image features into the diffusion sub-model for denoising, the method also includes: performing feature extraction on the facial image with the lower half obscured and the complete facial image respectively to obtain partial facial image features and complete facial image features; and splicing the partial facial image features, the complete facial image features and the noise matrix to obtain the noisy reference facial image features.

[0010] Optionally, the diffusion sub-model adopts a U-Net model including a multi-layer network; the audio fusion features and the noisy reference facial image features are input into the diffusion sub-model for denoising to obtain the target complete facial features, including: inputting the noisy reference facial image features into the first layer network of the diffusion sub-model, and inputting the audio fusion features into each layer network of the diffusion sub-model, and using the output result of the last layer network of the diffusion sub-model as the target complete facial features; wherein, starting from the second layer network of the diffusion sub-model, the input data of each layer network is the output data of the previous layer network and the audio fusion features.

[0011] Optionally, the audio emotion feature extraction submodel includes: a pre-trained emotion classification network and an emotion feature extraction network; inputting the driving audio into the audio content feature extraction submodel for feature extraction to obtain audio content features, including: inputting the driving audio into the pre-trained emotion classification network to obtain a predicted emotion type label; encoding the predicted emotion type label to obtain an audio emotion encoding vector; inputting the audio emotion encoding vector into the emotion feature extraction network for feature extraction to obtain the audio emotion features.

[0012] Optionally, the predicted emotion type label is encoded to obtain an audio emotion coding vector, including: based on a preset emotion coding length, the predicted emotion type label is pre-encoded to obtain multiple groups of emotion sub-codes, each group of emotion sub-codes contains two identical emotion sub-codes; for each group of emotion sub-codes, the sine value of one of the emotion sub-codes is determined, and the cosine value of another emotion sub-code is determined, so as to determine the emotion coding corresponding to each emotion sub-code, and based on the obtained multiple emotion codes, the audio emotion coding vector is determined.

[0013] Optionally, the pre-trained emotion classification network is obtained by inputting a first training data set constructed by a plurality of first sample audios and their emotion type labels into an initialized emotion classification network for training based on a first loss function; the first loss function is represented by the following expression:

[0014]

[0015] Among them, L ec represents the function value of the first loss function, i represents the sequence number of the first sample audio, N represents the total number of the first sample audio, y i represents the true emotion type label of the i-th first sample audio, represents the emotion type label predicted for the i-th first sample audio, and ln() represents the logarithm with the constant e as the base.

[0016] Optionally, determining the facial image generation model includes: constructing a facial image generation model to be trained, the facial image generation model to be trained including: an audio content feature extraction sub-model to be trained, an audio emotion feature extraction sub-model to be trained, and a diffusion sub-model to be trained; using multiple reference sample facial images, multiple second sample audios, and a sample noise matrix to construct a second training data set, wherein the reference sample facial images and the second sample audios are aligned one-to-one in time sequence; based on a second loss function, inputting the second training data set into the facial image generation model to be trained for iterative training to obtain the facial image generation model.

[0017] Optionally, the second loss function is expressed by the following expression:

[0018]

[0019] Wherein, L′ represents the function value of the second loss function, t represents the hierarchical number of the network in the diffusion sub-model to be trained, M() represents the prediction noise matrix output by the t-th layer network of the diffusion sub-model to be trained, ∈ represents the sample noise matrix, indicates that ∈ follows a normal distribution, z t represents one of the input data of the t-th layer network of the diffusion sub-model to be trained, that is, the output data of the t-1-th layer network of the diffusion sub-model to be trained, C is another one of the input data of the t-th layer network of the diffusion sub-model to be trained, and C is a sample audio fusion feature obtained by splicing at least based on the sample audio content feature and the sample audio emotion feature; ‖∈-M(z t ,t,C)‖ represents the Euclidean distance between the sample noise matrix and the predicted noise matrix output by the t-th layer network of the diffusion sub-model to be trained, Represents the operation of finding the expected value of the squared Euclidean distance of multiple samples.

[0020] An embodiment of the present invention also provides a facial image generation device, including: a model determination module, used to determine a facial image generation model, the facial image generation model includes an audio content feature extraction sub-model, an audio emotion feature extraction sub-model, and a diffusion sub-model; an audio feature extraction module, used to input the driving audio into the audio content feature extraction sub-model for feature extraction to obtain audio content features, and input the driving audio into the audio emotion feature extraction sub-model for feature extraction to obtain audio emotion features; an audio feature splicing module, used to perform splicing based on at least the audio content features and the audio emotion features to obtain audio fusion features; a complete facial feature determination module, used to input the audio fusion features and the noisy reference facial image features into the diffusion sub-model for denoising processing to obtain target complete facial features, wherein the noisy reference facial image features are obtained by splicing the image features of the reference facial image with the noise matrix; a facial image generation module, used to decode the target complete facial features to obtain a complete facial generated image.

[0021] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned facial image generation method are executed.

[0022] An embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned facial image generation method when running the computer program.

[0023] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0024] In an embodiment of the present invention, feature information of two dimensions, "content" and "emotion", is separated from the driving audio. Specifically, the driving audio is input into an audio content feature extraction submodel for feature extraction to obtain audio content features, and the driving audio is input into an audio emotion feature extraction submodel for feature extraction to obtain audio emotion features; then, splicing is performed based on at least the audio content features and the audio emotion features to obtain audio fusion features; the audio fusion features and the features of a noisy reference facial image are input into the diffusion submodel for denoising to obtain target complete facial features; finally, the target complete facial features are decoded to obtain a complete face generated image.

[0025] Among them, the audio content features include the content or semantics expressed by the speaker, which can correspond to the lip shape or mouth movement of the speaker during the speaking process; the audio emotional features include the emotions of the speaker during the expression process, which can correspond to the real emotions of the speaker during the speaking process. Therefore, compared with the prior art that only extracts features of the driving audio in a single dimension, this implementation scheme helps to obtain a face generation image that can accurately match the lip shape contained in the driving audio, accurately express the speaker's real emotions and contain more detailed textures by combining the audio content features (or "content") dimension and the audio emotional features (or "emotion") dimension. Furthermore, this implementation scheme, on the basis of combining the audio content features and the audio emotional features, also introduces the image features of the reference face image, which helps to generate a face image that conforms to the basic contour features of the face and has more stable quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of a method for generating a face image in an embodiment of the present invention;

[0027] Figure 2 is a partial flow chart of another method for generating a face image in an embodiment of the present invention;

[0028] Figure 3 is a schematic diagram of the architecture of a face image generation model in an embodiment of the present invention;

[0029] Figure 4 yes Figure 3 Schematic diagram of the architecture of the diffusion sub-model in the face image generation model shown;

[0030] Figure 5 It is a structural schematic diagram of a facial image generating device in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0032] Reference Figure 1 , Figure 1 This is a flowchart of a method for generating a face image in an embodiment of the present invention. The method for generating a face image can be applied to a terminal with a face image generation function, and the terminal can include but is not limited to: a mobile phone, a computer, a tablet computer, a smart wearable device (for example, a smart watch), a vehicle-mounted terminal device, and can also be a server, a cloud platform, etc.

[0033] The method may include steps S11 to S14:

[0034] Step S11: determining a face image generation model, wherein the face image generation model includes an audio content feature extraction sub-model, an audio emotion feature extraction sub-model, and a diffusion sub-model;

[0035] Step S12: inputting the driving audio into the audio content feature extraction submodel to perform feature extraction to obtain audio content features, and inputting the driving audio into the audio emotion feature extraction submodel to perform feature extraction to obtain audio emotion features;

[0036] Step S13: performing splicing based at least on the audio content feature and the audio emotion feature to obtain an audio fusion feature;

[0037] Step S14: inputting the audio fusion features and the reference facial image features with noise into the diffusion sub-model for denoising to obtain the target complete facial features, wherein the reference facial image features with noise are obtained by concatenating the image features of the reference facial image with the noise matrix;

[0038] Step S15: Decode the target complete face features to obtain a complete face generated image.

[0039] Furthermore, the method also includes: generating images of multiple complete human faces corresponding to multiple segments of driving audio with a time sequence, splicing the images in a time sequence order, and obtaining a digital human video.

[0040] In the specific implementation of step S11, the audio content feature extraction sub-model can adopt an existing neural network that can extract audio content features, wherein the audio content features can specifically include text feature information and semantic feature information corresponding to the driving audio, which is mainly used to affect the lip shape and mouth movement of the face generated image; the audio emotion feature extraction sub-model can adopt an existing neural network that can extract audio emotion features, wherein the audio emotion features can specifically include rhythm (for example, rhythm, pitch, sound intensity, etc.) feature information in the audio, which is mainly used to affect the expression, texture details, etc. of the face generated image.

[0041] In the specific implementation of step S12, the driving audio is input into the audio content feature extraction submodel for feature extraction to obtain audio content features, and the driving audio is input into the audio emotion feature extraction submodel for feature extraction to obtain audio emotion features.

[0042] The driving audio may be derived from the audio recorded in real time during the speaker's speech; or may be derived from the audio in a historically collected audio database; or may be extracted from a live real-time or pre-recorded video. The speaker may be selected from, but not limited to, lecturers, lecturers, reciters, business negotiators, and speakers in daily communication.

[0043] For example, the audio recorded in real time on site or selected from an audio database can be segmented to obtain multiple segments of driving audio with a time sequence. Each segment of driving audio has its own duration and corresponding text content, and usually also contains the personal feelings or emotions of the speaker. A single segment of driving audio can be used to generate a single complete face generation image accordingly. After the multiple complete face generation images generated corresponding to the multiple segments of driving audio with a time sequence are spliced ​​according to the time sequence, a face generation video (also called a digital human video) can be obtained.

[0044] Furthermore, the time interval of the input driving audio covers the time interval of generating a single frame of complete face image, that is, the time interval of generating a single frame of complete face image is within the time interval of the driving audio.

[0045] Furthermore, the ratio of the duration of the driving audio to the duration of the single-frame complete face generated image can be set to be greater than or equal to 5.

[0046] Taking a speaking process video containing a face image at 25 frames per second (FPS) as an example, a single face image corresponds to an audio duration of 40ms (i.e., occupies a duration of 40ms), and the duration of the driving audio input in the embodiment of the present invention can be at least 5 times (e.g., 10 times or even dozens of times) the audio duration corresponding to the single face image. In this way, the driving audio can not only include the information of the audio segment that is time-aligned with the complete face generation image, but also include the audio information before and after the audio segment.

[0047] In an embodiment of the present invention, the time interval of the driving audio is set to cover the time interval of a single-frame complete face generation image (for example, the time interval of the driving audio can be set to be located in the middle of the time interval of a single-frame complete face generation image) to provide the front-to-back time sequence and coherent audio information of the driving audio, which helps to improve the naturalness and smoothness of the mouth shape and emotion of the complete face generation image generated by the model. Furthermore, this embodiment sets the duration of the driving audio to be much longer than the duration of a single-frame complete face generation image (at least 5 times the order of magnitude), which helps to provide richer audio information for generating the complete face generation image and further improve the image quality compared to setting the duration of the two to be equal or a smaller multiple (for example, 1 time or 2 times).

[0048] Furthermore, the audio emotion feature extraction submodel may include: a pre-trained emotion classification network and an emotion feature extraction network; in the step S12, the driving audio is input into the audio emotion feature extraction submodel for feature extraction to obtain the audio emotion feature, which may specifically include: inputting the driving audio into the pre-trained emotion classification network to obtain a predicted emotion type label; encoding the predicted emotion type label to obtain an audio emotion encoding vector; inputting the audio emotion encoding vector into the emotion feature extraction network for feature extraction to obtain the audio emotion feature.

[0049] In a specific implementation, the emotion type label can be used to indicate a specific emotion type. In some embodiments, the emotion type can be selected from, but not limited to, happiness, sadness, anger, fatigue, anxiety, tension, surprise, etc. In other embodiments, the emotion type can be selected from, but not limited to, positive emotion, neutral emotion, and negative emotion.

[0050] Without limitation, scalars or Arabic numerals may be used as emotion type labels, for example, "0" may be used to indicate an emotion type of "happy", "1" may be used to indicate an emotion type of "sad", "3" may be used to indicate an emotion type of "tired", and so on.

[0051] Furthermore, the predicted emotion type label is encoded to obtain an audio emotion coding vector, including: based on a preset emotion coding length, the predicted emotion type label is pre-encoded to obtain multiple groups of emotion sub-codes, each group of emotion sub-codes contains two identical emotion sub-codes; for each group of emotion sub-codes, the sine value of one of the emotion sub-codes is determined, and the cosine value of another emotion sub-code is determined, so as to determine the emotion coding corresponding to each emotion sub-code, and based on the obtained multiple emotion codes, the audio emotion coding vector is determined.

[0052] Specifically, the following formula can be used to determine the audio emotion coding vector:

[0053] P = [sin(2 0 πE),cos(2 0 πE),sin(2 1 πE),cos(2 1 πE)…,sin(2 L-1 πE),cos(2 L-1 πE)];

[0054] Wherein, P represents the audio emotion coding vector, E represents the predicted emotion type label, L represents the number of emotion sub-coding groups obtained by pre-coding, 2 L-1 πE represents an emotion sub-code in the Lth group of emotion sub-codes, sin(2 L-1 πE) represents the sine value of one of the emotion sub-codes in the Lth group of emotion sub-codes, cos(2 L-1 πE) represents the cosine value of another emotion sub-code in the Lth group of emotion sub-codes, and [x] represents a vector composed of x.

[0055] Furthermore, the pre-trained emotion classification network is obtained by inputting a first training data set constructed by a plurality of first sample audios and their emotion type labels into an initialized emotion classification network for training based on a first loss function; the first loss function is represented by the following expression:

[0056]

[0057] Among them, L ec represents the function value of the first loss function, i represents the sequence number of the first sample audio, N represents the total number of the first sample audio, y i represents the true emotion type label of the i-th first sample audio, represents the emotion type label predicted for the i-th first sample audio, and ln() represents the logarithm with the constant e as the base.

[0058] In the specific implementation of step S13, splicing is performed at least based on the audio content feature and the audio emotion feature to obtain the audio fusion feature.

[0059] In a specific implementation, the audio content feature and the audio emotion feature may be directly concatenated to obtain the audio fusion feature.

[0060] In another specific implementation, the face image generation model may further include a key point feature extraction sub-model; the splicing operation of step S13 may include: splicing the audio content features, audio emotion features and face key point features; wherein the face key point features are obtained by extracting key points from the face image with the lower half occluded, and inputting the extracted key points into the key point feature extraction sub-model for feature extraction.

[0061] The face image with the lower half blocked (referred to as the masked face image) may be obtained by covering the lower half of the complete face image with a mask. The shape of the mask may be set according to actual needs. For example, it may be selected from, but not limited to, a semicircle, a rectangle, or a half face shape.

[0062] It should be noted that the blocked lower half should at least include the mouth area of ​​the complete face image. For example, the area where the lower half is located may only include the lip area; or may include the area from the nose tip to the chin; or may include the area from below the eyes to the chin.

[0063] The facial key points refer to the key points of the area of ​​the complete face image except the blocked lower half area. Specifically, it can include the key points of the contour of multiple core parts and / or the key points of the area within the contour, for example, it can be selected from but not limited to: eye contour key points, eyeball center points, nose contour key points, nose tip center points, eyebrow key points, head contour key points.

[0064] Regarding the method of extracting key points from a face image with the lower half blocked, an existing face key point extraction algorithm or model may be used for extraction, and the extracted key points may generally be expressed in the form of two-dimensional coordinate points or three-dimensional coordinate points.

[0065] In some embodiments, the driving audio and the facial image with the lower half blocked may come from the same speaker and contain the same emotion. For example, a video recording may be performed on the speaker's speaking process, and an audio stream and a facial image stream may be extracted from the recorded video; then the audio stream and the facial image stream may be segmented or sampled respectively to obtain at least one segment of driving audio and at least one frame of a complete facial image with time alignment; and then the lower half of the complete facial image may be blocked.

[0066] In other embodiments, the driving audio can be extracted from a video recorded of the speaker's speaking process, and the facial image with the lower half blocked can be obtained by blocking the lower half of a complete facial image obtained by pre-modeling. For example, the complete facial image can be a high-quality facial image obtained by pre-modeling with high clarity and standard contours.

[0067] In an embodiment of the present invention, based on the combination of the features of the two dimensions of audio content features and the audio emotion features, facial key point features extracted from a facial image with the lower half occluded are further introduced. On the one hand, since the facial key point features contain important information such as the head contour, key part contours, and facial posture of the face, it helps to make the complete face generated image generated by the model more stable and standardized, and the complete face generated images in the previous and subsequent time sequences have stronger consistency. On the other hand, the facial key point features do not contain the mouth or lip shape information of the face image, which allows the model to focus on learning the lip shape information in the driving audio, while avoiding the interference of the mouth or lip shape information in the image on the model learning, so that the complete face generated image generated by the model is closer to the lip shape of the driving audio.

[0068] In a specific implementation, the extracted audio content features, audio emotion features, and facial key point features are usually in the form of matrices or vectors, and the total number of rows and / or columns of the matrices of each feature are usually consistent, which facilitates subsequent feature splicing operations.

[0069] In a non-limiting manner, the solution for splicing two features to be spliced ​​is, for example, to completely splice each row element of one feature matrix to a preset position of the same row of another feature matrix (i.e., to splice every two rows of elements with the same row number). In this way, the information of each feature can be completely retained, ensuring the integrity and validity of the information contained in the audio fusion feature.

[0070] For another example, after removing the noise elements from each row of one feature matrix, the remaining elements of the row can be spliced ​​to the preset position of the same row of another feature matrix. In this way, invalid elements can be removed, the amount of data can be reduced, and the efficiency of subsequent operations can be improved.

[0071] The preset position (ie, the splicing position or the inserting position) may be a position after the last element of each row of the spliced ​​matrix, or a position before the first element of each row, or other appropriate positions.

[0072] As a non-limiting example, each row element of the matrix of audio emotion features can be completely spliced ​​to the position after the last element of the same row of the matrix of audio content features in accordance with the original order of the row elements to obtain a preliminary splicing matrix; then each row element of the matrix of facial key point features can be completely spliced ​​to the position after the last element of the same row of the preliminary splicing matrix in accordance with the original order of the row elements.

[0073] It should be pointed out that the above-described embodiments do not constitute a limitation on the stitching method, stitching order, and stitching position of each feature. In specific implementations, other appropriate methods can also be used to perform the stitching operation as long as it is conducive to generating a complete face image with better quality.

[0074] In the specific implementation of step S14, the audio fusion features and the noisy reference facial image features are input into the diffusion sub-model for denoising to obtain the target complete facial features, wherein the noisy reference facial image features are obtained by splicing the image features of the reference facial image with the noise matrix.

[0075] Specifically, the diffusion sub-model is used to denoise the noisy reference facial image features based on the audio fusion features, and the target complete facial image features finally outputted are the denoised complete facial image features. Wherein, the noise matrix can be a preset or randomly generated matrix, for example, a Gaussian noise matrix (i.e., a noise matrix that conforms to a normal distribution). The diffusion sub-model can adopt an existing neural network model that can denoise image features. For example, it can be selected from but not limited to a U-Net model containing a multi-layer network, a fully convolutional network (FCN), etc.

[0076] In a specific implementation, the diffusion sub-model adopts a U-Net model comprising a multi-layer network; the step S14 may specifically include: inputting the noisy reference facial image feature into the first layer network of the diffusion sub-model, and inputting the audio fusion feature into each layer network of the diffusion sub-model, and using the output result of the last layer network of the diffusion sub-model as the target complete facial feature; wherein, starting from the second layer network of the diffusion sub-model, the input data of each layer network is the output data of the previous layer network and the audio fusion feature.

[0077] In an embodiment of the present invention, since the audio fusion feature is a very critical input data for guiding the model to generate a complete face generated image that accurately matches the lip shape and expression of the driving audio, therefore, by utilizing a multi-layer network structure and inputting the audio fusion feature into each layer of the diffusion sub-model, compared to adopting a single-layer network structure or only inputting the audio fusion feature into the first layer of the network, this implementation scheme helps to perform deep calculations on the audio fusion feature and the noisy reference face image feature, so as to ultimately obtain a complete face generated image that has a higher degree of match with the lip shape and expression contained in the driving audio.

[0078] Furthermore, the reference facial image includes a complete facial image and a facial image with the lower half blocked from the same speaker and with the same emotion; in the step S14, before the audio fusion features and the noisy reference facial image features are input into the diffusion sub-model for denoising, it can also include: performing feature extraction on the facial image with the lower half blocked and the complete facial image respectively to obtain partial facial image features and complete facial image features; and splicing the partial facial image features, the complete facial image features and the noise matrix to obtain the noisy reference facial image features.

[0079] Among them, regarding the manner of splicing the features of each facial image and the noise matrix, specific reference may be made to the specific description of the feature splicing in step S13 above, which will not be repeated here.

[0080] In a specific implementation, the same or different image feature extraction models may be used to perform feature extraction on the face image with the lower half blocked and the complete face image respectively.

[0081] Compared with only introducing the features of the complete face image, in an embodiment of the present invention, the features of the face image with the lower half blocked and the complete face image are combined, wherein the features of the complete face image include the basic contour information and key position information of the complete face, which helps to reduce the randomness of the complete face generated image generated by the model and avoid generating face generated images with extreme mouth shapes or emotional states; the features of the face image with the lower half blocked include the basic contours and key position information of other parts except the mouth and other areas, which helps to further enhance the stability and consistency of the complete face generated image generated by the model while avoiding the interference of the mouth information in the image on the model, so that the model can focus on learning the mouth (i.e., mouth shape, mouth movement) features that drive the audio.

[0082] Regarding the method of obtaining the facial image with the lower half blocked and the complete facial image, please refer to the relevant content of the method for obtaining the facial image with the lower half blocked and its corresponding complete facial image above, which will not be repeated here.

[0083] Furthermore, the determination of the facial image generation model includes: constructing a facial image generation model to be trained, the facial image generation model to be trained includes: an audio content feature extraction sub-model to be trained, an audio emotion feature extraction sub-model to be trained, and a diffusion sub-model to be trained; using multiple reference sample facial images, multiple second sample audios, and a sample noise matrix to construct a second training data set, wherein the reference sample facial images and the second sample audios are aligned one-to-one in time sequence; based on a second loss function, inputting the second training data set into the facial image generation model to be trained for iterative training to obtain the facial image generation model.

[0084] The second loss function is expressed by the following expression:

[0085]

[0086] Wherein, L′ represents the function value of the second loss function, t represents the hierarchical number of the network in the diffusion sub-model to be trained, M() represents the prediction noise matrix output by the t-th layer network of the diffusion sub-model to be trained, ∈ represents the sample noise matrix, indicates that ∈ follows a normal distribution, z t represents one of the input data of the t-th layer network of the diffusion sub-model to be trained, that is, the output data of the t-1-th layer network of the diffusion sub-model to be trained, C is another one of the input data of the t-th layer network of the diffusion sub-model to be trained, and C is a sample audio fusion feature obtained by splicing at least based on the sample audio content feature and the sample audio emotion feature;

[0087] ‖∈-M(z t ,t,C)‖ represents the Euclidean distance between the sample noise matrix and the predicted noise matrix output by the t-th layer network of the diffusion sub-model to be trained, Indicates an operation of obtaining an expected value of the squared value of the Euclidean distance of multiple samples. Specifically, the number of samplings can be set in combination with the actual application scenario, and the embodiment of the present invention does not limit it.

[0088] Specifically, the sample noise matrix ∈ can be a randomly generated Gaussian sample noise matrix, that is, a sample noise matrix with a normal distribution. Similar to the model inference stage, the input data of the first layer network of the diffusion sub-model to be trained specifically includes: noisy reference sample face image features and sample audio fusion features, wherein the noisy reference sample face image features are obtained by splicing the image features of the reference sample face image and the sample noise matrix.

[0089] Furthermore, the multiple reference sample facial images may include multiple complete sample facial images and multiple sample facial images with their lower halves obstructed, wherein the multiple complete sample facial images and the multiple sample facial images with their lower halves obstructed are aligned one-to-one in time sequence, and the complete sample facial images and the sample facial images with their lower halves obstructed that are aligned one-to-one in time sequence may come from the same sample speaker and have the same emotion.

[0090] Correspondingly, the noisy reference sample facial image features may be obtained by concatenating the image features of the sample facial image with the lower half blocked, the image features of the complete sample facial image, and the noise matrix.

[0091] Specifically, the sample audio content feature is obtained by inputting the second sample audio into the audio content feature extraction sub-model to be trained for feature extraction, and the sample audio emotion feature is obtained by inputting the second sample audio into the audio emotion feature extraction sub-model to be trained for feature extraction.

[0092] Among them, corresponding to the process of generating a complete face generation image by the face image generation model (ie, the model inference process), during the model training process, the splicing operation of obtaining the sample audio fusion feature C can adopt at least the following two implementation methods.

[0093] In a specific implementation, the sample audio content feature and the sample audio emotion feature may be directly concatenated to obtain the audio fusion feature C.

[0094] In another specific embodiment, the facial image generation model to be trained may also include a key point feature extraction sub-model to be trained; the splicing operation may include: splicing the sample audio content features, the sample audio emotion features and the sample facial key point features; wherein the sample facial key point features are obtained by extracting key points from the sample facial image with the lower half obscured, and inputting the extracted key points into the key point feature extraction sub-model to be trained for feature extraction.

[0095] For more details about the model training process, refer to the previous article and Figure 1 The description of the model reasoning process in each step of the illustrated embodiment will not be repeated here.

[0096] Reference Figure 2 , Figure 2 is a partial flow chart of another method for generating a face image in an embodiment of the present invention. The another method for generating a face image may include Figure 1The steps S11 to S15 shown may further include steps S21 to S24, wherein the steps S21 to S24 may be performed before step S13.

[0097] In step S21, a first number of audios whose timing is before the driving audio and a second number of audios whose timing is after the driving audio are determined, which are recorded as first audio and second audio respectively.

[0098] In a specific implementation, the first number and the second number can be appropriately set in combination with actual application scenarios. In a non-limiting manner, the first number and the second number can be selected from appropriate values ​​in the interval [5, 10], for example, 8.

[0099] In step S22, each of the first audio and the second audio is input into the audio content feature extraction sub-model for feature extraction respectively, so as to obtain corresponding multiple first audio content features and multiple second audio content features.

[0100] In step S23, a weighted operation is performed on the plurality of first audio content features, the plurality of second audio content features, and the audio content feature to obtain a fused audio content feature.

[0101] For example, a weighted sum or average value may be calculated for multiple first audio content features, multiple second audio content features, and each group of elements located at the same position in the audio content features to obtain a fused element, and all the obtained fused elements constitute the fused audio feature.

[0102] In a specific implementation, the audio content feature extraction submodel may include an audio content feature extraction network for extracting the audio content features from the driving audio, and may also include a time domain filter for performing weighted operations. The weights of the weighted operations performed by the time domain filter may be learned and obtained during the training process of the face image generation model.

[0103] In another specific implementation, a weighted operation may be performed using a preset weight, wherein the weight of the audio content feature may be greater than the weights of the first audio content feature and the second audio content feature.

[0104] In step S24, the audio content feature is updated using the fused audio content feature.

[0105] In the embodiment of the present invention, by integrating multiple audio content features of the previous and next time series into the audio content features to be spliced, in the process of generating the current complete face generation image, the audio content information of the previous and next time series can be better combined, so that the final generated complete face generation image of the previous and next time series has a smoother and more coherent lip shape and expression state. Furthermore, image splicing of the smoother previous and next time series complete face generation images helps to obtain high-quality digital human videos with more natural and coherent lip shape and expression transitions, thereby improving the user experience.

[0106] Reference Figure 3 and Figure 4 , Figure 3 is a schematic diagram of the architecture of a face image generation model in an embodiment of the present invention; Figure 4 yes Figure 3 Schematic diagram of the architecture of the diffusion sub-model in the face image generation model shown.

[0107] Without limitation, the facial image generation model includes an audio content feature extraction sub-model, an audio emotion feature extraction sub-model, a diffusion sub-model, and may also include a key point feature extraction sub-model, and may also include an encoding sub-model and a decoding sub-model.

[0108] Among them, the encoding sub-model can be used to extract features of a reference facial image to obtain reference facial image features, and the reference facial image can specifically include: a complete facial image from the same speaker and with the same emotion, and a facial image with the lower half obscured (which can be referred to as a masked facial image); the decoding sub-model can be used to decode the target complete facial features output by the extended sub-model to obtain the generated complete facial image.

[0109] Without limitation, the diffusion sub-model adopts a U-Net model comprising a multi-layer network; the input data of the diffusion sub-model includes at least two items, one of which is a noisy reference facial image feature, wherein the noisy reference facial image feature can be obtained by splicing the reference facial image feature output by the encoding sub-model and the noise matrix; the other input data is an audio fusion feature obtained by splicing the audio content feature output by the audio content extraction sub-model, the audio emotion feature output by the audio emotion extraction sub-model, and the facial key point feature output by the key point feature extraction sub-model.

[0110] Furthermore, the noisy reference facial image features are input into the first network layer of the diffusion sub-model, the audio fusion features are input into each network layer of the diffusion sub-model, and the output result of the last network layer of the diffusion sub-model is used as the target complete facial features.

[0111] For details on the functions and operation processes of other sub-models of the face image generation model, see Figure 1 to Figure 2 The relevant contents in any embodiment will not be repeated here.

[0112] Reference Figure 5 , Figure 5 1 is a schematic diagram of the structure of a facial image generation device in an embodiment of the present invention. The facial image generation device may include:

[0113] A model determination module 51 is used to determine a face image generation model, wherein the face image generation model includes an audio content feature extraction sub-model, an audio emotion feature extraction sub-model, and a diffusion sub-model;

[0114] The audio feature extraction module 52 is used to input the driving audio into the audio content feature extraction sub-model for feature extraction to obtain audio content features, and input the driving audio into the audio emotion feature extraction sub-model for feature extraction to obtain audio emotion features;

[0115] An audio feature splicing module 53, configured to perform splicing based at least on the audio content feature and the audio emotion feature to obtain an audio fusion feature;

[0116] The complete face feature determination module 54 inputs the audio fusion feature and the reference face image feature with noise into the diffusion sub-model for denoising to obtain the target complete face feature, wherein the reference face image feature with noise is obtained by concatenating the image feature of the reference face image with the noise matrix;

[0117] The face image generation module 55 is used to decode the target complete face features to obtain a complete face generated image.

[0118] For the principle, specific implementation and beneficial effects of the face image generation device, please refer to the previous article and Figure 1 to Figure 2 The related description about the face image generation method shown is not repeated here.

[0119] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the face image generation method described in any of the above embodiments are executed. The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0120] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0121] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0122] An embodiment of the present invention further provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor runs the computer program, the steps of the facial image generation method described in any of the above embodiments are executed.

[0123] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0124] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0125] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0126] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0127] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A method for generating a face image, characterized in that: include: Determine a facial image generation model, wherein the facial image generation model includes an audio content feature extraction sub-model, an audio emotion feature extraction sub-model, and a diffusion sub-model; Inputting the driving audio into the audio content feature extraction submodel to perform feature extraction to obtain audio content features, and inputting the driving audio into the audio emotion feature extraction submodel to perform feature extraction to obtain audio emotion features; At least performing splicing based on the audio content feature and the audio emotion feature to obtain an audio fusion feature; Input the audio fusion feature and the noisy reference face image feature into the diffusion sub-model for denoising to obtain the target complete face feature, wherein the noisy reference face image feature is obtained by concatenating the image feature of the reference face image with the noise matrix; Decoding the target complete face features to obtain a complete face generated image; The diffusion sub-model adopts a U-Net model including a multi-layer network; the audio fusion feature and the noisy reference facial image feature are input into the diffusion sub-model for denoising to obtain the target complete facial feature, including: inputting the noisy reference facial image feature into the first layer network of the diffusion sub-model, and inputting the audio fusion feature into each layer network of the diffusion sub-model, and using the output result of the last layer network of the diffusion sub-model as the target complete facial feature; starting from the second layer network of the diffusion sub-model, the input data of each layer network is the output data of the previous layer network and the audio fusion feature; The method of determining a facial image generation model includes: constructing a facial image generation model to be trained, wherein the facial image generation model to be trained includes: a sub-model for extracting audio content features to be trained, a sub-model for extracting audio emotion features to be trained, and a sub-model for extracting diffusion features to be trained; using a plurality of reference sample facial images, a plurality of second sample audios, and a sample noise matrix to construct a second training data set, wherein the reference sample facial images and the second sample audios are aligned one-to-one in time sequence; based on a second loss function, inputting the second training data set into the facial image generation model to be trained for iterative training to obtain the facial image generation model; The second loss function is expressed by the following expression: ； in, represents the function value of the second loss function, t represents the hierarchical number of the network in the diffusion sub-model to be trained, M() represents the predicted noise matrix output by the t-th layer network of the diffusion sub-model to be trained, represents the sample noise matrix, express It follows a normal distribution. represents one of the input data of the t-th layer network of the diffusion sub-model to be trained, that is, the output data of the t-1-th layer network of the diffusion sub-model to be trained, C is another one of the input data of the t-th layer network of the diffusion sub-model to be trained, and C is a sample audio fusion feature obtained by splicing at least based on the sample audio content feature and the sample audio emotion feature; represents the Euclidean distance between the sample noise matrix and the predicted noise matrix output by the t-th layer network of the diffusion sub-model to be trained, Represents the operation of finding the expected value of the squared Euclidean distance of multiple samples.

2. The method according to claim 1, characterized in that The face image generation model also includes a key point feature extraction sub-model; The splicing is performed at least based on the audio content feature and the audio emotion feature, including: splicing the audio content features, audio emotion features, and facial key point features; The facial key point features are obtained by extracting key points from a facial image with the lower half of the face blocked, and then inputting the extracted key points into the key point feature extraction sub-model for feature extraction.

3. The method according to claim 1 or 2, characterized in that: Before performing splicing based at least on the audio content feature and the audio emotion feature, the method further includes: Determine a first number of audios whose timing is before the driving audio, and a second number of audios whose timing is after the driving audio, which are respectively recorded as first audios and second audios; Input each of the first audio and the second audio into the audio content feature extraction sub-model to perform feature extraction respectively, to obtain corresponding multiple first audio content features and multiple second audio content features; Performing a weighted operation on the plurality of first audio content features, the plurality of second audio content features, and the audio content feature to obtain a fused audio content feature; The audio content feature is updated using the fused audio content feature.

4. The method according to claim 1, characterized in that: The reference face image includes a complete face image and a face image with the lower half blocked from the same speaker and with the same emotion; Before inputting the audio fusion features and the noisy reference face image features into the diffusion sub-model for denoising, the method further includes: Extracting features of the face image with the lower half blocked and the complete face image respectively to obtain partial face image features and complete face image features; The partial facial image features, the complete facial image features and the noise matrix are concatenated to obtain the reference facial image features with noise.

5. The method according to claim 1, characterized in that The audio emotion feature extraction sub-model includes: a pre-trained emotion classification network and an emotion feature extraction network; Input the driving audio into the audio content feature extraction sub-model to extract features, and obtain audio content features, including: Inputting the driving audio into the pre-trained emotion classification network to obtain a predicted emotion type label; Encoding the predicted emotion type label to obtain an audio emotion encoding vector; The audio emotion coding vector is input into the emotion feature extraction network for feature extraction to obtain the audio emotion feature.

6. The method according to claim 5, characterized in that Encoding the predicted emotion type label to obtain an audio emotion encoding vector includes: Based on a preset emotion coding length, the predicted emotion type label is pre-coded to obtain multiple groups of emotion sub-codes, each group of emotion sub-codes containing two identical emotion sub-codes; For each group of emotion sub-codes, the sine value of one emotion sub-code and the cosine value of another emotion sub-code are determined, so as to determine the emotion codes corresponding to each emotion sub-code, and the audio emotion coding vector is determined based on the obtained multiple emotion codes.

7. The method according to claim 5 or 6, characterized in that: The pre-trained emotion classification network is obtained by inputting a first training data set constructed by a plurality of first sample audios and their emotion type labels into an initialized emotion classification network for training based on a first loss function; The first loss function is expressed by the following expression: = ； in, represents the function value of the first loss function, i represents the sequence number of the first sample audio, N represents the total number of the first sample audio, Indicates i The true emotion type label of the first sample audio, Expressing the i The emotion type label predicted by the first sample audio, ln() represents the logarithm with the constant e as the base.

8. A facial image generating device, characterized in that: include: A model determination module, used to determine a facial image generation model, wherein the facial image generation model includes an audio content feature extraction sub-model, an audio emotion feature extraction sub-model, and a diffusion sub-model; An audio feature extraction module, used for inputting the driving audio into the audio content feature extraction sub-model for feature extraction to obtain audio content features, and for inputting the driving audio into the audio emotion feature extraction sub-model for feature extraction to obtain audio emotion features; An audio feature splicing module, used for performing splicing based on at least the audio content feature and the audio emotion feature to obtain an audio fusion feature; A complete face feature determination module, used for inputting the audio fusion feature and the reference face image feature with noise into the diffusion sub-model for denoising to obtain the target complete face feature, wherein the reference face image feature with noise is obtained by concatenating the image feature of the reference face image with the noise matrix; A face image generation module, used for decoding the target complete face features to obtain a complete face generated image; The diffusion sub-model adopts a U-Net model including a multi-layer network; the complete face feature determination module further executes: inputting the reference face image feature with noise into the first layer network of the diffusion sub-model, and inputting the audio fusion feature into each layer network of the diffusion sub-model, and using the output result of the last layer network of the diffusion sub-model as the target complete face feature; starting from the second layer network of the diffusion sub-model, the input data of each layer network is the output data of the previous layer network and the audio fusion feature; The model determination module further executes: constructing a face image generation model to be trained, the face image generation model to be trained includes: a sub-model for extracting audio content features to be trained, a sub-model for extracting audio emotion features to be trained, and a sub-model for diffusing to be trained; using a plurality of reference sample face images, a plurality of second sample audios, and a sample noise matrix to construct a second training data set, wherein the reference sample face images and the second sample audios are aligned one by one in time sequence; based on a second loss function, inputting the second training data set into the face image generation model to be trained for iterative training to obtain the face image generation model; The second loss function is expressed by the following expression: ； in, represents the function value of the second loss function, t represents the hierarchical number of the network in the diffusion sub-model to be trained, M() represents the predicted noise matrix output by the t-th layer network of the diffusion sub-model to be trained, represents the sample noise matrix, express It follows a normal distribution. represents one of the input data of the t-th layer network of the diffusion sub-model to be trained, that is, the output data of the t-1-th layer network of the diffusion sub-model to be trained, C is another one of the input data of the t-th layer network of the diffusion sub-model to be trained, and C is a sample audio fusion feature obtained by splicing at least based on the sample audio content feature and the sample audio emotion feature; represents the Euclidean distance between the sample noise matrix and the predicted noise matrix output by the t-th layer network of the diffusion sub-model to be trained, Represents the operation of finding the expected value of the squared Euclidean distance of multiple samples.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the facial image generation method according to any one of claims 1 to 7 are executed.

10. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the facial image generation method according to any one of claims 1 to 7 are performed.

Images