Method and system for generating digital human with expression, storage medium and program product

Abstract

The invention provides a method and a system for generating a digital person with an expression, a storage medium and a program product. The method comprises the following steps: inputting an audio and a corresponding picture; inputting the audio into an audio emotion extraction network to extract a corresponding emotion category; extracting a corresponding template from the expression template according to the emotion category; inputting the template and the picture into a generative network together to generate a corresponding basic picture with an expression; after the basic picture is processed, image features are extracted, and meanwhile audio features are extracted from the audio; and inputting the image features and the audio features into a decoding network to obtain a picture with expressions. The invention provides a method and a system for generating a digital person with an expression, a storage medium and a program product, which can realize a task for generating a digital person with an expression and a plurality of expression changes by introducing a corresponding audio emotion extraction and expression template.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence technology, and in particular to a method, system, storage medium and program product for generating a digital human with expression. Background Art

[0002] At present, generative artificial intelligence (AIGC) technology is developing rapidly. Generative artificial intelligence specifically includes the generation of different content forms such as text, images, voice, and video. Digital humans are a very important application in the field of generative artificial intelligence. As the name suggests, digital humans are digital characters that are close to human images and created using generative artificial intelligence-related algorithms. Digital humans have a very wide range of application scenarios, including media publicity, tourism, metaverse, and so on. Common digital human images can be divided into 2D and 3D digital humans according to the dimensions of their display forms. The above two types of digital humans can be further subdivided according to whether they are virtual cartoon images or real-life replica images. The technologies used by the above different digital humans are also very different.

[0003] For 2D digital humans that replicate the image of real people, the most mainstream technology currently uses deep neural network models. According to different strategies, mainstream algorithms can be divided into two categories: single-stage and dual-stage. The single-stage model directly uses the input audio and corresponding pictures through the deep model to generate the corresponding image that matches the audio. The dual-stage model will generate the final image through two stages, in which the first stage will generate an intermediate state, the purpose of which is to reduce the difficulty of generation. Common intermediate states of the dual-stage model include key points, expression coefficients, etc.

[0004] The current mainstream models mainly focus on the task of generating simple digital humans. For the task of generating digital humans with facial expressions and multiple expression changes, there are relatively few related algorithm models and the implementation is also very difficult. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method, system, storage medium and program product for generating a digital human with expressions, aiming to generate a digital human with different expressions by introducing corresponding audio emotion extraction and expression templates.

[0006] In a first aspect, the present invention provides a method for generating a digital human with expression, comprising the following steps:

[0007] Input audio and corresponding pictures;

[0008] Input the audio into the audio emotion extraction network to extract the corresponding emotion category;

[0009] Extract corresponding templates from expression templates according to emotion categories;

[0010] Input the template and the picture together into the generative network to generate the corresponding basic picture with expression;

[0011] After the basic image is processed, the image features are extracted, and the audio features are extracted from the audio;

[0012] The image features and audio features are input into the decoding network to obtain pictures with expressions.

[0013] As a further improvement of the present invention, the input audio and corresponding pictures include: the audio and pictures are input in order according to rules, the audio is a single-channel audio data with a fixed sampling rate, and the picture provides a complete real-life image.

[0014] As a further improvement of the present invention, the step of inputting the audio into an audio emotion extraction network to extract the corresponding emotion category includes:

[0015] After the audio is converted into Mel spectrum, it is input into the audio emotion extraction network to extract the corresponding emotion category. The audio emotion extraction network is composed of a convolutional layer, a linear layer, and an attention mechanism. The emotion categories include smile, natural, sad, angry, frustrated, and surprised.

[0016] As a further improvement of the present invention, the expression template includes a facial expression coefficient and a facial reconstruction coefficient, different emotion categories correspond to different facial expression coefficients, and the facial reconstruction coefficient is fixed;

[0017] The method for obtaining the expression template includes:

[0018] Get training data containing different emotions;

[0019] Perform three-dimensional reconstruction on the training data to obtain the facial reconstruction coefficient and facial expression coefficients of different emotions;

[0020] The idea of ​​clustering algorithm is used to screen all facial expression coefficients, and a representative facial expression coefficient is selected for each type of emotion.

[0021] As a further improvement of the present invention, the step of inputting the template and the picture together into the generation network to generate the corresponding basic picture with expression includes:

[0022] Obtain corresponding face position information based on the picture;

[0023] Based on the template and face position information, the corresponding expression image is reconstructed;

[0024] Input the expression image and the picture into the trained generative network together to obtain the basic picture with expression.

[0025] As a further improvement of the present invention, the method of extracting image features after processing the basic image and extracting audio features from the audio includes:

[0026] The basic image is processed to obtain the mask image of the mouth area;

[0027] The mask image and the base image are stitched together by channel to obtain a stitched image;

[0028] An image encoder is used to extract image features from the spliced ​​image;

[0029] An audio encoder is used to extract audio features from audio.

[0030] As a further improvement of the present invention, the step of inputting the image features and the audio features into a decoding network to obtain a picture with an expression includes:

[0031] The image features and audio features are fused by splicing and then input into the decoding network to obtain a picture with expression.

[0032] In a second aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0033] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0034] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor. Compared with the prior art, the present invention has the following advantages:

[0035] The present invention provides a method, system, storage medium and program product for generating a digital human with expressions, which can realize the task of generating a digital human with expressions and multiple expression changes by introducing corresponding audio emotion extraction and expression templates. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic flow chart of a method for generating a digital human with expression provided by one embodiment of the present invention;

[0037] Figure 2 A schematic diagram of generating a basic image provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models described in the present invention are not the only limiting methods of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0041] In a first aspect, an embodiment of the present invention provides a method for generating a digital human with expression, such as Figure 1 As shown, the specific steps can be as follows:

[0042] S1: input audio and corresponding pictures;

[0043] Preferably, the input audio and corresponding pictures include: audio and pictures are input in order according to rules, the audio is a single-channel audio data with a fixed sampling rate, and the picture provides a complete real person image.

[0044] In one embodiment of the present invention, the input audio is single-channel audio data with a fixed sampling rate, the input image is a corresponding digital human image, and the input image and audio data length can be input in a custom rule order. For example, assuming that the audio sampling rate is 16000Hz and the input image comes from a 25FPS video, the audio can collect 1 / 25*16000=640 data, or it can collect 640 multiples of data, such as 3200 data.

[0045] S2: Input the audio into the audio emotion extraction network to extract the corresponding emotion category;

[0046] Preferably, the step of inputting the audio into an audio emotion extraction network to extract corresponding emotion categories comprises:

[0047] After the audio is converted into Mel spectrum, it is input into the audio emotion extraction network to extract the corresponding emotion category. The audio emotion extraction network is composed of a convolutional layer, a linear layer, and an attention mechanism. The emotion categories include smile, natural, sad, angry, frustrated, and surprised.

[0048] In one embodiment of the present invention, exemplarily, assuming that the audio sampling rate is 16000Hz, and the input image comes from a 25FPS video, the audio can collect 1 / 25*16000=640 data, and the audio can be subjected to a series of operations such as pre-emphasis, framing, windowing, Fourier transform, etc. to obtain a Mel spectrum, and the Mel spectrum is input into a trained audio emotion extraction network, which will output a number of categories, and the corresponding expression can be obtained according to the number of categories. Exemplarily, the structure of the network includes convolutional layers, linear layers, attention mechanisms, etc. The number of categories that the network can output is determined by the training data, and the number of emotion categories contained in the training data is the number of output categories of the network. Exemplarily, emotion categories include: smile, natural state, sadness, anger, frustration, surprise, etc.

[0049] S3: Extract corresponding templates from expression templates according to emotion categories;

[0050] Preferably, the expression template includes a facial expression coefficient and a facial reconstruction coefficient, different emotion categories correspond to different facial expression coefficients, and the facial reconstruction coefficient is fixed;

[0051] The method for obtaining the expression template includes:

[0052] Get training data containing different emotions;

[0053] Perform three-dimensional reconstruction on the training data to obtain the facial reconstruction coefficient and facial expression coefficients of different emotions;

[0054] The idea of ​​clustering algorithm is used to screen all facial expression coefficients, and a representative facial expression coefficient is selected for each type of emotion.

[0055] In one embodiment of the present invention, illustratively, assuming that the audio sampling rate is 16000 Hz and the input image comes from a 25FPS video, the audio can collect 1 / 25*16000=640 data. The audio is input into the audio emotion extraction network to extract the corresponding emotion category. According to the emotion category, we can obtain the corresponding template from the determined expression template. Exemplarily, the template can contain two parts of data: a facial expression coefficient and a facial reconstruction coefficient. Among them, different emotions will have different facial expression coefficients, and the facial reconstruction coefficient is a fixed parameter obtained from the training data set.

[0056] Exemplarily, the expression template is obtained by obtaining training data containing different emotions, performing three-dimensional reconstruction on the training data to obtain face reconstruction coefficients and facial expression coefficients of different emotions, using the idea of ​​clustering algorithm to screen all facial expression coefficients, and screening out a representative facial expression coefficient for each type of emotion.

[0057] Preferably, for a type of emotion, a representative facial expression coefficient is selected and calculated using the following formula:

[0058] Assume that a series of facial expression coefficients are {x1,x2,…,x n}, then any one of these data is selected as the expression coefficient u, the purpose of which is to minimize the following target value: According to the above process, the representative facial expression coefficient of each type of emotion can be obtained.

[0059] S4: Input the template and the image together into the generative network to generate the corresponding basic image with expression;

[0060] Preferably, the step of inputting the template and the picture together into a generation network to generate a corresponding basic picture with expression includes:

[0061] Obtain corresponding face position information based on the picture;

[0062] Based on the template and face position information, the corresponding expression image is reconstructed;

[0063] Input the expression image and the picture into the trained generative network together to obtain the basic picture with expression.

[0064] In one embodiment of the present invention, exemplarily, assuming that the audio sampling rate is 16000 Hz and the input picture comes from a 25FPS video, the audio can collect 1 / 25*16000=640 data. The audio is input into the audio emotion extraction network to extract the corresponding emotion category, and the corresponding template is extracted from the expression template according to the emotion category, including the facial expression coefficient and the facial reconstruction coefficient. The above template and the initially input picture are input into the generation network together to generate the corresponding basic picture with expression.

[0065] The entire specific process of generating the corresponding basic pictures with expressions can be referred to Figure 2 Among them, firstly, the corresponding face position information can be obtained from the input image, and the corresponding expression image can be reconstructed by combining the facial expression coefficient and the face reconstruction coefficient in the expression template. Then, the expression image and the input image are input into the trained generation network to obtain a basic image with expression.

[0066] S5: After the basic image is processed, the image features are extracted, and the audio features are extracted from the audio;

[0067] Preferably, after the basic image is processed, image features are extracted, and audio features are extracted from the audio, including:

[0068] The basic image is processed to obtain the mask image of the mouth area;

[0069] The mask image and the base image are stitched together by channel to obtain a stitched image;

[0070] An image encoder is used to extract image features from the spliced ​​image;

[0071] An audio encoder is used to extract audio features from audio.

[0072] In one embodiment of the present invention, exemplarily, assuming that the audio sampling rate is 16000Hz, and the input image comes from a 25FPS video, the audio can collect 1 / 25*16000=640 data. The audio is input into the audio emotion extraction network to extract the corresponding emotion category, and the corresponding template is extracted from the expression template according to the emotion category, including the facial expression coefficient and the facial reconstruction coefficient. The above template and the initially input image are input into the generation network together to generate the corresponding basic image with expression. The basic image with expression can be processed to obtain a mask image of the mouth area, and then the mask image and the basic image with expression are spliced ​​by channel, and then passed through an image encoder for image feature extraction; and the input audio is passed through an audio encoder for audio feature extraction. The above-mentioned mask image area can be obtained according to custom rules. For example, a rectangular area covering the mouth is used as a mask image.

[0073] S6: Input the image features and audio features into the decoding network to obtain a picture with expression.

[0074] Preferably, the step of inputting the image features and the audio features into a decoding network to obtain a picture with an expression comprises:

[0075] The image features and audio features are fused by splicing and then input into the decoding network to obtain pictures with expressions.

[0076] In one embodiment of the present invention, exemplarily, assuming that the audio sampling rate is 16000Hz, and the input picture comes from a 25FPS video, the audio can collect 1 / 25*16000=640 data. The audio is input into the audio emotion extraction network to extract the corresponding emotion category, and the corresponding template is extracted from the expression template according to the emotion category, including the facial expression coefficient and the facial reconstruction coefficient. The above template and the initially input picture are input into the generation network together to generate the corresponding basic picture with expression. After the basic picture is processed, the image features are extracted, and the audio features are extracted from the audio at the same time. The image features and audio features are input into the decoding network to obtain the final picture with expression, in which the image features and audio features are fused by splicing.

[0077] In a second aspect, an embodiment of the present invention provides a computer system, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0078] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0079] In a fourth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0080] The embodiments of the present invention provide a method, system, storage medium and program product for generating a digital human with expressions, which can realize the task of generating a digital human with expressions and multiple expression changes by introducing corresponding audio emotion extraction and expression templates.

Claims

1. A method for generating a digital human with expression, characterized in that: The following steps are involved: Input audio and corresponding pictures; Input the audio into the audio emotion extraction network to extract the corresponding emotion category; Extract corresponding templates from expression templates according to emotion categories; Input the template and the picture together into the generative network to generate the corresponding basic picture with expression; After the basic image is processed, the image features are extracted, and the audio features are extracted from the audio; The image features and audio features are input into the decoding network to obtain pictures with expressions.

2. The method for generating a digital human with expression according to claim 1, characterized in that: The input audio and corresponding pictures include: audio and pictures are input in order according to rules, the audio is a single-channel audio data with a fixed sampling rate, and the picture provides a complete real person image.

3. The method for generating a digital human with expression according to claim 1, characterized in that: The step of inputting the audio into the audio emotion extraction network to extract the corresponding emotion category includes: After the audio is converted into Mel spectrum, it is input into the audio emotion extraction network to extract the corresponding emotion category. The audio emotion extraction network is composed of a convolutional layer, a linear layer, and an attention mechanism. The emotion categories include smile, natural, sad, angry, frustrated, and surprised.

4. The method for generating a digital human with expression according to claim 1, characterized in that: The expression template includes a facial expression coefficient and a facial reconstruction coefficient. Different emotion categories correspond to different facial expression coefficients, and the facial reconstruction coefficient is fixed. The method for obtaining the expression template includes: Get training data containing different emotions; Perform three-dimensional reconstruction on the training data to obtain the facial reconstruction coefficient and facial expression coefficients of different emotions; The idea of ​​clustering algorithm is used to screen all facial expression coefficients, and a representative facial expression coefficient is selected for each type of emotion.

5. The method for generating a digital human with expression according to claim 1, characterized in that: The step of inputting the template and the picture together into the generation network to generate the corresponding basic picture with expression includes: Obtain corresponding face position information based on the picture; Based on the template and face position information, the corresponding expression image is reconstructed; Input the expression image and the picture into the trained generative network together to obtain the basic picture with expression.

6. The method for generating a digital human with expression according to claim 1, characterized in that: After the basic image is processed, the image features are extracted, and the audio features are extracted from the audio, including: The basic image is processed to obtain the mask image of the mouth area; The mask image and the base image are stitched together by channel to obtain a stitched image; An image encoder is used to extract image features from the spliced ​​image; An audio encoder is used to extract audio features from audio.

7. The method for generating a digital human with expression according to claim 1, characterized in that: The step of inputting the image features and the audio features into the decoding network to obtain a picture with an expression includes: The image features and audio features are fused by splicing and then input into the decoding network to obtain pictures with expressions.

8. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images