Method for generating a digital human having expressions, system, storage medium and program product

Abstract

The present invention provides a method for generating a digital human having expressions, a system, a storage medium and a program product. The method comprises the following steps: inputting audio and corresponding images; inputting the audio into an audio emotion extraction network to extract corresponding emotion categories; extracting corresponding templates from expression templates on the basis of the emotion categories; inputting the templates and the images together into a generation network to generate corresponding basic images with expressions; after the basic images are processed, extracting image features, and extracting audio features of the audio; and inputting the image features and the audio features into a decoding network to obtain images with expressions. The method for generating a digital human having expressions, the system, the storage medium and the program product which are provided by the present invention can introduce corresponding audio emotion extraction and expression templates, so as to implement tasks for generating digital humans having expressions and having multiple expression variations.

Description

Digital human generation method and system with expression, storage medium and program product

[0001] The present application claims priority to the Chinese patent application No. 202411304862.3, filed on September 18, 2024, and entitled "Digital human generation method and system with expression, storage medium and program product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of generative artificial intelligence technology, and in particular to a digital human generation method and system with expression, storage medium and program product. BACKGROUND

[0003] The current generative artificial intelligence (AIGC) technology is developing rapidly, and generative artificial intelligence specifically includes the generation of different content forms such as text, image, voice, and video. Digital human is a very important application in the field of generative artificial intelligence. Digital human, as the name implies, is a digital character image similar to human image created by using generative artificial intelligence related algorithms. Digital human has very wide application scenarios, including media promotion, tourism, metaverse, etc. Common digital human images can be divided into 2D and 3D digital humans according to the dimension of their presentation form, and the above two types of digital humans can be further subdivided according to whether they are virtual cartoon images or real-life replica images. The technologies used by the above different digital humans are also very different.

[0004] For 2D digital human with real-life replica image, the most mainstream technology currently uses a deep neural network model. According to different strategies, the mainstream algorithm can be divided into single-stage and two-stage categories. The single-stage model directly generates the corresponding image matching the input audio and the corresponding picture through a deep model. The two-stage model generates the final image through two stages, and the first stage generates an intermediate state, which aims to reduce the difficulty of generation. The intermediate state of the common two-stage model includes key points, expression coefficients, etc.

[0005] The current mainstream model mainly focuses on the generation task of pure digital human. For the generation task of digital human with expression and multiple expression changes, there are few related algorithm models, and the implementation difficulty is also very great. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a digital human generation method and system with expression, storage medium and program product, which aims to realize the generation of digital human with different expressions by introducing corresponding audio emotion extraction and expression template.

[0007] In a first aspect, the present application provides a digital human with expression generation method, comprising the following steps:

[0008] input audio and corresponding pictures;

[0009] extract the corresponding emotion category from the audio emotion extraction network;

[0010] extract the corresponding template from the expression template according to the emotion category;

[0011] input the template and the picture into the generation network to generate the corresponding expression-based basic picture;

[0012] After processing the basic picture, extract the image features, and at the same time, extract the audio features from the audio;

[0013] input the image features and the audio features into the decoding network to obtain the expression-based picture.

[0014] As a further improvement of the present application, the input audio and corresponding pictures include: input audio and pictures in a regular order, the audio is a single-channel audio data with a fixed sampling rate, and the picture provides a complete real person image.

[0015] As a further improvement of the present application, the audio is input into the audio emotion extraction network to extract the corresponding emotion category, which includes:

[0016] After converting the audio into a mel spectrum, input it into the audio emotion extraction network to extract the corresponding emotion category, the audio emotion extraction network is composed of convolutional layer, linear layer and attention mechanism, and the emotion category includes smile, natural, sadness, anger, frustration and surprise.

[0017] As a further improvement of the present application, the expression template includes face expression coefficient and face reconstruction coefficient, different emotion categories correspond to different face expression coefficients, and the face reconstruction coefficient is fixed;

[0018] The method for obtaining the expression template includes:

[0019] obtain training data containing different emotions;

[0020] perform three-dimensional reconstruction processing on the training data to obtain face reconstruction coefficient and face expression coefficient of different emotions;

[0021] Screen all face expression coefficients using the idea of clustering algorithm, and select a representative face expression coefficient for each emotion category.

[0022] As a further improvement of the present application, the template and the picture are input into the generation network to generate the corresponding expression-based basic picture, which includes:

[0023] Obtaining corresponding facial position information based on the picture;

[0024] Reconstructing a corresponding expression image based on the template and the facial position information;

[0025] Inputting the expression image and the picture into a trained generation network to obtain a basic picture with expression.

[0026] As a further improvement of the present application, after the basic picture is processed, image features are extracted, and audio feature extraction is performed on the audio, comprising:

[0027] The basic picture is processed to obtain a mouth region Mask image;

[0028] The Mask image and the basic picture are spliced by channel to obtain a spliced image;

[0029] An image encoder is used to extract image features from the spliced image;

[0030] An audio encoder is used to extract audio features from the audio.

[0031] As a further improvement of the present application, the image features and the audio features are input into a decoding network to obtain a picture with expression, comprising:

[0032] The image features and the audio features are fused by splicing and then input into a decoding network to obtain a picture with expression.

[0033] In a second aspect, the present application provides a computer system comprising a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method of the first aspect.

[0034] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method of the first aspect.

[0035] In a fourth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the method of the first aspect. Compared with the prior art, the present application has the beneficial effects that:

[0036] The present application provides a digital human generation method, system, storage medium and program product with expression, which can realize the generation of digital human with expression and multiple expression changes by introducing corresponding audio emotion extraction and expression template. BRIEF DESCRIPTION OF DRAWINGS

[0037] Fig. 1 is a flowchart of a method for generating an expressive digital human according to an embodiment of the present application;

[0038] Fig. 2 is a schematic diagram of generating a base picture according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the accompanying drawings. Obviously, the described embodiments are some but not all of the embodiments of the present application, and the steps S1, S2, … in the described embodiments of the present application are not the only execution steps of the present application; and the various models described in the present application are not the only limited ways of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0040] In the present application, the computer device / system refers to the related entities applied to the computer, such as hardware, combination of hardware and software, software or software in execution, etc. In detail, for example, the software includes but is not limited to process running on the processor, processor, object, executable software, execution thread, program and / or computer. In addition, the application program or script program running on the server, and the server can also be software. One or more software can be in the process and / or thread of execution, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer readable media.

[0041] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0042] In a first aspect, the present application provides a method for generating an expressive digital human, as shown in Fig. 1, and the specific steps can be as follows:

[0043] S1: input audio and corresponding pictures;

[0044] Preferably, the input audio and corresponding pictures include: inputting audio and pictures in a regular order, the audio is a single-channel audio data with a fixed sampling rate, and the pictures provide a complete real human image.

[0045] In an embodiment of the present application, the input audio is single-channel and fixed-sampling-rate audio data, the input picture is a corresponding digital human image picture, and the input picture and the audio data length can be input in sequence according to a self-defined rule. For example, assuming that the audio sampling rate is 16000 Hz and the input picture is derived from a 25 FPS video, the audio can collect 1 / 25*16000=640 data, or a multiple of 640 data, such as 3200 data.

[0046] S2: inputting the audio into an audio emotion extraction network to extract a corresponding emotion category;

[0047] Preferably, the inputting the audio into the audio emotion extraction network to extract the corresponding emotion category comprises:

[0048] After the audio is converted into a mel spectrum, the audio is input into an audio emotion extraction network to extract a corresponding emotion category, the audio emotion extraction network is composed of a convolution layer, a linear layer, and an attention mechanism, and the emotion category includes smile, natural, sadness, anger, frustration, and surprise.

[0049] In an embodiment of the present application, for example, assuming that the audio sampling rate is 16000 Hz and the input picture is derived from a 25 FPS video, the audio can collect 1 / 25*16000=640 data, and a mel spectrum can be obtained through a series of operations such as pre-emphasis, framing, windowing, and Fourier transform. The mel spectrum is input into an already trained audio emotion extraction network, the network outputs a category number, and the corresponding expression can be obtained according to the category number. For example, the structure of the network includes a convolution layer, a linear layer, and an attention mechanism. The number of categories that can be output by the network is determined by the training data, and the number of emotion categories contained in the training data is the number of output categories of the network. For example, the emotion categories include smile, natural state, sadness, anger, frustration, and surprise.

[0050] S3: extracting a corresponding template from an expression template according to the emotion category;

[0051] Preferably, the expression template includes a face expression coefficient and a face reconstruction coefficient, different emotion categories correspond to different face expression coefficients, and the face reconstruction coefficient is fixed.

[0052] The method for obtaining the expression template comprises:

[0053] Obtain training data containing different emotions;

[0054] Perform three-dimensional reconstruction processing on the training data to obtain a face reconstruction coefficient and a face expression coefficient of different emotions.

[0055] The clustering algorithm is adopted to screen all facial expression coefficients, and a representative facial expression coefficient is screened for each type of emotion.

[0056] In an embodiment of the present application, it is assumed that the audio sampling rate is 16000Hz, and the input picture is derived from a 25FPS video, and then the audio can collect 1 / 25*16000=640 data. The audio is input to the audio emotion extraction network to extract the corresponding emotion category. According to the emotion category, the corresponding template can be obtained from the determined expression template. The template can include two parts of data: facial expression coefficients and facial reconstruction coefficients. Different emotions have different facial expression coefficients, and the facial reconstruction coefficients are fixed parameters obtained from the training data set.

[0057] The expression template is obtained by: obtaining training data containing different emotions, performing three-dimensional reconstruction processing on the training data to obtain facial reconstruction coefficients and facial expression coefficients of different emotions, adopting the idea of the clustering algorithm to screen all facial expression coefficients, and screening a representative facial expression coefficient for each type of emotion.

[0058] Preferably, for a type of emotion, a representative facial expression coefficient can be calculated by the following formula:

[0059] Suppose a series of facial expression coefficients are {x1, x2,..., x n , and an arbitrary one is selected as the expression coefficient u from these data, and the purpose is to minimize the following target value: According to the above process, a representative facial expression coefficient of each type of emotion can be obtained.

[0060] S4: inputting the template and the picture into the generation network to generate a corresponding expression-based picture;

[0061] Preferably, the inputting the template and the picture into the generation network to generate a corresponding expression-based picture comprises:

[0062] obtaining corresponding facial position information based on the picture;

[0063] reconstructing to obtain a corresponding expression image based on the template and the facial position information;

[0064] inputting the expression image and the picture into the trained generation network to obtain an expression-based picture.

[0065] In an embodiment of the present application, it is assumed that the audio sampling rate is 16000Hz and the input picture is derived from a 25FPS video, and then the audio can collect 1 / 25*16000=640 data. The audio is input to the audio emotion extraction network to extract the corresponding emotion category, and the corresponding template is extracted from the expression template according to the emotion category, including the facial expression coefficient and the facial reconstruction coefficient. The above template and the initially input picture are input to the generation network to generate the corresponding expression-based basic picture.

[0066] The entire specific process of generating the corresponding expression-based basic picture can refer to FIG. 2. First, the corresponding facial position information can be obtained from the input picture, and in combination with the facial expression coefficient and the facial reconstruction coefficient in the expression template, the corresponding expression image can be reconstructed, and then the expression image and the input picture are input to the trained generation network to obtain an expression-based basic picture.

[0067] S5: After processing the basic picture, the image features are extracted, and the audio feature extraction is performed on the audio;

[0068] Preferably, the image features are extracted after processing the basic picture, and the audio feature extraction is performed on the audio, including:

[0069] The basic picture is processed to obtain a mouth region Mask image;

[0070] The Mask image and the basic picture are spliced according to the channel to obtain a spliced image;

[0071] An image encoder is used to extract the image features of the spliced image;

[0072] An audio encoder is used to extract the audio features of the audio.

[0073] In an embodiment of the present application, it is assumed that the audio sampling rate is 16000Hz and the input picture is derived from a 25FPS video, and then the audio can collect 1 / 25*16000=640 data. The audio is input to the audio emotion extraction network to extract the corresponding emotion category, and the corresponding template is extracted from the expression template according to the emotion category, including the facial expression coefficient and the facial reconstruction coefficient. The above template and the initially input picture are input to the generation network to generate the corresponding expression-based basic picture. The expression-based basic picture is processed to obtain a mouth region Mask image, and then the Mask image and the expression-based basic picture are spliced according to the channel, and then an image encoder is used to extract the image features. The input audio is input to an audio encoder to extract the audio features. The above Mask image region can be obtained according to a self-defined rule. For example, a rectangular region covering the mouth is used as the Mask image.

[0074] S6: inputting the image feature and the audio feature into a decoding network to obtain an expression picture.

[0075] Preferably, the inputting the image feature and the audio feature into a decoding network to obtain an expression picture comprises:

[0076] The image feature and the audio feature are fused by splicing and then input into a decoding network to obtain an expression picture.

[0077] In an embodiment of the present application, it is assumed that the audio sampling rate is 16000Hz and the input picture is derived from a 25FPS video, and then 640 data can be collected from 1 / 25*16000. The audio is input into an audio emotion extraction network to extract a corresponding emotion category, and a corresponding template is extracted from an expression template according to the emotion category, including a facial expression coefficient and a facial reconstruction coefficient. The above template and the initially input picture are input into a generation network to generate a corresponding expression basic picture. After processing the basic picture, an image feature is extracted, and an audio feature is extracted from the audio. The image feature and the audio feature are input into a decoding network to obtain a final expression picture, wherein the image feature and the audio feature are fused by splicing.

[0078] In a second aspect, an embodiment of the present application provides a computer system, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to realize the steps of the method of the first aspect.

[0079] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the method of the first aspect.

[0080] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is executed by a processor to realize the steps of the method of the first aspect.

[0081] The embodiment of the present application provides a digital person generation method, system, storage medium and program product with expression, which can realize the generation of digital person with expression and various expression changes by introducing corresponding audio emotion extraction and expression template.

Claims

1. An expression digital human generation method characterized by, The method comprises the following steps: inputting audio and corresponding pictures; extracting corresponding emotion categories from an audio emotion extraction network by inputting the audio; extracting corresponding templates from expression templates according to the emotion categories; generating corresponding expression-based pictures by inputting the templates and the pictures into a generation network; extracting image features after processing the expression-based pictures, and extracting audio features from the audio; inputting the image features and the audio features into a decoding network to obtain expression pictures.

2. The method of claim 1, wherein, The inputting of the audio and the corresponding pictures comprises inputting the audio and the pictures in a regular order, wherein the audio is single-channel audio data with a fixed sampling rate, and the pictures provide complete real human images.

3. The method of claim 1, wherein, The extracting of the corresponding emotion categories from the audio emotion extraction network comprises: converting the audio into a mel spectrum and inputting the mel spectrum into the audio emotion extraction network to extract the corresponding emotion categories, wherein the audio emotion extraction network is composed of a convolution layer, a linear layer and an attention mechanism, and the emotion categories include smiling, natural, sad, angry, frustrated and surprised.

4. The method of claim 1, wherein, The expression templates include face expression coefficients and face reconstruction coefficients, different emotion categories correspond to different face expression coefficients, and the face reconstruction coefficients are fixed. The method for obtaining the expression templates comprises: obtaining training data containing different emotions; performing three-dimensional reconstruction processing on the training data to obtain face reconstruction coefficients and face expression coefficients corresponding to different emotions; screening all face expression coefficients by using the idea of a clustering algorithm, and screening a representative face expression coefficient for each category of emotion.

5. The method of claim 1, wherein, The generating of the corresponding expression-based pictures by inputting the templates and the pictures into the generation network comprises: obtaining corresponding face position information based on the pictures; reconstructing to obtain corresponding expression images based on the templates and the face position information; inputting the expression images and the pictures into a trained generation network to obtain expression-based pictures.

6. The method of claim 1, wherein, The extracting of the image features after processing the expression-based pictures and the extracting of the audio features from the audio comprise: obtaining a mouth region mask image of the expression-based pictures through processing; splicing the mask image and the expression-based pictures by channels to obtain a spliced image; extracting image features from the spliced image by using an image encoder; extracting audio features from the audio by using an audio encoder.

7. The method of claim 1, wherein, The inputting of the image features and the audio features into the decoding network to obtain the expression pictures comprises: fusing the image features and the audio features by splicing, and then inputting the fused features into the decoding network to obtain the expression pictures.

8. A computer system comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 7. The processor executes the computer program to implement the steps of the method of any one of claims 1-7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-7.

Images