Digital human generation method and device, equipment and medium

Abstract

The invention relates to the technical field of computer vision, and provides a digital human generation method, device and equipment and a medium, and the method comprises the steps: obtaining the appearance information, audio information, emotion information and posture information of a to-be-generated digital human; inputting the appearance information, the audio information, the emotion information and the posture information of the to-be-generated digital person into a digital person generation model to obtain a digital person generation result output by the digital person generation model; wherein the digital human generation model is obtained by training based on a digital human training sample and a digital human video tag corresponding to the digital human training sample; and the digital human generation model is used for performing feature extraction on appearance information, audio information, emotion information and posture information of an input to-be-generated digital human, and performing feature fusion on the extracted features to generate a corresponding digital human video based on the fused features. According to the method, rich emotional expressions and diversified postures can be given to the generated digital human, so that the generated digital human is more harmonious and natural.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a digital person generation method and device, equipment and medium. BACKGROUND

[0002] With the development of artificial intelligence (AI) technology, generative artificial intelligence (AIGC) creation tools have become an important technology to assist humans, greatly improving technological productivity and work efficiency. Virtual person (or digital person) technology is an important part of AIGC, which builds virtual character images based on deep learning and other technologies, and has the same interactive ability as a person.

[0003] Currently, digital person generation technology mainly generates digital person images and videos through deep learning models. These technologies usually rely on a large amount of training data and complex model architectures to achieve precise control of digital person actions and expressions.

[0004] However, the above-mentioned method has deficiencies in multi-modal interaction and cannot achieve the synchronous generation of text, voice, action and other modalities, resulting in a lack of naturalness and fluency of digital people when interacting with users, and failing to provide an immersive interactive experience. SUMMARY

[0005] The present application provides a digital person generation method, device, equipment and medium to solve the defects of the prior art that multi-modal interaction has deficiencies, resulting in a lack of naturalness and fluency of digital people when interacting with users, and can give the generated digital person rich emotional expression and diversified posture, making the generated digital person more coordinated and natural.

[0006] The present application provides a digital person generation method, comprising: obtaining appearance information, audio information, emotional information and posture information of a digital person to be generated; inputting the appearance information, audio information, emotional information and posture information of the digital person to be generated into a digital person generation model to obtain a digital person generation result output by the digital person generation model; wherein the digital person generation model is trained based on a digital person training sample and a digital person video label corresponding to the digital person training sample; the digital person generation model is used to extract features of the input appearance information, audio information, emotional information and posture information of the digital person to be generated respectively, and fuse the extracted features to generate a corresponding digital person video based on the fused features.

[0007] According to the digital human generation method provided by the application, the digital human generation model comprises a feature extraction layer, a speech generation layer and a digital human generation layer, appearance information, audio information, emotional information and posture information of a digital human to be generated are input into the digital human generation model, and a digital human generation result output by the digital human generation model is obtained, comprising: when the type of the audio information is determined as speech, the audio information and the appearance information, emotional information and posture information are input into the feature extraction layer for feature extraction respectively, and corresponding audio features, appearance reference features, emotional features and posture features are obtained; when the type of the audio information is determined as text, the audio information is input into the speech generation layer for speech generation, and corresponding audio features are obtained, and the appearance information, emotional information and posture information are input into the feature extraction layer for feature extraction respectively, and corresponding appearance reference features, emotional features and posture features are obtained; the appearance reference features, emotional features, posture features and audio features are input into the digital human generation layer for feature fusion, and digital human generation is performed based on the fused features, and a digital human generation result output by the digital human generation layer is obtained.

[0008] According to the digital human generation method provided by the application, the digital human generation layer comprises a digital human generation unit and at least two sequentially arranged feature fusion units, appearance reference features, emotional features, posture features and audio features are input into the digital human generation layer for feature fusion, and digital human generation is performed based on the fused features, and a digital human generation result output by the digital human generation layer is obtained, comprising: the appearance reference features, emotional features, posture features and audio features are input into each feature fusion unit respectively, and the output result of a preceding feature fusion unit is input into a subsequent feature fusion unit, and the output result of a tail feature fusion unit is input into the digital human generation unit; wherein, for a head feature fusion unit, feature fusion is performed according to the input appearance reference features, emotional features, posture features and audio features, corresponding fused features are obtained, and the fused features are input into a subsequent feature fusion unit as the output result; for a non-head feature fusion unit, feature fusion is performed according to the input appearance reference features, emotional features, posture features, audio features and the output result of a preceding feature fusion unit, corresponding fused features are obtained, and the fused features are input into a subsequent feature fusion unit or the digital human generation unit as the output result; for the digital human generation unit, digital human video generation is performed according to the fused features output by the tail feature fusion unit, and a digital human generation result is obtained.

[0009] According to the digital human generation method provided by the application, the appearance reference feature, the emotion feature, the posture feature and the audio feature are fused according to the input, or the appearance reference feature, the emotion feature, the posture feature, the audio feature and the output result of the previous feature fusion unit are fused to obtain the corresponding fusion feature, including: fusing the input appearance reference feature, emotion feature, posture feature and audio feature, or fusing the input appearance reference feature, emotion feature, posture feature, audio feature and output result of the previous feature fusion unit, and converting the fusion feature through an embedding layer to obtain an embedding vector; adding position coding to the embedding vector to obtain an encoded vector; using a self-attention mechanism to obtain a self-attention output according to the encoded vector; performing residual connection on the self-attention layer output and the encoded vector, and performing layer normalization on the residual connection result to obtain a layer normalization result; using a feedforward neural network to perform nonlinear transformation according to the layer normalization result, and performing residual connection and layer normalization on the nonlinear transformation result and the layer normalization result to obtain the corresponding feature representation.

[0010] According to the digital human generation method provided by the application, the appearance reference feature, the emotion feature, the posture feature and the audio feature are fused according to the input, or the appearance reference feature, the emotion feature, the posture feature, the audio feature and the output result of the previous feature fusion unit are fused to obtain the corresponding fusion feature, including: fusing the input appearance reference feature, emotion feature, posture feature and audio feature, or fusing the input appearance reference feature, emotion feature, posture feature, audio feature and output result of the previous feature fusion unit, and converting the fusion feature through an embedding layer to obtain an embedding vector; adding position coding to the embedding vector to obtain an encoded vector; using a self-attention mechanism to obtain a self-attention output according to the encoded vector; performing residual connection on the self-attention layer output and the encoded vector, and performing layer normalization on the residual connection result to obtain a layer normalization result; using a feedforward neural network to perform nonlinear transformation according to the layer normalization result, and performing residual connection and layer normalization on the nonlinear transformation result and the layer normalization result to obtain the corresponding feature representation.

[0011] According to the digital human generation method provided by the application, before inputting the to-be-generated digital human information into the digital human generation model, the digital human training sample and the digital human video label corresponding to the digital human training sample are obtained, the digital human training sample includes audio text training information, appearance image training information, emotion training information and posture training information, and the digital human video label includes a digital human video sample and a digital human audio sample; when the type of the audio text training information is determined as voice, the audio text training information and the appearance image training information, the emotion training information and the posture training information are input into the feature extraction layer for feature extraction respectively, to obtain the appearance sample feature, the audio training feature, the emotion training feature and the posture training feature output by the feature extraction layer; when the type of the audio text training information is determined as text, the audio text training information is input into the voice generation layer to predict a first position representation of the corresponding audio in the embedding space based on the audio text training information, to obtain the audio training feature, and to predict a second position representation of the corresponding audio in the embedding space based on the audio text training information, and the appearance image training information, the emotion training information and the posture training information are input into the feature extraction layer for feature extraction respectively, to obtain the appearance sample feature, the audio training feature, the emotion training feature and the posture training feature output by the feature extraction layer; wherein the embedding space is used to represent a high-dimensional vector space for mapping different types of data into continuous vector representations; according to the first position representation and the second position representation, a target loss function is obtained in combination with an optimization objective; the audio training feature, the image sample feature, the emotion training feature and the posture training feature are input into the digital human generation layer to obtain a digital human generation video output by the digital human generation layer; according to the digital human generation video and the digital human video label corresponding to the digital human training sample, a video loss function is constructed; according to the target loss function and the video loss function, a total loss function is obtained, and based on the minimum total loss function, a corresponding trained digital human generation model is obtained.

[0012] According to the digital human generation method provided by the application, the appearance information, audio information, emotion information and posture information of the to-be-generated digital human are obtained, including: obtaining the appearance information, audio information, emotion label and posture label of the to-be-generated digital human; according to the emotion label, corresponding emotion resources are extracted from a preset emotion database to obtain corresponding emotion information; wherein the preset emotion database is constructed in advance based on each emotion resource corresponding to different emotion labels; according to the posture label, corresponding posture information is extracted from a preset posture database; wherein the preset posture database is constructed in advance based on the posture information corresponding to different posture labels.

[0013] The application further provides a digital human generation device, comprising: an information acquisition module, which acquires appearance information, audio information, emotion information and posture information of a digital human to be generated; a digital human generation module, which inputs the appearance information, audio information, emotion information and posture information of the digital human to be generated into a digital human generation model to obtain a digital human generation result output by the digital human generation model; wherein the digital human generation model is trained based on a digital human training sample and a digital human video label corresponding to the digital human training sample; and the digital human generation model is used to respectively extract features of the input appearance information, audio information, emotion information and posture information of the digital human to be generated, and perform feature fusion on the extracted features to generate a corresponding digital human video based on the fused features.

[0014] The application further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the digital human generation method according to any one of the above when executing the computer program.

[0015] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the digital human generation method according to any one of the above.

[0016] The application further provides a computer program product, comprising a computer program, wherein the computer program is executable on a processor to implement the digital human generation method according to any one of the above.

[0017] The digital human generation method, device, equipment and medium provided by the application can provide a comprehensive data basis for subsequent digital human generation by acquiring appearance, audio, emotion and posture information of a digital human to be generated, ensure that the generated digital human is not only an appearance model, but also a complete entity with sound, emotion and action, and allow a user to customize the appearance, sound, emotion and posture of the digital human according to requirements, so that the generated digital human is more personalized and meets the requirements of a specific scene, and further utilize a digital human generation model to extract features in the input information and fuse the extracted features of different modalities to endow the generated digital human with rich emotional expression and diversified posture, so that the generated digital human is more coordinated and natural, and a large amount of time and computing resources are saved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0019] Figure 1 is a flowchart of a digital person generation method provided by the present application. Figure 2 is an architecture diagram of a digital person generation model provided by the present application. Figure 3 is a structural diagram of a digital person generation device provided by the present application. Figure 4 is a structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0021] Figure 1 is a flowchart of a digital person generation method provided by the present application, as shown in Figure 1 , the method comprises: S11, obtaining appearance information, audio information, emotional information and posture information of a digital person to be generated; S12, inputting the appearance information, audio information, emotional information and posture information of the digital person to be generated into a digital person generation model to obtain a digital person generation result output by the digital person generation model; wherein the digital person generation model is trained based on a digital person training sample and a digital person video label corresponding to the digital person training sample; the digital person generation model is used for respectively extracting features of the input appearance information, audio information, emotional information and posture information of the digital person to be generated, and performing feature fusion on the extracted features, so as to generate a corresponding digital person video based on the fused features.

[0022] It should be noted that the step number "S1N" in the present specification does not represent the sequence of the digital person generation method, which will be described below in detail. Figure 2 The digital person generation method of the present application is described.

[0023] Step S11, obtaining appearance information, audio information, emotional information and posture information of a digital person to be generated.

[0024] In the embodiment, the appearance information, the audio information, the emotion information and the posture information of the digital human to be generated are obtained, including: obtaining appearance information, audio information, emotion labels and posture labels of the digital human to be generated; extracting corresponding emotion resources from a preset emotion database according to the emotion labels to obtain corresponding emotion information; wherein the preset emotion database is constructed in advance based on each emotion resource corresponding to different emotion labels; and extracting corresponding posture information from a preset posture database according to the posture labels; wherein the preset posture database is constructed in advance based on posture information corresponding to different posture labels.

[0025] It should be noted that by obtaining the appearance and audio of the digital human to be generated, the image and voice of the digital human are customized according to specific needs, so that they are more in line with the needs of target audiences or application scenarios, and by extracting resources from corresponding preset databases through emotion labels and posture labels, the digital human is endowed with rich emotional expression, so that the digital human can exhibit diversified postures, enhance its expressiveness and interactivity, make it more lively and more infectious, and improve user experience.

[0026] In step S12, the appearance information, audio information, emotion information and posture information of the digital human to be generated are input into a digital human generation model to obtain a digital human generation result output by the digital human generation model; wherein the digital human generation model is trained based on digital human training samples and digital human video labels corresponding to the digital human training samples; the digital human generation model is used to extract features of the input appearance information, audio information, emotion information and posture information of the digital human to be generated respectively, and fuse the extracted features to generate a corresponding digital human video based on the fused features.

[0027] In the embodiment, the digital human generation model includes a feature extraction layer, a voice generation layer and a digital human generation layer, the appearance information, audio information, emotion information and posture information of the digital human to be generated are input into the digital human generation model to obtain a digital human generation result output by the digital human generation model, including: when the type of the audio information is determined to be voice, the audio information and the appearance information, emotion information and posture information are input into the feature extraction layer for feature extraction respectively to obtain corresponding audio features, appearance reference features, emotion features and posture features; when the type of the audio information is determined to be text, the audio information is input into the voice generation layer for voice generation to obtain corresponding audio features, and the appearance information, emotion information and posture information are input into the feature extraction layer for feature extraction respectively to obtain corresponding appearance reference features, emotion features and posture features; the appearance reference features, emotion features, posture features and audio features are input into the digital human generation layer for feature fusion, and the digital human is generated based on the fused features to obtain a digital human generation result output by the digital human generation layer.

[0028] It should be noted that in order to facilitate the generation of corresponding audio features based on the input audio information, when it is determined that the type of audio information is text, before inputting the audio information into the speech generation layer for speech generation, the text is preprocessed, including word segmentation, part-of-speech tagging, etc., to extract the semantic information of the text. Assuming that the input text is T, the preprocessed text is T' represented as: T' = Text Preprocessing(T) In addition, the preprocessed text is input into the speech generation layer to gradually convert noise into audio features along the velocity vector field obtained by pre-training the speech generation layer from random noise distribution according to the text information. The audio features can include pitch, speech rate, volume, and other parameter features of the audio to generate natural and fluent speech. Assuming that the preprocessed text is T', the generated audio features are S represented as: S = Text-to-Speech Model( T') In addition, after obtaining the emotion features, posture features, and audio features, the required emotions, postures, and speech of each frame of the digital human video to be generated can be determined according to the requirements of the digital human video to be generated, and time stamps are assigned to the obtained emotion features, posture features, and audio features, so that subsequent time sequence alignment of multi-modal features in each video frame can be realized according to the time stamps, ensuring that the corresponding generated digital human video frames are generated according to the required emotions, postures, and speech, and the generated digital human is endowed with rich emotional expression and diverse postures.

[0029] It should be noted that in order to increase the diversity of use scenarios and the richness of emotion control, the corresponding emotion information can be obtained by multiple types, including text, image, and speech. Accordingly, when extracting features from emotion information, corresponding emotion features are extracted for each type.

[0030] Further, when the emotion information is text, the corresponding feature extraction layer can use a pre-trained sentiment analysis model to extract emotion features from the text. Assuming that the input text is T, the emotion features are E t represented as: E t = Sentiment Analysis Model ( T ) When the emotion information is speech, the corresponding feature extraction layer can use a pre-trained audio encoder (such as emotion2vec) to extract emotion features from the audio. Assuming that the input audio is A, the emotion features are E a represented as: E a = Audio Encoder( A ) When the emotional information is an image, a corresponding feature extraction layer can use a pre-trained expression encoder to extract emotional features from the image. Assuming that the input image is I, the emotional features are E i is expressed as: E i = Expression Encoder( I ) It is worth noting that specific emotional feature extraction can be performed according to the type actually involved and its corresponding pre-trained encoder or model, which is not further limited here.

[0031] In addition, the feature extraction of the posture information can use a pre-trained pose encoder to convert the posture information into a posture feature vector. Assuming that the input posture information is P, the posture features are F p is expressed as: F p = Pose Encoder(P) Specifically, referring to Figure 2 , the digital human generation layer includes a digital human generation unit and at least two sequentially arranged feature fusion units. Accordingly, the appearance reference features, emotional features, posture features, and audio features are input into the digital human generation layer for feature fusion, and a digital human is generated based on the fused features to obtain a digital human generation result output by the digital human generation layer, including: the appearance reference features, emotional features, posture features, and audio features are input into each feature fusion unit, and the output result of a preceding feature fusion unit is input into a subsequent feature fusion unit, and the output result of a tail feature fusion unit is input into the digital human generation unit; wherein, for a head feature fusion unit, the appearance reference features, emotional features, posture features, and audio features are fused according to the input to obtain corresponding fused features, which are input into a subsequent feature fusion unit as output results; for a non-head feature fusion unit, the appearance reference features, emotional features, posture features, audio features, and output results of a preceding feature fusion unit are fused according to the input to obtain corresponding fused features, which are input into a subsequent feature fusion unit or the digital human generation unit as output results; and for the digital human generation unit, the fused features output by the tail feature fusion unit are used to generate a digital human video to obtain a digital human generation result.

[0032] It should be noted that through the multiple sequentially arranged feature fusion units, gradual fusion of features is achieved, each feature fusion unit integrates the fusion results of the previous stage and new input features, so that the final fused features contain more rich information, enabling the digital human to more subtly express the association between different features and improving the quality of digital human generation.

[0033] Further, the feature fusion is performed according to the input appearance reference feature, the emotion feature, the posture feature, and the audio feature, or the feature fusion is performed according to the input appearance reference feature, the emotion feature, the posture feature, the audio feature, and the output result of the previous feature fusion unit, to obtain corresponding fusion features, including: performing the feature fusion on the input appearance reference feature, the emotion feature, the posture feature, and the audio feature, or performing the feature fusion on the input appearance reference feature, the emotion feature, the posture feature, the audio feature, and the output result of the previous feature fusion unit, and converting the fusion features through an embedding layer to obtain embedding vectors; adding position encoding to the embedding vectors to obtain encoding vectors; obtaining self-attention output by using a self-attention mechanism according to the encoding vectors; performing residual connection on the self-attention layer output and the encoding vectors, and performing layer normalization on the residual connection result to obtain layer normalization result; performing nonlinear transformation on the layer normalization result by using a feedforward neural network, and performing residual connection and layer normalization on the nonlinear transformation result and the layer normalization result to obtain corresponding feature representation.

[0034] It should be noted that the embedding layer can be a fully connected layer or a pre-trained embedding matrix, which can be configured according to actual design requirements or prior experience. The position encoding can be absolute position encoding such as sine, cosine function, or relative position encoding, which is not limited further herein.

[0035] In addition, the self-attention output is obtained by using a self-attention mechanism according to the encoding vectors, including: generating a query vector (Q), a key vector (K), and a value vector (V) by passing the encoding vectors through three different fully connected layers respectively; obtaining self-attention weights by combining a softmax operation according to the query vector (Q) and the key vector (K); and generating self-attention output by weighted summation of the self-attention weights and the value vector (V).

[0036] Further, the self-attention weights are obtained by combining a softmax operation according to the query vector (Q) and the key vector (K), including: calculating the dot product between the query vector (Q) and the key vector (K), dividing by the square root of the key vector dimension for scaling, and converting the scaled dot product result into self-attention weights by a softmax operation.

[0037] In an optional embodiment, the feature fusion is performed on the input appearance reference feature, the emotion feature, the posture feature, and the audio feature to obtain corresponding fusion features, including: splicing the input appearance reference feature, the emotion feature, the posture feature, and the audio feature to obtain corresponding fusion features. It should be noted that the features belonging to the same digital human video frame are spliced together according to the timestamps of the multi-modal features, and the spliced features belonging to different digital human video frames are spliced together according to the video time sequence, so as to accurately generate corresponding digital human video frame sequences according to the fusion features subsequently.

[0038] In addition, the input appearance reference features, emotion features, posture features, audio features, and output results of the previous feature fusion unit are fused to obtain corresponding fusion features, including: input appearance reference features, emotion features, posture features, and audio features are respectively fused with the corresponding modal features in the output results of the previous feature fusion unit according to the corresponding modalities to obtain the fusion features of the corresponding modalities; wherein the modality is used to represent the type of the corresponding feature; the fusion features of each modality are spliced to obtain the corresponding fusion features. It should be noted that the splicing of the fusion features of each modality can refer to the manner of splicing the input appearance reference features, emotion features, posture features, and audio features described above, which will not be repeated here.

[0039] In an optional embodiment, after obtaining the fusion features output by the tail feature fusion unit, it further includes: in order to adapt to the input requirements of the digital human generation unit, the fusion features are converted into the input format required by the digital human generation unit through a multi-layer perceptron (MLP). Assuming that the comprehensive features are F, and the converted features are F’ represented as: F’ = MLP(F) In an optional embodiment, for the digital human generation unit, according to the fusion features output by the tail feature fusion unit, the digital human video is generated to obtain the digital human generation result, including: according to the fusion features output by the tail feature fusion unit, the video frame conversion is performed to obtain a plurality of digital human video frames; the plurality of digital human video frames are spliced in time sequence to obtain the digital human generation result.

[0040] It should be noted that the digital human generation unit can adopt a multi-modal conditional portrait animation generation OmniHuman model, and the fusion features output by the tail feature fusion unit are used as the input of the model to generate the corresponding digital human video according to the input fusion features. Assuming that the generated digital human video is V, then: V = OmniHuman Model(F’) In addition, in the process of generating the digital human video, the emotion features are injected into the digital human generation process to control the emotional expression of the generated digital human video, so as to ensure that the generated video has corresponding emotional expression.

[0041] In an optional embodiment, before the digital human information to be generated is input into the digital human generation model, the method comprises: before the digital human information to be generated is input into the digital human generation model, obtaining a digital human training sample and a digital human video label corresponding to the digital human training sample, the digital human training sample comprising audio text training information, appearance image training information, emotion training information and posture training information, and the digital human video label comprising a digital human video sample and a digital human audio sample; when the type of the audio text training information is speech, inputting the audio text training information and the appearance image training information, the emotion training information and the posture training information into a feature extraction layer for feature extraction respectively to obtain appearance sample features, audio training features, emotion training features and posture training features output by the feature extraction layer; when the type of the audio text training information is text, inputting the audio text training information into a speech generation layer to predict a first position representation of a corresponding audio in an embedding space based on the audio text training information to obtain an audio training feature, and predict a second position representation of the corresponding audio in the embedding space based on the audio text training information, and inputting the appearance image training information, the emotion training information and the posture training information into the feature extraction layer for feature extraction respectively to obtain the appearance sample features, the audio training features, the emotion training features and the posture training features output by the feature extraction layer; wherein the embedding space is used to represent a high-dimensional vector space that maps different types of data into continuous vector representations; obtaining a target loss function based on the first position representation and the second position representation in combination with an optimization objective; inputting the audio training features, the image sample features, the emotion training features and the posture training features into a digital human generation layer to obtain a digital human generation video output by the digital human generation layer; constructing a video loss function based on the digital human generation video and the digital human video label corresponding to the digital human training sample; obtaining a total loss function based on the target loss function and the video loss function, and obtaining a trained digital human generation model based on the minimum total loss function.

[0042] Specifically, predicting a first position representation of a corresponding audio in an embedding space based on audio text training information, and predicting a second position representation of the corresponding audio in the embedding space based on the audio text training information, comprises: based on the audio text training information, starting from a first random noise distribution, gradually converting noise into audio along an initialized velocity vector field to obtain a first position representation of a predicted audio corresponding to the audio text training information in the embedding space; based on the audio text training information, starting from a second random noise distribution, gradually converting noise into audio along an initialized velocity vector field to obtain a second position representation of a predicted audio corresponding to the audio text training information in the embedding space; wherein the second random noise distribution is different from the first random noise distribution.

[0043] It should be noted that the velocity vector field is defined through the conversion between the audio text training information and the corresponding audio sample, so as to train the velocity vector field of the speech generation model through iteration, make the speech generation model learn the mapping relationship between the audio text training information and the audio information corresponding to the audio text training information, optimize the velocity vector field, realize data flow matching in the speech generation process, and make the speech generation process more stable and efficient.

[0044] In addition, the digital human training sample includes random noise in addition to all the contents included in the to-be-generated digital human information, so as to help the model capture subtle changes in the data distribution by learning how to convert the random noise into meaningful output, generate more realistic and delicate results, increase the diversity of the generated results, and improve the generation quality.

[0045] To sum up, the embodiment of the present application provides a comprehensive data basis for subsequent digital human generation by obtaining the appearance, audio, emotion and posture information of the to-be-generated digital human, ensures that the generated digital human is not only an appearance model, but also a complete entity with voice, emotion and action, and allows users to customize the appearance, voice, emotion and posture of the digital human according to the needs, so that the generated digital human is more personalized and meets the needs of a specific scene, and further uses the digital human generation model to extract the features in the input information and fuse the extracted features of different modalities to endow the generated digital human with rich emotional expression and diversified posture, so that the generated digital human is more coordinated and natural, and a large amount of time and computing resources are saved.

[0046] The digital human generation device provided by the present application is described below, and the digital human generation device described below can be correspondingly referred to the digital human generation method described above.

[0047] Figure 3 A structural schematic diagram of a digital human generation device is shown, and the device comprises: An information acquisition module 31 acquires appearance information, audio information, emotion information and posture information of a to-be-generated digital human; A digital human generation module 32 inputs the appearance information, audio information, emotion information and posture information of the to-be-generated digital human into a digital human generation model to obtain a digital human generation result output by the digital human generation model; wherein the digital human generation model is trained based on a digital human training sample and a digital human video label corresponding to the digital human training sample; the digital human generation model is used for respectively extracting features of the input appearance information, audio information, emotion information and posture information of the to-be-generated digital human, and fusing the extracted features, so as to generate a corresponding digital human video based on the fused features.

[0048] In the embodiment, the information acquisition module 31 comprises: an information acquisition unit, which acquires appearance information, audio information, emotion labels, and posture labels of a digital person to be generated; an emotion extraction unit, which extracts corresponding emotion resources from a preset emotion database according to the emotion labels to obtain corresponding emotion information; wherein the preset emotion database is constructed in advance based on various emotion resources corresponding to different emotion labels; and a posture extraction unit, which extracts corresponding posture information from a preset posture database according to the posture labels; wherein the preset posture database is constructed in advance based on posture information corresponding to different posture labels.

[0049] In addition, the digital person generation model comprises a feature extraction layer, a speech generation layer, and a digital person generation layer, and the digital person generation module 32 comprises: a feature acquisition unit, which, when the type of the audio information is speech, inputs the audio information and the appearance information, emotion information, and posture information into the feature extraction layer for feature extraction respectively to obtain corresponding audio features, appearance reference features, emotion features, and posture features; when the type of the audio information is text, inputs the audio information into the speech generation layer for speech generation to obtain corresponding audio features, and inputs the appearance information, emotion information, and posture information into the feature extraction layer for feature extraction respectively to obtain corresponding appearance reference features, emotion features, and posture features; and a digital person generation unit, which inputs the appearance reference features, emotion features, posture features, and audio features into the digital person generation layer for feature fusion, and generates a digital person based on the fused features to obtain a digital person generation result output by the digital person generation layer.

[0050] Specifically, the digital person generation layer comprises a digital person generation unit and at least two feature fusion units arranged in sequence. The digital person generation unit is configured to: input the appearance reference features, emotion features, posture features, and audio features into each feature fusion unit respectively, and input the output result of a preceding feature fusion unit into a subsequent feature fusion unit, and input the output result of a tail feature fusion unit into the digital person generation unit; wherein, for a head feature fusion unit, the appearance reference features, emotion features, posture features, and audio features are fused according to the input to obtain corresponding fused features, and the fused features are input as the output result into a subsequent feature fusion unit; for a non-head feature fusion unit, the appearance reference features, emotion features, posture features, and audio features and the output result of a preceding feature fusion unit are fused according to the input to obtain corresponding fused features, and the fused features are input as the output result into a subsequent feature fusion unit or the digital person generation unit; and for the digital person generation unit, the fused features output by the tail feature fusion unit are used to generate a digital person video to obtain a digital person generation result.

[0051] Further, the feature fusion unit is configured to: fuse the input appearance reference feature, the emotion feature, the pose feature, and the audio feature, or fuse the input appearance reference feature, the emotion feature, the pose feature, the audio feature, and an output result of a previous feature fusion unit, and convert the fused features through an embedding layer to obtain an embedding vector; add position encoding to the embedding vector to obtain an encoded vector; obtain a self-attention output by using a self-attention mechanism according to the encoded vector; perform residual connection on the self-attention layer output and the encoded vector, and perform layer normalization on a result of the residual connection to obtain a layer normalization result; perform nonlinear transformation on the layer normalization result by using a feedforward neural network, and perform residual connection and layer normalization on a result of the nonlinear transformation and the layer normalization result to obtain a corresponding feature representation.

[0052] In an optional embodiment, the feature fusion unit is further configured to: concatenate the input appearance reference feature, the emotion feature, the pose feature, and the audio feature to obtain a corresponding fused feature.

[0053] In addition, the feature fusion unit is further configured to: fuse the input appearance reference feature, the emotion feature, the pose feature, and the audio feature with corresponding modal features in an output result of a previous feature fusion unit according to corresponding modalities, to obtain fused features of the corresponding modalities; wherein the modalities are used to represent types of the corresponding features; and concatenate the fused features of the modalities to obtain a corresponding fused feature.

[0054] In an optional embodiment, the digital human generation module 22 further includes a multi-layer perception unit, which, after obtaining the fused features output by the tail feature fusion unit, converts the fused features into an input format required by the digital human generation unit by using a multi-layer perceptron (MLP) to adapt to the input requirements of the digital human generation unit.

[0055] In an optional embodiment, the digital human generation unit is configured to: perform video frame conversion on the fused features output by the tail feature fusion unit according to the tail arrangement to obtain a plurality of digital human video frames; and concatenate the plurality of digital human video frames in time sequence to obtain a digital human generation result.

[0056] In an optional embodiment, the apparatus further comprises: a training data acquisition module, before inputting the digital human information to be generated into the digital human generation model, acquiring a digital human training sample and a digital human video label corresponding to the digital human training sample, the digital human training sample comprising audio text training information, appearance image training information, emotion training information and posture training information, and the digital human video label comprising a digital human video sample and a digital human audio sample; a feature training module, when the type of the audio text training information is speech, inputting the audio text training information and the appearance image training information, the emotion training information and the posture training information into a feature extraction layer for feature extraction respectively to obtain appearance sample features, audio training features, emotion training features and posture training features output by the feature extraction layer; when the type of the audio text training information is text, inputting the audio text training information into a speech generation layer to predict a first position representation of a corresponding audio in an embedding space based on the audio text training information to obtain an audio training feature, and predict a second position representation of the corresponding audio in the embedding space based on the audio text training information, and inputting the appearance image training information, the emotion training information and the posture training information into the feature extraction layer for feature extraction respectively to obtain the appearance sample features, the audio training features, the emotion training features and the posture training features output by the feature extraction layer; wherein the embedding space is used to represent a high-dimensional vector space that maps different types of data into continuous vector representations; a first loss function construction module, according to the first position representation and the second position representation, combining an optimization objective to obtain a target loss function; a video training module, inputting the audio training features, the image sample features, the emotion training features and the posture training features into the digital human generation layer to obtain a digital human generation video output by the digital human generation layer; a second function construction module, constructing a video loss function according to the digital human generation video and the digital human video label corresponding to the digital human training sample; a model determination module, according to the target loss function and the video loss function, obtaining a total loss function, and based on the total loss function being minimum, obtaining a corresponding trained digital human generation model.

[0057] Specifically, the feature training module is configured to: based on the audio text training information, starting from a first random noise distribution, gradually convert noise into audio along an initialized velocity vector field to obtain a first position representation of a predicted audio corresponding to the audio text training information in the embedding space; based on the audio text training information, starting from a second random noise distribution, gradually convert noise into audio along an initialized velocity vector field to obtain a second position representation of a predicted audio corresponding to the audio text training information in the embedding space; wherein the second random noise distribution is different from the first random noise distribution.

[0058] In summary, the embodiment of the present application obtains appearance, audio, emotion and posture information of the digital person to be generated by the information acquisition module, provides a comprehensive data basis for subsequent digital person generation, ensures that the generated digital person is not only an appearance model, but also a complete whole with voice, emotion and action, and allows users to customize the appearance, voice, emotion and posture of the digital person according to the needs, so that the generated digital person is more personalized and meets the needs of a specific scene, and further utilizes the digital person generation model by the digital person generation module, extracts the features in the input information, and fuses the extracted features of different modalities, so as to endow the generated digital person with rich emotional expression and diversified posture, so that the generated digital person is more coordinated and natural, and a large amount of time and computing resources are saved.

[0059] Figure 4 An example of a schematic diagram of the physical structure of an electronic device is shown as Figure 4 The electronic device can include a processor 410, a communications interface 420, a memory 430 and a communications bus 440, wherein the processor 410, the communications interface 420 and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke the logical instructions in the memory 430 to execute the digital person generation method, which includes: obtaining appearance information, audio information, emotion information and posture information of a digital person to be generated; inputting the appearance information, audio information, emotion information and posture information of the digital person to be generated into a digital person generation model to obtain a digital person generation result output by the digital person generation model; wherein the digital person generation model is trained based on a digital person training sample and a digital person video label corresponding to the digital person training sample; the digital person generation model is used for respectively extracting features of the input appearance information, audio information, emotion information and posture information of the digital person to be generated, and fusing the extracted features, so as to generate a corresponding digital person video based on the fused features.

[0060] Further, the logic instructions in the memory 430 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0061] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the digital person generation method provided by the above-mentioned methods. The method comprises: obtaining appearance information, audio information, emotion information and posture information of a digital person to be generated; inputting the appearance information, audio information, emotion information and posture information of the digital person to be generated into a digital person generation model to obtain a digital person generation result output by the digital person generation model; wherein the digital person generation model is trained based on a digital person training sample and a digital person video label corresponding to the digital person training sample; and the digital person generation model is used for respectively extracting features of the input appearance information, audio information, emotion information and posture information of the digital person to be generated, and performing feature fusion on the extracted features to generate a corresponding digital person video based on the fused features.

[0062] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the digital person generation method provided by the above-mentioned methods. The method comprises: obtaining appearance information, audio information, emotion information and posture information of a digital person to be generated; inputting the appearance information, audio information, emotion information and posture information of the digital person to be generated into a digital person generation model to obtain a digital person generation result output by the digital person generation model; wherein the digital person generation model is trained based on a digital person training sample and a digital person video label corresponding to the digital person training sample; and the digital person generation model is used for respectively extracting features of the input appearance information, audio information, emotion information and posture information of the digital person to be generated, and performing feature fusion on the extracted features to generate a corresponding digital person video based on the fused features.

[0063] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0064] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0065] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a digital human, characterized in that: include: Obtaining appearance information, audio information, emotion information, and posture information of the digital human to be generated; Inputting the appearance information, audio information, emotion information and posture information of the digital human to be generated into a digital human generation model, and obtaining a digital human generation result output by the digital human generation model; wherein the digital human generation model is trained based on digital human training samples and digital human video labels corresponding to the digital human training samples; The digital human generation model is used to extract features of the input appearance information, audio information, emotional information and posture information of the digital human to be generated, and to fuse the extracted features to generate a corresponding digital human video based on the fused features.

2. The digital human generation method according to claim 1, characterized in that: The digital human generation model includes a feature extraction layer, a speech generation layer, and a digital human generation layer. The appearance information, audio information, emotion information, and posture information of the digital human to be generated are input into the digital human generation model, and the digital human generation result output by the digital human generation model is obtained, including: When it is determined that the type of the audio information is speech, the audio information and the appearance information, the emotion information, and the posture information are input into the feature extraction layer for feature extraction to obtain corresponding audio features, appearance reference features, emotion features, and posture features; When it is determined that the type of the audio information is text, the audio information is input into the speech generation layer for speech generation to obtain corresponding audio features, and the appearance information, the emotion information, and the posture information are input into the feature extraction layer for feature extraction to obtain corresponding appearance reference features, emotion features, and posture features; The appearance reference features, the emotion features, the posture features and the audio features are input into the digital human generation layer for feature fusion, and a digital human is generated based on the fused features to obtain a digital human generation result output by the digital human generation layer.

3. The method for generating a digital human according to claim 2, wherein: The digital human generation layer includes a digital human generation unit and at least two sequentially arranged feature fusion units. The appearance reference feature, the emotion feature, the posture feature, and the audio feature are input into the digital human generation layer for feature fusion. The digital human is generated based on the fused features. The digital human generation result output by the digital human generation layer includes: The appearance reference feature, the emotion feature, the posture feature and the audio feature are input into each of the feature fusion units respectively, and the output result of the first feature fusion unit is input into the second feature fusion unit, and the output result of the tail feature fusion unit is input into the digital human generation unit; wherein, For the head feature fusion unit, feature fusion is performed based on the input appearance reference features, emotional features, posture features and audio features to obtain the corresponding fusion features, and the output result is input into the subsequent feature fusion unit; For the non-head feature fusion unit, feature fusion is performed based on the input appearance reference features, emotion features, posture features, audio features and the output results of the previous feature fusion unit to obtain the corresponding fused features, and the output results are input into the subsequent feature fusion unit or the digital human generation unit; The digital human generation unit generates a digital human video based on the fusion features output by the tail feature fusion unit to obtain a digital human generation result.

4. The method for generating a digital human according to claim 3, wherein: Perform feature fusion based on the input appearance reference features, emotion features, posture features, and audio features, or perform feature fusion based on the input appearance reference features, emotion features, posture features, audio features and the output results of a previous feature fusion unit to obtain corresponding fused features, including: Performing feature fusion on the input appearance reference features, emotion features, posture features, and audio features, or performing feature fusion on the input appearance reference features, emotion features, posture features, audio features, and the output result of a previous feature fusion unit, and transforming the fused features through an embedding layer to obtain an embedding vector; Adding a positional encoding to the embedding vector to obtain an encoding vector; According to the encoding vector, a self-attention mechanism is used to obtain a self-attention output; Performing a residual connection between the self-attention layer output and the encoding vector, and performing layer normalization on the residual connection result to obtain a layer normalization result; According to the layer normalization result, a feedforward neural network is used to perform nonlinear transformation, and the nonlinear change result and the layer normalization result are residually connected and layer normalized to obtain corresponding feature representation.

5. The method for generating a digital human according to claim 3, wherein: The input appearance reference features, emotion features, posture features and audio features are fused to obtain the corresponding fusion features, including: The input appearance reference features, emotion features, posture features and audio features are spliced ​​to obtain the corresponding fusion features; The input appearance reference features, emotion features, posture features, audio features and the output results of the previous feature fusion unit are fused to obtain the corresponding fused features, including: fusing the input appearance reference features, emotion features, posture features, and audio features with the features of the corresponding modalities in the output results of the previous feature fusion unit according to the corresponding modalities to obtain fused features of the corresponding modalities; wherein the modalities are used to characterize the types of the corresponding features; The fusion features of each modality are spliced ​​together to obtain corresponding fusion features.

6. The method for generating a digital human according to claim 2, wherein: Before inputting the information of the digital human to be generated into the digital human generation model, the process includes: Obtaining digital human training samples and digital human video tags corresponding to the digital human training samples, wherein the digital human training samples include audio text training information, appearance image training information, emotion training information, and posture training information, and the digital human video tags include digital human video samples and digital human audio samples; When it is determined that the type of the audio text training information is speech, the audio text training information and the appearance image training information, the emotion training information, and the posture training information are input into the feature extraction layer for feature extraction, respectively, to obtain appearance sample features, audio training features, emotion training features, and posture training features output by the feature extraction layer; When it is determined that the type of the audio text training information is text, the audio text training information is input into the speech generation layer to predict the first position representation of the corresponding audio in the embedding space based on the audio text training information to obtain audio training features, and the second position representation of the corresponding audio text in the embedding space is predicted based on the audio text training information, and the appearance image training information, the emotion training information and the posture training information are input into the feature extraction layer for feature extraction respectively, to obtain the appearance sample features, audio training features, emotion training features and posture training features output by the feature extraction layer; wherein the embedding space is used to represent a high-dimensional vector space that maps different types of data into continuous vector representations; Obtaining a target loss function based on the first position representation and the second position representation in combination with an optimization objective; Inputting the audio training features, the image sample features, the emotion training features and the posture training features into the digital human generation layer to obtain a digital human generated video output by the digital human generation layer; Constructing a video loss function according to the digital human generated video and the digital human video labels corresponding to the digital human training samples; According to the target loss function and the video loss function, a total loss function is obtained, and based on minimizing the total loss function, a corresponding trained digital human generation model is obtained.

7. The method for generating a digital human according to claim 1, wherein: Obtain the appearance information, audio information, emotion information, and posture information of the digital human to be generated, including: Obtain the appearance information, audio information, emotion label and posture label of the digital human to be generated; According to the emotion tag, corresponding emotion resources are extracted from a preset emotion database to obtain corresponding emotion information; wherein the preset emotion database is previously constructed based on various emotion resources corresponding to different emotion tags; According to the posture tag, corresponding posture information is extracted from a preset posture database; wherein the preset posture database is previously constructed based on posture information corresponding to different posture tags.

8. A digital human generation device, characterized in that: include: An information acquisition module, which acquires the appearance information, audio information, emotion information and posture information of the digital human to be generated; A digital human generation module inputs the appearance information of the digital human to be generated, the audio information, the emotion information, and the posture information into a digital human generation model, and obtains a digital human generation result output by the digital human generation model; wherein the digital human generation model is trained based on digital human training samples and digital human video labels corresponding to the digital human training samples; The digital human generation model is used to extract features of the input appearance information, audio information, emotional information and posture information of the digital human to be generated, and to fuse the extracted features to generate a corresponding digital human video based on the fused features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the digital human generation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the digital human generation method according to any one of claims 1 to 7 is implemented.

Images