A voice-driven 2D digital human video generation method and readable storage medium

By obtaining the audio data of the target voice, a 3D posture sequence is generated and the user image is generated to generate head and body action videos. Using intelligent fusion technology, the problem of insufficient realism and naturalness of digital human videos in the existing technology is solved, and voice and The precise matching of digital human movements and video fluency provide efficient and convenient solutions.

CN118250529BActive Publication Date: 2024-11-15DMAI (GUANGZHOU) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410658653.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-27
Publication Date
2024-11-15
Estimated Expiration
2044-05-27

AI Technical Summary

Technical Problem

The prior art is difficult to generate voice-driven 2D digital human videos with a high sense of reality and nature, especially in the challenge of accurately matching target voice with digital human head and body movements while maintaining the smoothness and coherence of the video. .

Method used

By obtaining the audio data of the target voice, a corresponding 3D pose sequence is generated, and a head and body action video is generated in combination with user images. The intelligent fusion technology is seamlessly combined to form a smooth 2D digital human video. The specific steps include audio encoding, pose generation, light processing and volume rendering, and finally the fusion device generates the final 2D digital human video.

Benefits of technology

It realizes the accurate matching of voice and digital human movements, improves the realism and nature of digital human videos, and provides efficient and convenient solutions for related applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118250529B_ABST
    Figure CN118250529B_ABST
Patent Text Reader

Abstract

The present invention discloses a voice-driven 2D digital human video generation method and a readable storage medium, comprising: first obtaining a target voice, and obtaining a corresponding 3D posture sequence by audio data processing. Then, combining the target voice and the 3D posture sequence to generate an accurate head movement video. At the same time, using the user image and the 3D posture sequence to generate a body movement video. Finally, through intelligent fusion technology, the head and body movement videos are seamlessly combined to form a smooth 2D digital human video. Such a design achieves accurate matching of voice and digital human movements, improves the realism and naturalness of the digital human video, and provides an efficient and convenient solution for related applications.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a voice-driven 2D digital human video generation method and a readable storage medium. Background Art

[0002] In the digital age, voice-driven 2D digital human video generation technology has broad application prospects in virtual role performance, game interaction, online education and other fields. However, existing technologies still face challenges in generating digital human videos with high realism and naturalness. In particular, how to accurately match the target voice with the head and body movements of the digital human while maintaining the smoothness and coherence of the video has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The object of the present invention is to provide a voice-driven 2D digital human video generation method and a readable storage medium.

[0004] In a first aspect, an embodiment of the present invention provides a voice-driven 2D digital human video generation method, comprising:

[0005] Get the target voice;

[0006] According to the audio data of the target speech, obtaining a 3D posture sequence corresponding to the audio data;

[0007] According to the target speech and the 3D posture sequence, a head movement video corresponding to the target speech is obtained;

[0008] Obtaining a body motion video according to the user image and the 3D posture sequence;

[0009] According to the head action video and the body action video, a 2D digital human video is obtained by fusion.

[0010] In a possible implementation, acquiring, according to the audio data of the target speech, a 3D gesture sequence corresponding to the audio data includes:

[0011] Encoding the audio data through an audio encoder to obtain an audio feature vector sequence corresponding to the audio data;

[0012] The audio feature vector sequence is input into a gesture generator based on a plurality of gated recurrent units to obtain the 3D gesture sequence.

[0013] In a possible implementation, the audio encoder includes a primary-order coding unit and a high-order coding unit, the high-order coding unit is constructed by a multi-layer linear network layer and a nonlinear activation function, and the audio data is encoded by the audio encoder to obtain an audio feature vector sequence corresponding to the audio data, including:

[0014] Encoding the audio data by the primary encoding unit to obtain a plurality of initial audio feature vectors, each of which includes the logarithmic amplitude and energy of the Mel spectrum of the current frame;

[0015] Performing feature conversion on the multiple initial audio feature vectors by the high-order coding unit to obtain the audio feature vector sequence, wherein the audio feature vector sequence includes a high-order audio feature vector corresponding to each frame of audio data;

[0016] The step of inputting the audio feature vector sequence into a gesture generator based on a plurality of gated recurrent units to obtain the 3D gesture sequence comprises:

[0017] Inputting the high-order audio feature vector corresponding to the i-th frame and the posture data corresponding to the i-1-th frame into the pre-trained posture generator to obtain the posture data corresponding to the i-th frame, where i is a positive integer greater than 1, and when i=1, the posture data corresponding to the i-1-th frame is randomly initialized posture data, and the posture data includes the absolute position of the human skeleton key points and the rotation value of the human skeleton key points of the current frame;

[0018] The acquired multiple posture data are integrated to obtain the 3D posture sequence, and the 3D posture sequence corresponds one-to-one to the elements of the audio feature vector sequence.

[0019] In a possible implementation, obtaining a head movement video corresponding to the target voice according to the target voice and the 3D gesture sequence includes:

[0020] Encoding the audio data through an audio encoder to obtain an audio feature vector sequence corresponding to the audio data;

[0021] Extracting a 3D head posture sequence from the 3D posture sequence, and inputting the 3D head posture sequence into a light processor to obtain 3D position information corresponding to the 3D head posture sequence, wherein the 3D position information includes the 3D position of the sampling point corresponding to the 3D head posture of each frame in the 3D head posture sequence in the light direction;

[0022] Inputting the 3D position information into a position encoder to obtain a position encoding feature sequence;

[0023] Inputting the position coding feature sequence, the audio feature vector sequence and the light direction of the sampling point into a pre-trained neural radiation field to obtain the color and density information of each sampling point;

[0024] Volume rendering is performed on the color and density information of each sampling point to obtain the head movement video, where the head movement video includes facial expressions and lip movements.

[0025] In one possible implementation, the neural radiation field is expressed by the formula: As the loss function training, refers to the frames in the training sample, refers to the frames generated by the neural radiation field, Refers to the weight coefficient.

[0026] In a possible implementation, obtaining a body motion video according to the user image and the 3D posture sequence includes:

[0027] The user image and the 3D posture sequence are input into a preset diffusion model to obtain the body movement video.

[0028] In a possible implementation, the fusing the head action video and the body action video to obtain a 2D digital human video includes:

[0029] Dividing the body action video into a pending head action video and a pending body action video based on a preset portrait segmentation algorithm;

[0030] The head action video and the pending body action video are input into a fusion device trained by a discriminator, so that the head action video replaces the pending head action video to obtain the 2D digital human video.

[0031] In a possible implementation, the loss function of the discriminator is:

[0032] ;

[0033] in, represents the true sample, represents the output of the discriminator, represents the output of the fuser, represents the head action video and pending body action video input to the fuser, The larger it is, the more accurate the discriminator's judgment is;

[0034] The loss function of the fuser is:

[0035] ;

[0036] in, represents the output of the discriminator, represents the output of the fuser, represents the head action video and pending body action video input to the fuser, represents the hyperparameter, Represents the head region of the head action video, Represents the head area of ​​the video generated by the fuser, The smaller the value, the more accurate the fuser output is.

[0037] In a second aspect, an embodiment of the present invention provides a voice-driven 2D digital human video generation device, comprising:

[0038] An acquisition module, used to acquire target speech;

[0039] A processing module, configured to obtain a 3D gesture sequence corresponding to the audio data of the target voice according to the audio data of the target voice; obtain a head movement video corresponding to the target voice according to the target voice and the 3D gesture sequence; and obtain a body movement video according to the user image and the 3D gesture sequence;

[0040] A generation module is used to fuse the head action video and the body action video to obtain a 2D digital human video.

[0041] In a third aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located controls the computer device to execute a voice-driven 2D digital human video generation method in at least one possible implementation of the first aspect.

[0042] Compared with the prior art, the beneficial effects provided by the present invention include: using a voice-driven 2D digital human video generation method and a readable storage medium disclosed by the present invention, including: first obtaining a target voice, and obtaining a corresponding 3D posture sequence by audio data processing. Then, combining the target voice and the 3D posture sequence to generate an accurate head movement video. At the same time, using the user image and the 3D posture sequence to generate a body movement video. Finally, through intelligent fusion technology, the head and body movement videos are seamlessly combined to form a smooth 2D digital human video. Such a design achieves accurate matching of voice and digital human movements, improves the realism and naturalness of digital human videos, and provides an efficient and convenient solution for related applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic diagram of the steps of a voice-driven 2D digital human video generation method provided in an embodiment of the present invention;

[0045] Figure 2 A schematic block diagram of the structure of a voice-driven 2D digital human video generation device provided in an embodiment of the present invention;

[0046] Figure 3 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0049] In order to solve the technical problems in the aforementioned background technology, Figure 1 This is a flow chart of a voice-driven 2D digital human video generation method provided in an embodiment of the present disclosure. The voice-driven 2D digital human video generation method is introduced in detail below.

[0050] Step S201, obtaining target speech;

[0051] Step S202, acquiring a 3D gesture sequence corresponding to the audio data according to the audio data of the target speech;

[0052] Step S203, obtaining a head movement video corresponding to the target voice according to the target voice and the 3D posture sequence;

[0053] Step S204, obtaining a body movement video according to the user image and the 3D posture sequence;

[0054] Step S205: fusing the head action video and the body action video to obtain a 2D digital human video.

[0055] In an embodiment of the present invention, exemplarily, the server receives an audio file uploaded by a user, and the audio file contains the target voice that the user wants the 2D digital human to express. For example, the user uploads an audio of himself reading a poem, and this audio is the target voice that the server will process next. The server uses the built-in speech recognition and posture estimation module to analyze the target voice. The server first identifies the syllables and intonations in the voice, and then predicts the corresponding 3D posture sequence based on these voice features. This 3D posture sequence describes the head and body movements that the digital human should take when expressing this voice. For example, when the tone of "happy" is recognized, the server can generate an expression posture of looking up and smiling; when the tone of "sad" is recognized, a posture of bowing the head and thinking can be generated. The server renders and generates a head movement video synchronized with the target voice based on the 3D posture sequence generated in the previous step and the preset digital human head model. In this process, the server ensures that the digital human's mouth shape, facial expression and head movement are consistent with the content of the target voice. For example, when the target voice says "hello", the digital human's mouth shape will accurately simulate the mouth movement of the word "hello". The user provides the server with a photo or full-body image of himself as the body template of the digital human. The server uses this template and the 3D posture sequence to generate a body movement video of the digital human. In this process, the server adjusts the posture of the digital human body template according to the body movement data in the 3D posture sequence to match the emotion and rhythm expressed by the target voice. For example, if the target voice is an impassioned speech, the generated body movement video will show the impassioned posture of the digital human waving his arms and standing upright. Finally, the server fuses the head movement video and the body movement video to generate a complete 2D digital human video. In this process, the server ensures that the connection between the head and body movements is natural and smooth, without any disconnection or incoordination. The generated 2D digital human video not only retains the original emotion and rhythm of the user's voice, but also enhances the visual expression effect through the vivid performance of the digital human. For example, the final generated 2D digital human video may be a scene of a digital human giving a speech on a podium, with its head expression and body movements highly consistent with the original voice content and vivid and lifelike.

[0056] In the embodiment of the present invention, the aforementioned step S202 may be implemented through the following example execution.

[0057] Encoding the audio data through an audio encoder to obtain an audio feature vector sequence corresponding to the audio data;

[0058] The audio feature vector sequence is input into a gesture generator based on a plurality of gated recurrent units to obtain the 3D gesture sequence.

[0059] In an embodiment of the present invention, exemplarily, after receiving the target voice uploaded by the user, the server will first use the built-in audio encoder to process the audio data. The function of the audio encoder is to convert the original audio signal into a mathematical representation that is easier to process, namely, an audio feature vector sequence. This process is similar to converting an article into a series of keywords for subsequent analysis and understanding. For example, the target voice received by the server is an audio about weather forecast. The audio encoder will analyze the audio segment by segment, extract the key sound features, such as pitch, sound intensity, timbre, rhythm and intonation of the voice, and convert these features into a series of audio feature vectors. These vectors capture the essential information in the audio and provide accurate input for subsequent gesture generation. Next, the server will input the audio feature vector sequence obtained in the previous step into a gesture generator composed of multiple gated recurrent units (GRU). This gesture generator is a deep learning model that has learned how to predict the corresponding 3D gesture sequence from the audio feature vector sequence after a large amount of training data. The gated recurrent unit is a variant of the recurrent neural network, which is particularly suitable for processing sequence data, such as audio, text, etc. In the pose generator, multiple gated recurrent units are connected in series to form a deep network structure. When the audio feature vector sequence is input into this network, each gated recurrent unit will calculate the state and output of the next unit based on the current input and the output of the previous unit. In this way, the pose generator can gradually generate a corresponding 3D pose sequence based on the audio feature vector sequence. Taking the audio of the weather forecast as an example, the pose generator will predict the head and body poses that the digital human should take when expressing this sentence based on the information in the audio feature vector sequence, such as the speech content of "Today's weather is fine". These poses are encoded into a 3D pose sequence, which contains information such as the position, rotation angle and movement speed of the digital human at different time points. This 3D pose sequence provides an important data basis for the subsequent generation of head action videos and body action videos.

[0060] In an embodiment of the present invention, the audio encoder includes a primary coding unit and a high-order coding unit, and the high-order coding unit is constructed by a multi-layer linear network layer and a non-linear activation function. The aforementioned step of encoding the audio data through the audio encoder to obtain an audio feature vector sequence corresponding to the audio data can be implemented through the following example.

[0061] Encoding the audio data by the primary encoding unit to obtain a plurality of initial audio feature vectors, each of which includes the logarithmic amplitude and energy of the Mel spectrum of the current frame;

[0062] Performing feature conversion on the multiple initial audio feature vectors by the high-order coding unit to obtain the audio feature vector sequence, wherein the audio feature vector sequence includes a high-order audio feature vector corresponding to each frame of audio data;

[0063] The step of inputting the audio feature vector sequence into a gesture generator based on a plurality of gated recurrent units to obtain the 3D gesture sequence comprises:

[0064] Inputting the high-order audio feature vector corresponding to the i-th frame and the posture data corresponding to the i-1-th frame into the pre-trained posture generator to obtain the posture data corresponding to the i-th frame, where i is a positive integer greater than 1, and when i=1, the posture data corresponding to the i-1-th frame is randomly initialized posture data, and the posture data includes the absolute position of the human skeleton key points and the rotation value of the human skeleton key points of the current frame;

[0065] The acquired multiple posture data are integrated to obtain the 3D posture sequence, and the 3D posture sequence corresponds one-to-one to the elements of the audio feature vector sequence.

[0066] In an embodiment of the present invention, exemplarily, the server first uses a primary coding unit to process the uploaded audio data. The primary coding unit is mainly responsible for extracting basic acoustic features from the audio. In this process, the server will frame the audio and calculate the logarithmic amplitude and energy of the Mel spectrum of each frame. These features capture the spectral structure and energy distribution of the audio, providing a basis for the subsequent encoding process. For example, assuming that the uploaded audio is a singing performance, the primary coding unit will analyze the singer's voice frame by frame and extract the logarithmic amplitude of the Mel spectrum of each frame. These amplitudes reflect the spectral characteristics of the singer's pronunciation, such as pitch and timbre. At the same time, the energy feature reflects the intensity and dynamic changes of the singer's pronunciation. Next, the server passes the multiple initial audio feature vectors extracted by the primary coding unit to the high-order coding unit for further feature conversion. The high-order coding unit is constructed by a multi-layer linear network layer and a nonlinear activation function, which can perform deep abstraction and representation learning on the initial features. Continuing with the singing performance as an example, the high-order coding unit will perform complex transformations and combinations on the initial audio feature vectors to capture deeper information in the audio, such as the singer's pronunciation habits, emotional expression, etc. After processing by the high-order encoding unit, the server obtains a series of high-order audio feature vectors, which are more compact and expressive, providing a powerful input for subsequent pose generation. The server now inputs the high-order audio feature vector sequence into a pose generator based on multiple gated recurrent units (GRUs). This pose generator has been pre-trained to predict the corresponding 3D pose sequence based on the input audio feature vector sequence. In specific operations, the server processes the audio data frame by frame. For the high-order audio feature vector corresponding to the audio data of the i-th frame, the server inputs it into the pose generator together with the pose data of the previous frame (i.e., the i-1th frame). When processing the first frame (i.e., i=1), since there is no pose data of the previous frame for reference, the server uses randomly initialized pose data as input. The GRU network inside the pose generator combines the audio features of the current frame with the pose data of the previous frame to predict the pose data of the current frame. These data include detailed information such as the absolute position and rotation values ​​of the key points of the human skeleton, which together constitute the 3D pose representation of the current frame. As the processing progresses frame by frame, the server accumulates a complete 3D posture sequence. This sequence corresponds one-to-one with the original audio feature vector sequence in time, ensuring the synchronization between audio and posture. For example, in a singing performance, as the singer's voice rises and falls, the posture generator predicts the body movements and facial expressions that match it. These predicted posture data are not only closely linked to the audio content, but can also present realistic performance effects visually.

[0067] The aforementioned step S203 can be implemented through the following example execution.

[0068] Encoding the audio data through an audio encoder to obtain an audio feature vector sequence corresponding to the audio data;

[0069] Extracting a 3D head posture sequence from the 3D posture sequence, and inputting the 3D head posture sequence into a light processor to obtain 3D position information corresponding to the 3D head posture sequence, wherein the 3D position information includes the 3D position of the sampling point corresponding to the 3D head posture of each frame in the 3D head posture sequence in the light direction;

[0070] Inputting the 3D position information into a position encoder to obtain a position encoding feature sequence;

[0071] Inputting the position coding feature sequence, the audio feature vector sequence and the light direction of the sampling point into a pre-trained neural radiation field to obtain the color and density information of each sampling point;

[0072] Volume rendering is performed on the color and density information of each sampling point to obtain the head movement video, where the head movement video includes facial expressions and lip movements.

[0073] In an embodiment of the present invention, exemplarily, the server first uses an audio encoder to encode the uploaded target voice. This process is similar to the previous description. The audio encoder converts the original audio data into an audio feature vector sequence. These feature vectors capture the key sound characteristics in the audio and provide important input information for the subsequent generation of head action videos. For example, assuming that the target voice is an audio segment telling a story, the audio encoder will analyze the audio segment by segment, extract key features such as the narrator's speech speed, intonation, and emotion, and convert these features into an audio feature vector sequence. Next, the server extracts a 3D head posture sequence related to the head action from the previously generated 3D posture sequence. This sequence describes the movement trajectory and posture changes of the head of the digital person when expressing the target voice. Subsequently, the server inputs the 3D head posture sequence into the light processor. The task of the light processor is to calculate the 3D position information of the light direction of the sampling point corresponding to each 3D head posture. This information is crucial for the subsequent generation of head action videos with real light and shadow effects. Taking the audio of telling a story as an example, when the narrator describes a scene, his head can turn or tilt as the story progresses. The light processor calculates the corresponding light direction based on these head movements to ensure that the light and shadow changes in the generated head movement video match the real scene. The server then inputs the 3D position information output by the light processor into the position encoder. The role of the position encoder is to convert these 3D position information into a form that is easier for neural networks to process, namely, a position encoding feature sequence. This feature sequence captures the spatial position relationship of the head movement and provides a key input for the subsequent neural radiance field processing. Now, the server inputs the position encoding feature sequence, the audio feature vector sequence, and the light direction of the sampling point into the pre-trained neural radiance field. The neural radiance field is a deep learning model that can predict the color and density information of each sampling point based on the input feature vector and light direction. This information is a key element in generating high-quality head movement videos. Taking the storytelling audio as an example, when the narrator describes a clear sky, the neural radiance field can predict a bright blue as the background color and assign corresponding color and density values ​​to each sampling point based on the head movement and light direction. Finally, the server uses volume rendering technology to render the color and density information output by the neural radiance field to generate the final head movement video. This video not only includes the digital human's head movement trajectory and posture changes, but also presents realistic expressions and lip movement effects. Taking the storytelling audio as an example, after volume rendering processing, the server will generate a high-quality head movement video, in which the digital human's head turns and tilts as the story progresses, accompanied by vivid expressions and accurate lip movement effects, bringing an immersive viewing experience to the audience.

[0074] In this embodiment of the present invention, the neural radiation field is obtained through a loss function: Trained, where refers to the frames in the training sample, refers to the frames generated by the neural radiation field, Refers to the weight coefficient.

[0075] In the embodiment of the present invention, the weight coefficient is defined as follows: the value of the area belonging to the background is 0, the value of the area belonging to the head is 1, and the value of the area belonging to the mouth is 2. In summary, the head action generation module inputs voice and 3D posture, and outputs head action video.

[0076] In the embodiment of the present invention, the aforementioned step S204 may be implemented through the following example execution.

[0077] The user image and the 3D posture sequence are input into a preset diffusion model to obtain the body movement video.

[0078] In an embodiment of the present invention, exemplarily, after receiving the user image and 3D posture sequence uploaded by the user, the server will input these data into a preset diffusion model. This diffusion model is a deep learning model. After learning a large amount of training data, it has mastered how to generate a coherent and natural body action video from a static user image and a dynamic 3D posture sequence. For example, suppose that a user uploads a full-body photo of himself as a user image, and provides a 3D posture sequence, which describes the motion trajectory of a dance action. The server will first input this photo and the 3D posture sequence into the diffusion model. Inside the diffusion model, the model will first analyze the user image and extract important elements such as the user's physical features, clothing style, and background information. Then, the model will combine the dynamic information in the 3D posture sequence, such as the position change and rotation value of the key points of the skeleton, to predict the body shape and motion trajectory of the user when performing the dance action. Next, the diffusion model will use this predicted information to generate each frame of the body action video. In this process, the model will ensure that the generated video frame is visually consistent with the user image, while reflecting the dynamic changes in the 3D posture sequence. To achieve this, the model can use some advanced image processing techniques, such as image fusion, motion blur, and light and shadow rendering, to enhance the coherence and realism of the video. Finally, the server will output a complete body movement video, which shows the whole body movement of the user when performing dance moves. Users can watch this video to check whether the generated dance moves meet their expectations, and make subsequent adjustments and optimizations accordingly. It should be noted that the above process is automatically completed within the server. Users only need to upload the corresponding data and wait for a while to get their customized body movement video. This technology provides users with a new and highly personalized way of video generation, which has broad application prospects and market potential.

[0079] In the embodiment of the present invention, the aforementioned step S205 can be implemented through the following example execution.

[0080] Dividing the body action video into a pending head action video and a pending body action video based on a preset portrait segmentation algorithm;

[0081] The head action video and the pending body action video are input into a fusion device trained by a discriminator, so that the head action video replaces the pending head action video to obtain the 2D digital human video.

[0082] In an embodiment of the present invention, exemplarily, the server first processes the body action video using a preset portrait segmentation algorithm. This portrait segmentation algorithm can intelligently identify the portrait area in the video and separate it from the background. In this process, the server divides the body action video into two parts: a pending head action video and a pending body action video. For example, suppose the body action video is a video of a dance performance, which contains the dancer's whole body movements. Through the portrait segmentation algorithm, the server can accurately identify the dancer's head area and use it as a pending head action video, while the rest of the dancer's body is used as a pending body action video. Next, the server inputs the head action video and the pending body action video into a fusion device trained by a discriminator. This fusion device is specially trained to seamlessly fuse the head action with the body action to generate a coherent 2D digital human video. In this process, the fusion device uses the guidance information provided by the discriminator to determine the matching degree of the head action video with the pending body action video, and intelligently fuses the two videos based on this information. Specifically, the fuser replaces the head area in the head action video with the corresponding area in the pending head action video, while retaining the rest of the body in the pending body action video. Taking a dance performance video as an example, suppose the head action video shows the head movements of a dancer making various expressions, while the pending body action video is the body movements of the dancer performing a dance. Through the processing of the fuser, the server can generate a new 2D digital human video in which the dancer's head movements and body movements are perfectly integrated, presenting a vivid and coherent dance performance effect. Finally, the server outputs the fused 2D digital human video. This video not only retains the body dynamic information in the original body action video, but also integrates the rich expressions and precise lip movement effects in the head action video, providing users with a highly realistic virtual character performance experience. Users can watch this video to appreciate the wonderful performance of the digital human, or share it with others for display and communication.

[0083] In the embodiment of the present invention, the loss function of the discriminator is:

[0084] ;

[0085] in, represents the true sample, represents the output of the discriminator, represents the output of the fuser, represents the head action video and pending body action video input to the fuser, The larger it is, the more accurate the discriminator's judgment is;

[0086] The loss function of the fuser is:

[0087] ;

[0088] in, represents the output of the discriminator, represents the output of the fuser, represents the head action video and pending body action video input to the fuser, represents the hyperparameter, Represents the head region of the head action video, Represents the head area of ​​the video generated by the fuser, The smaller the value, the more accurate the fuser output is.

[0089] In order to more clearly describe the solution provided by the embodiments of the present application, a more complete implementation method is provided below.

[0090] In the embodiment of the present invention, a speech is input and passes through four modules, namely, a 3D posture generation module, a head movement generation module, a body movement generation module, and a fusion module, and finally outputs a 2D digital human video that is consistent with the speech duration and frame rate. The specific implementation steps are:

[0091] Step 1: Get a voice of any length. The voice can be the user's own voice or the voice generated by the Text-To-Speech (TTS) algorithm.

[0092] Step 2: Input the speech in step 1 into the 3D posture generation module, which will generate a rhythmic 3D posture sequence based on the audio, where the 3D posture includes the absolute position of the key points of the human skeleton and the rotation value of the key points of the human skeleton (usually expressed in the form of quaternions).

[0093] Step 3, input the 3D posture sequence obtained in step 2 and the speech obtained in step 1 into the head movement generation module to obtain a head movement video. The head movement video here refers to the video of the part above the neck, including facial expressions and lip movements. It can be understood as a digital human video that is talking and only contains the head, and the lip movements are matched with the speech.

[0094] Step 4 can be performed simultaneously with step 3. The 3D posture sequence obtained in step 2 and the preset user image are input into the body movement generation module to generate a body movement video. The limb movements in the body movement video here are consistent with the movements represented by the 3D posture sequence, but the expressions and lips do not change.

[0095] Step 5: Input the head movement video obtained in step 3 and the body movement video obtained in step 4 into a fusion optimization module to obtain a final 2D digital human video, which includes both body movements and head movements, as well as facial expressions and lip movements.

[0096] In order to implement the above solution, the present invention includes four modules, each of which is independent of each other in terms of function. The following will introduce the relevant contents of each module in detail.

[0097] 1. 3D gesture generation module. The function of this module is to generate a 3D gesture sequence corresponding to the rhythm of the speech. , where the 3D posture includes the absolute position of the key points of the human skeleton and the rotation value of the key points of the human skeleton, and the obtained results are used by subsequent modules. An implementation method of this module is as follows Figure 2 As shown. It is mainly composed of an audio encoder and a gesture generator. The function of the audio encoder is to convert the original audio into a feature vector for easy model understanding. Here the number of audio frames Generally, it depends on the sampling rate of the audio and the number of samples per frame. Given a piece of audio, the audio encoder needs to first obtain the characteristics of the audio itself, which is called the initial feature vector ,in The initial eigenvector It consists of two parts, one is the logarithmic amplitude of the Mel spectrum of the current frame, and the other is the energy of the current frame. The calculation process of these two parts is not repeated here. The next step is to convert the initial feature vector Converted to a more abstract representation called a feature vector , the purpose is to generate a feature vector that is more conducive to the understanding of the subsequent gesture generator. One implementation method of converting the above feature vector can be through a neural network model, which can be composed of multiple linear network layers plus nonlinear activation functions. The final audio encoder will output a feature vector sequence of length n The pose generator is used to generate a 3D pose sequence. Here, it is necessary to ensure that the generated 3D pose sequence is consistent with the input audio in rhythm. One implementation of the pose generator can be a neural network model composed of multiple gated recurrent units (GRU). The reason for using GRU here is that it has good timing and can ensure the consistency between the poses of consecutive frames. The generation of the pose of the i-th frame is consistent with the audio feature vector Related to the pose generated in the i-1th frame, when generating the first frame pose, since there is no previous pose input, one implementation method is to randomly initialize a frame pose as the initial input. The final pose generation module will generate a 3D pose sequence , the number of sequence frames is n. Next, we will introduce how to construct the training samples of the 3D posture generation module. Each training sample consists of a segment of audio and its corresponding 3D posture sequence. The number of frames and frame rate of the audio and posture sequence must be consistent, and the rhythm of the 3D posture sequence must match the rhythm of the audio. Manual intervention is required here to assist in labeling. The L1 loss function is used to calculate the loss between the output during training and the target 3D posture sequence, and the neural network parameters are gradually optimized. In summary, the input of the 3D posture generation module is speech, and the output is a 3D posture sequence. .

[0098] Head action generation module. The function of this module is to output head action videos corresponding to speech. The head action video here refers to the video above the neck, including facial expressions and lip movements. It can be understood as a video of a digital human who is talking and only contains the head, and the lip movements match the speech. The head action generation module can include an audio encoder, a light processor, a position encoder, a neural radiance field, and a volume rendering. The audio encoder here is similar to the audio encoder in the 3D posture generation module. Its main function is to convert the original audio into a feature vector. Here we need to introduce the part of generating videos. Video generation is mainly completed by the light processor, neural radiance fields (NeRF) and volume rendering. Neural radiance fields are a computer vision technology. The trained neural radiance field can construct an implicit mapping of objects in three-dimensional space to two-dimensional images. The camera position and camera view are input, and the neural radiance field outputs the color and density of each point on each ray under the view. Finally, the pixel value of each point on the 2D image is obtained through volume rendering. It is a relatively mature technology in the field of computer vision. The determination of each ray and the sampling of points on the ray are completed by the ray processor. The implementation method of the ray processor refers to the ray tracing algorithm, which is a relatively mature algorithm. The camera position and camera angle are input, and the ray tracing algorithm will output the ray direction corresponding to each pixel on the imaging 2D plane and the 3D position of the sampling point on the ray. The position encoder encodes the 3D position output by the ray processor in order to map it to a high-dimensional space, which is beneficial to the training of the neural radiation field. One implementation method of the position encoder is a multi-layer linear network layer plus a nonlinear activation function. The encoded 3D position of the sampling point and the encoded audio will be merged into a feature vector, which will be input into the neural radiation field together with the ray direction of the current sampling point to obtain color and density information. In the same way, the color and density information of all sampling points can be obtained, and then the pixel value of each point on the 2D image can be obtained through volume rendering. It should be noted that the input 3D pose sequence comes from the 3D pose generation module during inference, and the head pose in the 3D pose is used. The head pose in the training sample is used during training. The construction of training samples will be introduced later. The head pose can reflect the position and viewing angle of the camera in disguise. Although the generated video should be the result of a fixed camera position and a fixed viewing angle in theory, the movement is relative. The movement of the head in the camera picture can be understood as the head is actually still, while the camera is moving. For example, turning the head to the right can be converted to turning the camera to the left. Therefore, as long as the head pose, including the position and rotation information of the head, is obtained, it is equivalent to obtaining the position and viewing angle of the camera.When constructing training samples, we first need a video that only contains the target person's head, which is generally longer than 1 minute. We use a portrait segmentation algorithm, such as BiSeNet, to separate the head from the background. During training, we only focus on the generation of the head, and the mouth area can be obtained through the portrait segmentation algorithm for subsequent optimization. In addition, we need to estimate the head posture, including 3D position and rotation information. One implementation method is to use a head posture estimation algorithm, such as OpenFace. The loss function of each frame during training can be defined as:

[0099] ;

[0100] in refers to the frames in the training sample, refers to the frames generated by the neural radiation field, The weight coefficient has the same size as the frame size. One way to define the weight coefficient is that the value of the area belonging to the background is 0, the value of the area belonging to the head is 1, and the value of the area belonging to the mouth is 2. In summary, the input of the head action generation module is voice and 3D posture, and the output is head action video.

[0101] Body motion generation module. The function of this module is to generate a body motion video based on a 3D posture sequence and a user image. The user image can be a frame of a full-body image extracted from the training sample of the head motion generation module. One implementation of this module can use the DreamPose algorithm, which is a diffusion model-based method that can generate relatively stable body motions. The DreamPose algorithm inputs a 3D posture sequence and a full-body image of the user, and can generate a motion video with the same number of frames as the posture sequence, and the generated motion is the same as the motion represented by the 3D posture. The body motion video here also includes the head, but the head here does not have any expression or lip movement. In summary, the body motion generation module inputs user images and 3D posture sequences, and outputs a body motion video.

[0102] Fusion optimization module. The function of this module is to fuse the body movement video generated by the body movement module and the head movement video generated by the head movement module to generate the final 2D digital human video. The fusion optimization module can include a head detection module, a fusion device and a discriminator. The head detection module is used to detect the head area in the body movement video and the head movement video, and output it in the form of a mask. One implementation method can use a portrait segmentation algorithm, such as BiSeNet, to identify the head area. Replace the head area of ​​the body movement video with the head area in the head movement video to obtain a 2D digital human video to be optimized. The direct replacement of the head area here will inevitably produce unnatural situations, such as missing pixels or misalignment. Therefore, a fusion device and a discriminator are added to optimize the 2D digital human video to be optimized. The whole of the fusion device and the discriminator is similar to the Generative Adversarial Network (GAN). During inference, the fusion device inputs the 2D digital human video to be optimized, generates a more realistic and natural image in the defective part of the replacement area, and keeps other areas unchanged. The optimized 2D digital human video is output. During training, the output optimized video will be handed over to the discriminator to determine whether it is realistic enough. One implementation of the fusion device is to use the U-Net network structure, where the input and output resolutions are consistent, and it is often used for image enhancement; one implementation of the discriminator is a multi-layer two-dimensional convolutional layer plus a nonlinear activation function. During training, only the replacement area, that is, the head area, needs to be intercepted and input into the network in the fusion device. The input of the discriminator includes true samples and false samples. The false samples are the output of the fusion device, and the true samples are the target user videos used in the training of the head action generation module. In order to maintain the continuity between frames, and at the same time combine the loss function of traditional adversarial generation, the loss function of the discriminator here can be defined as:

[0103] ;

[0104] in represents the true sample, Represents the output of the discriminator, usually a probability value. The closer it is to 1, the more real it is. represents the output of the fusion, that is, the false sample, represents the image input to the fuser, The bigger the better. The loss function of the fuser can be defined as:

[0105] ;

[0106] in represents the output of the discriminator, represents the output of the fusion, that is, the false sample, Represents the image that is input to the fuser. Represents a custom hyperparameter, one setting value can be 0.2, Represents the head region of the head action video, Represents the head area of ​​the video generated by the fuser, The smaller the better. Plus The purpose of this item is to make the fuser focus on solving the defects caused by splicing without destroying the continuity of the original video. In summary, the input of the fusion optimization module is the body action video and the head action video, and the output is the final 2D digital human video.

[0107] The above four modules together constitute a complete voice-driven 2D digital human video generation method.

[0108] This design can generate a complete 2D digital human video including head and body movements based on voice. Many current methods only focus on the generation of head movements or mouth movements. Even if body movements are included, they are basically unrelated to the input voice. This method is related to voice regardless of body movements or head movements, and has stronger consistency. It can reduce labor costs. Traditionally, recording a spoken video of a target user requires preparing a venue, preparing a script, multiple recordings, and editing. The spoken video can be used. Now, you only need to input the user's voice to directly generate a video, which greatly saves time and labor costs. In actual use, you only need to input a voice to directly obtain the target user's 2D digital human video, which is very convenient and quick for users.

[0109] Please refer to Figure 2 , Figure 2 A voice-driven 2D digital human video generation device 110 provided in an embodiment of the present invention includes:

[0110] An acquisition module 1101 is used to acquire a target speech;

[0111] The processing module 1102 is used to obtain a 3D gesture sequence corresponding to the audio data of the target voice according to the audio data of the target voice; obtain a head movement video corresponding to the target voice according to the target voice and the 3D gesture sequence; and obtain a body movement video according to the user image and the 3D gesture sequence;

[0112] The generation module 1103 is used to obtain a 2D digital human video by fusing the head action video and the body action video.

[0113] It should be noted that the implementation principle of the aforementioned voice-driven 2D digital human video generation device 110 can refer to the implementation principle of the aforementioned voice-driven 2D digital human video generation method, which will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the voice-driven 2D digital human video generation device 110 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above voice-driven 2D digital human video generation device 110. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or an instruction in software form.

[0114] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0115] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned voice-driven 2D digital human video generation device 110. Figure 3 As shown, Figure 3The computer device 100 provided in the embodiment of the present invention is a structural block diagram. The computer device 100 includes a voice-driven 2D digital human video generation device 110 , a memory 111 , a processor 112 and a communication unit 113 .

[0116] To achieve data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these components can be achieved through one or more communication buses or signal lines. The voice-driven 2D digital human video generation device 110 includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the voice-driven 2D digital human video generation device 110 stored in the memory 111, such as the software function modules and computer programs included in the voice-driven 2D digital human video generation device 110.

[0117] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the aforementioned voice-driven 2D digital human video generation method.

[0118] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A speech-driven 2D digital human video generation method, It is characterized in that include: Get the target voice; According to the audio data of the target speech, obtaining a 3D posture sequence corresponding to the audio data; According to the target speech and the 3D posture sequence, a head movement video corresponding to the target speech is obtained; Obtaining a body motion video according to the user image and the 3D posture sequence; According to the head movement video and the body movement video, a 2D digital human video is obtained by fusing; The step of obtaining a head movement video corresponding to the target speech according to the target speech and the 3D posture sequence includes: Encoding the audio data through an audio encoder to obtain an audio feature vector sequence corresponding to the audio data; Extracting a 3D head posture sequence from the 3D posture sequence, and inputting the 3D head posture sequence into a light processor to obtain 3D position information corresponding to the 3D head posture sequence, wherein the 3D position information includes the 3D position of the sampling point corresponding to the 3D head posture of each frame in the 3D head posture sequence in the light direction; Inputting the 3D position information into a position encoder to obtain a position encoding feature sequence; Inputting the position coding feature sequence, the audio feature vector sequence and the light direction of the sampling point into a pre-trained neural radiation field to obtain the color and density information of each sampling point; Volume rendering is performed on the color and density information of each sampling point to obtain the head movement video, where the head movement video includes facial expressions and lip movements.

2. The method according to claim 1, It is characterized in that The step of acquiring a 3D gesture sequence corresponding to the audio data according to the audio data of the target speech includes: Encoding the audio data through an audio encoder to obtain an audio feature vector sequence corresponding to the audio data; The audio feature vector sequence is input into a gesture generator based on a plurality of gated recurrent units to obtain the 3D gesture sequence.

3. The method according to claim 2, It is characterized in that The audio encoder includes a primary encoding unit and a high-order encoding unit, the high-order encoding unit is constructed by a multi-layer linear network layer and a nonlinear activation function, and the audio data is encoded by the audio encoder to obtain an audio feature vector sequence corresponding to the audio data, including: Encoding the audio data by the primary encoding unit to obtain a plurality of initial audio feature vectors, each of which includes the logarithmic amplitude and energy of the Mel spectrum of the current frame; Performing feature conversion on the multiple initial audio feature vectors by the high-order coding unit to obtain the audio feature vector sequence, wherein the audio feature vector sequence includes a high-order audio feature vector corresponding to each frame of audio data; The step of inputting the audio feature vector sequence into a gesture generator based on a plurality of gated recurrent units to obtain the 3D gesture sequence comprises: Input the high-order audio feature vector corresponding to the i-th frame and the posture data corresponding to the i-1-th frame into the pre-trained posture generator to obtain the posture data corresponding to the i-th frame, where i is a positive integer. When i=1, the posture data corresponding to the i-1-th frame is randomly initialized posture data, and the posture data includes the absolute position of the human skeleton key points and the rotation value of the human skeleton key points of the current frame; The acquired multiple posture data are integrated to obtain the 3D posture sequence, and the 3D posture sequence corresponds one-to-one to the elements of the audio feature vector sequence.

4. The method according to claim 1, It is characterized in that The neural radiation field is given by the formula: It is obtained by training the loss function, where refers to the frames in the training sample, refers to the frames generated by the neural radiation field, Refers to the weight coefficient.

5. The method according to claim 1, It is characterized in that The step of obtaining a body motion video according to the user image and the 3D posture sequence includes: The user image and the 3D posture sequence are input into a preset diffusion model to obtain the body movement video.

6. The method according to claim 1, It is characterized in that The step of fusing the head action video and the body action video to obtain a 2D digital human video includes: Dividing the body action video into a pending head action video and a pending body action video based on a preset portrait segmentation algorithm; The head action video and the pending body action video are input into a fusion device trained by a discriminator, so that the head action video replaces the pending head action video to obtain the 2D digital human video.

7. The method according to claim 6, It is characterized in that The loss function of the discriminator is: ; in, represents the true sample, represents the output of the discriminator, represents the output of the fuser, represents the head action video and pending body action video input to the fuser, The larger it is, the more accurate the discriminator's judgment is. is the mathematical expectation; The loss function of the fuser is: ; in, represents the output of the discriminator, represents the output of the fuser, represents the head action video and pending body action video input to the fuser, represents the hyperparameter, Represents the head region of the head action video, Represents the head area of ​​the video generated by the fuser, The smaller it is, the more accurate the output of the fusion device is. is the mathematical expectation.

8. A voice-driven 2D digital human video generation device, It is characterized in that include: An acquisition module, used to acquire target speech; A processing module, configured to obtain a 3D gesture sequence corresponding to the audio data of the target voice according to the audio data of the target voice; obtain a head movement video corresponding to the target voice according to the target voice and the 3D gesture sequence; and obtain a body movement video according to the user image and the 3D gesture sequence; A generation module, used for fusing the head action video and the body action video to obtain a 2D digital human video; The processing module is specifically used for: The audio data is encoded by an audio encoder to obtain an audio feature vector sequence corresponding to the audio data; a 3D head posture sequence is extracted from the 3D posture sequence, and the 3D head posture sequence is input into a light processor to obtain 3D position information corresponding to the 3D head posture sequence, wherein the 3D position information includes the 3D position of the light direction of the sampling point corresponding to the 3D head posture of each frame in the 3D head posture sequence; the 3D position information is input into a position encoder to obtain a position encoding feature sequence; the position encoding feature sequence, the audio feature vector sequence and the light direction of the sampling point are input into a pre-trained neural radiation field to obtain color and density information of each sampling point; the color and density information of each sampling point is volume rendered to obtain the head movement video, wherein the head movement video includes facial expressions and lip movements.

9. A readable storage medium, It is characterized in that The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the voice-driven 2D digital human video generation method according to any one of claims 1 to 7.