Digital human video generation method and system based on audio driving

Abstract

The invention discloses a digital human video generation method and system based on audio driving, and the method comprises the steps: S1, obtaining input audio data, carrying out the audio preprocessing and audio feature extraction of an audio, and obtaining an audio feature vector reflecting the audio content, emotion and rhythm; s2, carrying out picture preprocessing and picture feature extraction on an input face picture to obtain face appearance features; s3, mapping the audio features and the face appearance features into expression parameters and action parameters; and S4, generating a complete digital human video based on the input face picture and the generated expression parameters, action parameters and audio information. According to the method, the digital human video can be generated only by taking the human face picture as the basic material and combining the audio information, so that the dependence on an image video with a certain duration is eliminated, the threshold of generating the digital human video is greatly reduced, and the expression is more real by adopting a new technical scheme.

Description

Technical Field

[0001] The present application relates to the technical field of virtual digital human generation, and in particular to a method and system for generating digital human videos based on audio drive. Background Art

[0002] In the digital age, virtual digital humans have gained increasing attention in various fields such as entertainment, education and training, and corporate services. Currently, there are two main methods for generating digital humans:

[0003] 1. The user provides a video of a certain length, and then combines the text posture vector to replace the lip shape in the template video to generate a driving video, which is then combined with the user's photo to generate a digital human.

[0004] 2. Generate a digital human video based on the image photo, but the generated video can only move the lips, and the rest remains still.

[0005] Existing digital human generation suffers from several drawbacks: It requires a video of the person's image to generate a digital human video, which not only requires the video but also meets certain duration requirements. Digital human videos generated solely from audio-driven images only move the lips, while other facial expressions struggle to adapt accordingly, resulting in an unnatural and stilted appearance. Summary of the Invention

[0006] To solve the technical problems existing in the prior art, this application provides a method for generating digital human videos based on audio drive, comprising the following steps:

[0007] S1: Obtain input audio data, perform audio preprocessing and audio feature extraction on the audio, and obtain an audio feature vector reflecting the audio content, emotion, and rhythm;

[0008] S2: Perform image preprocessing and image feature extraction on the input face image to obtain face appearance features;

[0009] S3: Mapping the audio features and facial appearance features into expression parameters and action parameters;

[0010] S4: Based on the input face image and the generated expression parameters and action parameters, each frame of the digital human video is generated, and each generated frame of the image is combined in chronological order and audio information is added to generate a complete digital human video.

[0011] Furthermore, step S3 includes:

[0012] S31: Input the audio feature vector and the facial appearance feature vector into the deep learning diffusion model to obtain the corresponding digital human expression parameters and action parameters;

[0013] S32: Based on the features extracted from the face image, the generated expression and action parameters are adjusted and optimized.

[0014] Furthermore, step S31 specifically includes:

[0015] S311: Get the extracted facial appearance features and audio feature vectors and splice them to get the spliced ​​feature F cat ;

[0016] S312: Splice feature F cat Send it to the linear layer to get the linear feature F Linear ;

[0017] S313: Linear feature F Linear Send it to the activation layer and use the ReLU activation function to get the expression parameter m′ e and action parameter m′ t .

[0018] Furthermore, step S32 includes:

[0019] S321: Obtain facial appearance features and generate optimization coefficient β through linear layer and activation layer;

[0020] S322: The expression parameter m′ e and action parameter m′ t Multiply by the optimization coefficient β to obtain new expression parameters and action parameters;

[0021] S323: Inputting the expression parameters and the action parameters into the position coding layer respectively to obtain the expression features and action features after position coding;

[0022] S324: Input the position-encoded expression features and action features into the linear layer to obtain the optimized expression parameter m e and action parameter m t .

[0023] Furthermore, the audio preprocessing and audio feature extraction in step S1 include:

[0024] Audio preprocessing: noise reduction and normalization of input audio data;

[0025] Audio extraction features, including:

[0026] Read the input audio data, use the whisper model to transcribe the audio, use the sliding window method to segment the audio, and obtain the audio's time series features;

[0027] The temporal features of the audio are divided into blocks according to the video frame rate, so that the continuous audio features are aligned with the discrete video frames to obtain the aligned audio block features.

[0028] Furthermore, the step S2 of preprocessing the input face image and extracting features from the image includes:

[0029] S21: Image preprocessing, including cropping and scaling of images;

[0030] S22: Image feature extraction: Based on the pre-trained facial feature extraction model, key feature points and appearance features in the facial image are extracted, including:

[0031] S221: Input the pre-processed image into the face detection model to obtain the position information of the face part and the coordinates of key feature points;

[0032] S222: Input the preprocessed image into the autovariation encoder to obtain the encoded appearance features;

[0033] S223: After normalizing the encoded appearance features, the final image appearance features are obtained.

[0034] The present invention also discloses a digital human video generation system for implementing the digital human video generation method, comprising an audio feature extraction module, a face image feature extraction module, an expression and action mapping module, and a digital human video generation module, wherein:

[0035] The audio feature extraction module is used to obtain input audio data, perform audio preprocessing and audio feature extraction on the audio, and obtain an audio feature vector that reflects the audio content, emotion and rhythm;

[0036] The face image feature extraction module is used to perform image preprocessing and image feature extraction on the input face image to obtain face appearance features;

[0037] The expression and action mapping module is used to map the audio features and facial appearance features into expression parameters and action parameters;

[0038] The digital human video generation module is used to generate each frame of the digital human video based on the input face image and the generated expression parameters and action parameters, and to combine each generated frame in chronological order and add audio information to generate a complete digital human video.

[0039] Furthermore, the expression action mapping module includes a mapping unit and an optimization unit, wherein:

[0040] The mapping unit is used to input the audio feature vector and the facial appearance feature vector into the deep learning diffusion model to obtain the corresponding digital human expression parameters and action parameters;

[0041] The optimization unit is used to adjust and optimize the generated expression and action parameters based on the features extracted from the face image.

[0042] The present invention also provides a storage medium, on which a computer program is stored. When the computer program is run by a processor, the method for generating a digital human video is executed.

[0043] The present invention also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the digital human video generation method as described.

[0044] In actual applications, the modules described in the method and system disclosed in this application can be deployed in multiple modules on one target server, or each module can be independently deployed on a different target server. In particular, as needed, in order to provide more powerful computing processing capabilities, the modules can also be deployed on cluster target servers as needed.

[0045] It can be seen that the technical effects achieved by the technology adopted in this application are:

[0046] 1. Innovation in driving mode: This invention only requires facial images as basic materials and combines them with audio information to generate digital human videos. It gets rid of the dependence on image videos of a certain length, greatly lowers the threshold for digital human video generation, enables users to obtain the required digital human videos more conveniently, and improves the efficiency and flexibility of digital human video generation.

[0047] 2. Improved Liveliness and Naturalness: The generated digital human videos not only move their lips, but also their facial expressions and head movements respond to the audio, achieving a more natural and vivid digital human image. This improvement makes digital human videos more expressive and can better meet the needs of users in various scenarios, such as virtual anchors, video production, education, and entertainment, providing users with a more realistic and immersive experience.

[0048] 3. Expanded Applications: Due to the flexibility and high-quality output of this method, it has broad application prospects in multiple fields. For example, in virtual social scenarios, users can quickly generate personalized digital human avatars for communication and interaction through simple photo and voice input. In the advertising field, it can produce more attractive and appealing digital human advertising videos, improving advertising effectiveness.

[0049] In order to have a clearer and more comprehensive understanding of the present application, the specific implementation methods of the present application are described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0051] Figure 1 This is a flow chart of a method for generating digital human videos based on audio drive according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solution of the present application is a method for generating digital human videos based on audio drive, comprising the following steps:

[0053] S1: Obtain input audio data, perform audio preprocessing and audio feature extraction on the audio, and obtain an audio feature vector reflecting the audio content, emotion, and rhythm;

[0054] S2: Perform image preprocessing and image feature extraction on the input face image to obtain face appearance features;

[0055] S3: Mapping the audio features and facial appearance features into expression parameters and action parameters;

[0056] S4: Based on the input face image and the generated expression parameters and action parameters, each frame of the digital human video is generated, and each generated frame of the image is combined in chronological order and audio information is added to generate a complete digital human video.

[0057] The technical solution of the present application is described in detail below with reference to various specific embodiments.

[0058] S1: Obtain input audio data, perform audio preprocessing and audio feature extraction on the audio, and obtain an audio feature vector reflecting the audio content, emotion, and rhythm;

[0059] Receive external input audio data, process the input audio data, and improve the audio quality through pre-processing operations such as noise reduction and normalization. Audio pre-processing and audio feature extraction specifically include:

[0060] Audio preprocessing: noise reduction and normalization of input audio data. Preprocessing operations such as noise reduction and normalization can greatly improve the audio quality.

[0061] Audio extraction features, including:

[0062] First, read the input audio data and ensure that the input audio is at a standard sampling frequency, such as 16kHz or above. Then, perform the above-mentioned denoising, normalization and other preprocessing to ensure the accuracy of subsequent processing. After the above processing, use the whisper model to transcribe the audio. Other models of debate topics can also be used to implement it. Use a sliding window method to segment the audio and obtain the audio's time series features. As an implementation method, the sliding window and the method of extracting the audio's time series features include the following:

[0063] Set up the sliding window, including:

[0064] Define the window length (such as window_size, usually a few seconds such as 5 seconds);

[0065] Define the step size (hop_size, usually less than or equal to the window length, such as 2.5 seconds) to achieve overlapping windows.

[0066] Processing by section, including:

[0067] Starting from the beginning of the audio, gradually cut the audio clips;

[0068] Run the model on each segment to obtain transcribed text and / or audio features.

[0069] Extract features, including: In addition to the transcribed text, also extract Whisper's hidden state in the middle layer or the final logits as time series features, which are feature representations within each window and are used for subsequent audio time series modeling.

[0070] Secondly, the temporal features of the audio are divided into blocks according to the video frame rate, so that the continuous audio features are aligned with the discrete video frames to obtain the aligned audio block features.

[0071] S2: Preprocessing the input face image and extracting features from the image to obtain face appearance features. As a preferred embodiment, the method includes:

[0072] S21: Image preprocessing, including cropping and scaling of images;

[0073] According to the input face image, the region of interest (ie, the face region) is cropped out. The cropping can be performed based on the detection results of the face detection model.

[0074] The cropped face area is uniformly scaled to a predetermined size (e.g., 112x112 pixels) to ensure consistency in subsequent model inputs. The aspect ratio of the face is maintained during the scaling process to avoid deformation. Proportional scaling and padding are performed when necessary.

[0075] S22: Image feature extraction: Based on the pre-trained facial feature extraction model, key feature points and appearance features in the facial image are extracted, including:

[0076] S221: Input the preprocessed image into the face detection model. The face model that can be selected is MTCNN or RetinaFace, etc., to obtain the position information of the face part, the position of the face border, and the coordinates of key feature points (such as the positions of key points such as eyes, nose tip, and mouth corners).

[0077] S222: Input the preprocessed image into the autovariation encoder, and obtain the encoded low-dimensional encoding vector through the encoder, which represents the appearance characteristics of the face (such as texture, skin color, details, etc.).

[0078] S223: After normalizing the encoded appearance features, the features are made to have a uniform scale and distribution, thereby obtaining the final image appearance features.

[0079] S3: Mapping the audio features and facial appearance features into expression parameters and action parameters

[0080] This step S3 specifically includes S31 and S32, wherein:

[0081] S31: Input the audio feature vector and the facial appearance feature vector into the deep learning diffusion model to obtain the corresponding digital human expression parameters and action parameters. As a preferred implementation, specifically including:

[0082] S311: Get the extracted facial appearance features and audio feature vectors and splice them to get the spliced ​​feature F cat ;

[0083] S312: Splice feature F cat Send it to the linear layer to get the linear feature F Linear ;

[0084] S313: Linear feature F Linear Send it to the activation layer and use the ReLU activation function to get the expression parameter m′ e and action parameter m′ t , the output vector can be regarded as the expression parameters (such as smile, anger, etc.) and action parameters (such as head rotation, gesture, etc.) of the corresponding face.

[0085] S32: Based on the features extracted from the face image, the generated expression and action parameters are adjusted and optimized. As a preferred embodiment, step S32 includes:

[0086] S321: Obtain facial appearance features, and generate optimization coefficients β through linear layers and activation layers. This set of coefficients is used to adjust the amplitude and details of the subsequently generated expression and action parameters.

[0087] S322: The expression parameter m′ e and action parameter m′ t Multiplying by the optimization coefficient β obtains new expression parameters and action parameters. This step dynamically adjusts the amplitude of the parameters to make the final expression and action closer to the facial features, enhancing the naturalness and personalization.

[0088] S323: Input the expression parameters and action parameters into the position coding layer respectively to obtain the expression features and action features after position coding. Position coding can enhance the spatial position features of the parameters, help the model capture the spatial or temporal relationship between the parameters, and improve the continuity and naturalness of the expression.

[0089] S324: Input the position-encoded expression features and action features into the linear layer to obtain the optimized expression parameter m e and action parameter m t ,The spatial information of position encoding is integrated through linear transformation to obtain the final parameters that are more natural and fit the facial features.

[0090] Step S32 enables the generated digital human animation to better fit individual facial features and dynamic expression requirements while maintaining naturalness and smoothness.

[0091] S4: Based on the input face image and the generated expression parameters and action parameters, each frame of the digital human video is generated, and each generated frame of the image is combined in chronological order and audio information is added to generate a complete digital human video.

[0092] First, after the aforementioned S1-S3 steps, the basic facial image, expression parameters and action parameters are obtained. These parameters are used to edit or deform each frame of the facial image, synthesize dynamic expressions and actions, and generate continuous facial images frame by frame. This can be achieved using 3D or 2D deformation models, 3D face models (such as FLAME, 3DMM), 2D face deformation networks (such as FaceLandmark-based Morphing, Facial Motion Transfer models), or conditional generation models such as GAN, VAE, and Transformers for facial deformation fusion.

[0093] Secondly, all frames are synthesized into a video in chronological order, that is, continuous frames are sampled according to the set frame rate (such as 25fps, 30fps), and each frame image is saved as a continuous picture (such as JPEG or PNG format), and then tools such as OpenCV or FFmpeg are used to transcode the sequence frames into a video (MP4, AVI, etc. format).

[0094] Finally, use tools such as FFmpeg to superimpose the audio track into the video to obtain a complete digital human.

[0095] Based on the above embodiments, the present application further provides a digital human video generation system, including an audio feature extraction module, a face image feature extraction module, an expression and action mapping module, and a digital human video generation module, wherein:

[0096] The audio feature extraction module is used to obtain input audio data, perform audio preprocessing and audio feature extraction on the audio, and obtain an audio feature vector that reflects the audio content, emotion and rhythm;

[0097] The face image feature extraction module is used to perform image preprocessing and image feature extraction on the input face image to obtain face appearance features;

[0098] The expression and action mapping module is used to map the audio features and facial appearance features into expression parameters and action parameters;

[0099] The digital human video generation module is used to generate each frame of the digital human video based on the input face image and the generated expression parameters and action parameters, and to combine each generated frame in chronological order and add audio information to generate a complete digital human video.

[0100] The expression action mapping module includes a mapping unit and an optimization unit, where:

[0101] The mapping unit is used to input the audio feature vector and the facial appearance feature vector into the deep learning diffusion model to obtain the corresponding digital human expression parameters and action parameters;

[0102] The optimization unit is used to adjust and optimize the generated expression and action parameters based on the features extracted from the face image.

[0103] An embodiment of the present application further provides a storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the method for generating a digital human video as described above is executed.

[0104] An embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the digital human video generation method as described.

[0105] It should be noted that, those skilled in the art can understand that all or part of the split steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0106] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating digital human video based on audio drive, characterized in that: The following steps are involved: S1: Obtain input audio data, perform audio preprocessing and audio feature extraction on the audio, and obtain an audio feature vector reflecting the audio content, emotion, and rhythm; S2: Perform image preprocessing and image feature extraction on the input face image to obtain face appearance features; S3: Mapping the audio features and facial appearance features into expression parameters and action parameters; S4: Based on the input face image and the generated expression parameters and action parameters, each frame of the digital human video is generated, and each generated frame of the image is combined in chronological order and audio information is added to generate a complete digital human video.

2. The method for generating a digital human video according to claim 1, wherein: Step S3 includes: S31: Input the audio feature vector and the facial appearance feature vector into the deep learning diffusion model to obtain the corresponding digital human expression parameters and action parameters; S32: Based on the features extracted from the face image, the generated expression and action parameters are adjusted and optimized.

3. The method for generating a digital human video according to claim 2, wherein: Step S31 specifically includes: S311: Get the extracted facial appearance features and audio feature vectors and splice them to get the spliced ​​feature F cat ; S312: Splice feature F cat Send it to the linear layer to get the linear feature F Linear ; S313: Linear feature F Linear Send it to the activation layer and use the ReLU activation function to get the expression parameter m′ e and action parameter m′ t .

4. The method for generating a digital human video according to claim 2, wherein: Step S32 includes: S321: Obtain facial appearance features and generate optimization coefficient β through linear layer and activation layer; S322: The expression parameter m′ e and action parameter m′ t Multiply by the optimization coefficient β to obtain new expression parameters and action parameters; S323: Inputting the expression parameters and the action parameters into the position coding layer respectively to obtain the expression features and action features after position coding; S324: Input the position-encoded expression features and action features into the linear layer to obtain the optimized expression parameter m e and action parameter m t .

5. The method for generating a digital human video according to claim 1, wherein: The audio preprocessing and audio feature extraction in step S1 include: Audio preprocessing: noise reduction and normalization of input audio data; Audio extraction features, including: Read the input audio data, use the whisper model to transcribe the audio, use the sliding window method to segment the audio, and obtain the audio's time series features; The temporal features of the audio are divided into blocks according to the video frame rate, so that the continuous audio features are aligned with the discrete video frames to obtain the aligned audio block features.

6. The method for generating a digital human video according to claim 1, wherein: In step S2, the input face image is preprocessed and features are extracted from the image, including: S21: Image preprocessing, including cropping and scaling of images; S22: Image feature extraction: Based on the pre-trained facial feature extraction model, key feature points and appearance features in the facial image are extracted, including: S221: Input the pre-processed image into the face detection model to obtain the position information of the face part and the coordinates of key feature points; S222: Input the preprocessed image into the autovariation encoder to obtain the encoded appearance features; S223: After normalizing the encoded appearance features, the final image appearance features are obtained.

7. A digital human video generation system for implementing the digital human video generation method according to any one of claims 1 to 6, characterized in that: It includes audio feature extraction module, face image feature extraction module, expression action mapping module, and digital human video generation module. in: The audio feature extraction module is used to obtain input audio data, perform audio preprocessing and audio feature extraction on the audio, and obtain an audio feature vector that reflects the audio content, emotion and rhythm; The face image feature extraction module is used to perform image preprocessing and image feature extraction on the input face image to obtain face appearance features; The expression and action mapping module is used to map the audio features and facial appearance features into expression parameters and action parameters; The digital human video generation module is used to generate each frame of the digital human video based on the input face image and the generated expression parameters and action parameters, and to combine each generated frame in chronological order and add audio information to generate a complete digital human video.

8. The digital human video generation system according to claim 7, wherein: The expression action mapping module includes a mapping unit and an optimization unit, where: The mapping unit is used to input the audio feature vector and the facial appearance feature vector into the deep learning diffusion model to obtain the corresponding digital human expression parameters and action parameters; The optimization unit is used to adjust and optimize the generated expression and action parameters based on the features extracted from the face image.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is run by the processor, the method for generating a digital human video according to any one of claims 1 to 6 is executed.

10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the digital human video generation method as described in any one of claims 1-6.

Images