Efficient method for driving quadratic element picture expressions and actions based on audio

By using an audio-driven method for generating two-dimensional image expressions and actions, a sequence of faces and poses is constructed. A neural network module is then used to generate realistic expressions and actions, overcoming the limitations and computational speed issues in existing technologies and achieving real-time emotional responses and high-quality virtual character performance.

CN119048655APending Publication Date: 2024-11-29SHENZHEN QIANJI TECHNOLOGY CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411077206.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing technologies have many limitations on human images, resulting in poor image quality, slow computation speed, inability to run in real time on mobile devices, lack of audio-driven emotion models, and limited expression and torso rotation.

Method used

This method uses audio analysis to drive facial expressions and movements in 2D images, constructs facial feature sequences and pose sequences, extracts 3D facial mesh and head pose sequences using a transformer-based model, projects them onto 2D facial landmarks, and combines neural network modules to generate expressions and poses, including a face converter, low-resolution rotator, and editor, to optimize image quality.

Benefits of technology

It achieves real-time perception of user emotions, generates realistic expressions and actions, improves dialogue quality, can run on mobile devices, adapts to any image for emotional expression, and enhances the expressiveness of virtual characters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048655A_ABST
    Figure CN119048655A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of video generation, in particular to an efficient method for driving quadratic element picture expressions and actions based on audio, and the method comprises the steps: extracting audio data features, and constructing a face feature sequence and a posture sequence; inputting a picture, and obtaining face feature points of quadratic elements; and the face features and postures are mapped to figure features of any two-dimensional figure picture, so that refined rapid driving is realized. According to the method, under the condition that a character skeleton model does not need to be independently established, feature matching is achieved, real-time driving of different emotion expressions of character pictures is achieved, the system is a quadratic element image dialogue system based on audio driving, the system uses audio analysis to drive a quadratic element picture to directly dialogue with people, and the system is convenient to use. According to the method, the emotion of the user can be perceived in real time, emotional response is given, the dialogue quality is improved, the emotional demand of the user can be better met, and many defects existing in an existing method are overcome.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to an efficient method for driving expressions and actions of two-dimensional images based on audio. Background Art

[0002] With the popularization of the Internet and the rise of social media, the demand for sentiment analysis has gradually increased. A large amount of text and voice data contains people's emotional information. At the same time, the development of artificial intelligence technology has provided better technical support and data processing methods for sentiment analysis. Users' demand for emotional chatbot images has also promoted the development of sentiment analysis.

[0003] Existing methods have many defects:

[0004] 1. There are many restrictions on the input of human images, and the face needs to be centered;

[0005] 2. For the occluded parts, there are problems with the quality of the generated images. When the occluded parts of the character rotate and become visible, they are often blurry;

[0006] 3. Existing mainstream methods are often slow in calculation, require more powerful GPUs to run in real time, and cannot be effectively used on mobile devices;

[0007] 4. There is no audio-driven emotion model, and various facial expressions need to be controlled manually;

[0008] 5. Currently, only small-angle rotation of facial organs and movement of the torso can be achieved. Summary of the invention

[0009] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides an efficient method for driving the expressions and actions of two-dimensional pictures based on audio. It is a system that uses audio analysis to drive two-dimensional pictures to directly communicate with people. It can complete real-time perception of user emotions, give emotional responses, improve the quality of conversations, and better meet the emotional needs of users.

[0010] To achieve the above object, the present invention provides the following technical solutions:

[0011] An efficient method for driving expressions and actions of two-dimensional images based on audio, comprising the following steps:

[0012] S1, extract audio data and construct facial feature sequence and posture sequence;

[0013] S1.1, audio extraction features;

[0014] S1.2, convert the audio extracted features into 3D facial mesh to obtain the facial feature sequence;

[0015] S1.3, extract posture from audio and obtain posture sequence;

[0016] S1.3, build a material library and introduce reference information;

[0017] S2, converting the facial feature sequence and posture sequence in S1 into input, and then driving the two-dimensional image photo to complete the two-dimensional image action sequence;

[0018] S2.1, input the picture, obtain the second dimension face feature points through face recognition and feature extraction;

[0019] S2.2, input the facial feature sequence and pose sequence calculated in S1, after obtaining the mesh and pose sequences, convert them into a 2D facial landmarks sequence to obtain a 45-dimensional pose vector, and then drive the pose vector through the two-dimensional expression system to generate a new image with a new pose;

[0020] S2.3, the posture-converted image and appearance flow graph are output, enlarged, and passed to the editor to improve the image quality, and finally returned to the user.

[0021] Preferably, in step S1, a transformer-based model is used to extract a 3D facial mesh and head pose sequence from the audio input and project them into a 2D facial landmark.

[0022] Preferably, in step S1.1, the audio feature H is extracted using the pre-trained wav2vec 2.0 1:T =(h 1 ,...,h T ), and perform corresponding emotion recognition to obtain the emotional feature C 1:T =(c 1 ,...,c T ), the audio features and emotional features are encoded by the encoder to obtain the new audio features A 1:T , the specific calculation method is as follows:

[0023] A 1:T =∑(αH 1:T βC 1:T ).

[0024] Preferably, in step S1.2, the extracted audio feature A 1:T =(a 1 ,...,a T ) A simple architecture consisting of a 34-layer ResNet and two fc layers is used to convert the extracted audio features into a 3D facial mesh M 1:T =(m 1 ,...,mT ), the training loss is L1loss, and the specific calculation method is as follows:

[0025]

[0026] Preferably, in step S1.3, the extracted audio feature A 1:T =(a 1 ,...,a T ) to infer the action P of the head posture 1:T =(p 1 ,...,p T ), build an independent 34-layer ResNet and use transformer as decoder to decode the pose sequence. Using transformer will increase the generation speed;

[0027] The audio features are integrated into the decoder through the cross-attention mechanism. The improved model structure consists of three parts: a shared encoder, a CTC decoder, and an attention decoder. The shared encoder is an independent 34-layer ResNet. The CTC decoder is a fully connected layer and a softmax layer. The attention decoder contains multiple transformer layers. The loss function formula is:

[0028] L combin (x,y)=λL ctc (x,y)+(1-λ)L aed (x,y).

[0029] Preferably, in step S2.1, the input image is a "half-length portrait" of a humanoid cartoon character and is processed into a 512×512 image. The character image can be obtained through image generation technology or can be a copyrighted self-drawn image.

[0030] Preferably, in step S2.2, the facial feature sequence and pose sequence calculated in S1 are input, and after obtaining the mesh and pose sequences, they are converted into a 2D facial landmarks sequence using perspective projection to obtain a 45-dimensional pose vector, of which 39 are used to control the expression of the character, and 6 are used to control the rotation of the face and body;

[0031] Among them, the system consists of 5 neural networks and can be divided into two modules. Three networks constitute the face converter module, and the remaining two networks constitute the "low-resolution rotator" and "editor" respectively.

[0032] Preferably, the face converter module is responsible for changing the facial expression of the character, and the face converter module is responsible for analyzing the features of 2D facial landmarks and reference images, integrating this information into the generation process, and ensuring that the generated facial movements are consistent with the input audio and reference images.

[0033] Preferably, the low-resolution rotator and the editor together constitute a body rotator module, which is responsible for calculating the overall image effect of rotating the head and body with non-expression parameters according to the posture and appearance flow graph, and calculating a low-resolution image feature.

[0034] Preferably, the editor is also responsible for optimizing the details of the generated facial movements, such as the accuracy of lip sync and the naturalness of facial expressions, so that the final animation is more realistic and expressive;

[0035] Among them, the image after the facial image feature points are changed by perspective projection is I direct , the image of the torso after the appearance flow graph changes is I wrapper , the Mask of the Alpha channel is I alpha, The loss function is:

[0036] L total =∑(I wrapper -I wrapper ’ )+∑(I direct -I direct ’ );

[0037] The final image after processing through the body rotator and editor is:

[0038] I total (x,y)=λI alpha I direct (x,y)+(1-λ)I alpha I wrapper (x,y).

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The first stage of the present invention focuses on feature extraction of audio, constructing facial feature sequences and posture sequences, extracting key features from the audio, and generating a detailed posture sequence. This posture sequence contains the head posture and body posture information required for each frame of animation, laying a solid foundation for subsequent video generation steps, and can complete real-time perception of user emotions, give emotional responses, improve the quality of conversations, and better meet the emotional needs of users.

[0041] The second stage of the present invention is to accurately map the facial features and posture information generated in the first stage to the character features on any given two-dimensional character picture, which can realize the refined and rapid driving of the two-dimensional character picture, so that the originally static picture can show rich expressions and actions in real time according to the audio input, enhance the expressiveness of the virtual character, can adapt to any picture, and can realize feature matching to achieve real-time driving of different emotional expressions of the character picture without the need to independently establish a character skeleton model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0043] Figure 1 It is a schematic diagram of the overall system framework of the present invention;

[0044] Figure 2 It is a schematic diagram of the first stage framework of the present invention;

[0045] Figure 3 This is a schematic diagram of the functional architecture of the first stage of the present invention;

[0046] Figure 4 It is a schematic diagram of an audio posture feature extraction module of the present invention;

[0047] Figure 5 This is a schematic diagram of the functional architecture of the second stage of the present invention;

[0048] Figure 6 This is an example diagram of the two-dimensional facial feature points of the present invention;

[0049] Figure 7 This is a partial example diagram of facial expression changes of the present invention;

[0050] Figure 8 It is a schematic diagram of a body rotator module of the present invention;

[0051] Fig. 9 It is a partial schematic diagram of the body rotator effect of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0053] like Figure 1 - Fig. 9 As shown, this embodiment proposes an efficient method for driving two-dimensional image expressions and actions based on audio, including the following steps:

[0054] S1. Extract audio data and construct facial feature sequences and posture sequences. Use a transformer-based model to extract 3D facial mesh and head pose sequences from audio input and project them into 2D facial landmarks. This stage can capture subtle expressions and lip movements from the audio, as well as head movements synchronized with the audio rhythm.

[0055] S1.1, audio feature extraction, use the pre-trained wav2vec 2.0 to extract audio features H 1:T =(h 1 ,...,h T ), and perform corresponding emotion recognition to obtain the emotional feature C 1:T =(c 1 ,...,c T ), the audio features and emotional features are encoded by the encoder to obtain the new audio features A 1:T , the specific calculation method is as follows:

[0056] A 1:T =∑(αH 1;T βC 1;T ).

[0057] S1.2, convert the audio extracted features to 3D facial mesh, obtain the facial feature sequence, and extract the audio features A 1 :T =(a 1 ,...,a T ) A simple architecture consisting of a 34-layer ResNet and two fc layers is used to convert the extracted audio features into a 3D facial mesh M 1:T =(m 1 ,...,m T), the training loss is L1loss, and the specific calculation method is as follows:

[0058]

[0059] S1.3, extract the posture from the audio and obtain the posture sequence; extract the audio feature A 1:T =(a 1 ,...,a T ) to infer the action P of the head posture 1:T =(p 1 ,...,p T ), but does not share weights with the audio to mesh module, because the pose is more closely related to the rhythm and pitch in the audio, which is different from the focus of the audio to mesh task. Then, an independent 34-layer ResNet is established, considering the influence of the previous state, and using transformer as a decoder to decode the pose sequence. Using transformer will increase the generation speed;

[0060] Among them, the audio features are integrated into the decoder through the cross-attention mechanism. The improved model structure consists of three parts, namely the shared Encoder (feature extraction network), CTC decoder, and Attention decoder. The shared Encoder is an independent 34-layer ResNet, the CTC decoder is a fully connected layer and a softmax layer, and the Attention decoder contains multiple transformer layers. The loss function formula is:

[0061] L combin (x,y)=λL ctc (x,y)+(1-λ)L aed (x,y);

[0062] The Attention decoder uses a cross-attention mechanism to focus on different aspects of the audio features and generates corresponding postures based on these features. This enables the decoder to more accurately predict the posture that best matches the audio input. The transformer used can capture the coherence of posture changes over time when processing time series data, which means that the generated posture sequence is not only synchronized with the audio input, but also the transitions between postures are smooth and natural.

[0063] S1.3. Build a material library. When generating postures and facial features, you can introduce reference information from the material library or a specified video, calculate the style encoding of the reference video, and perform style-adaptive posture or facial expression generation.

[0064] S2, converts the facial feature sequence and posture sequence in S1 into input, and then drives the two-dimensional image photo to complete the two-dimensional image action sequence.

[0065] S2.1. Input image. The input image is a "half-length photo" of a humanoid cartoon character and is processed into a 512×512 image. This character image can be obtained through image generation technology or a copyrighted self-drawn image. The two-dimensional facial feature points are obtained through face recognition and feature extraction.

[0066] S2.2, input the facial feature sequence and pose sequence calculated in S1, after obtaining the mesh and pose sequences, use perspective projection to convert them into a 2D facial landmarks sequence to obtain a 45-dimensional pose vector, and then drive the pose vector to generate a new pose image through the two-dimensional expression system;

[0067] Among them, 39 of the above 45 parameters are used to control the character's expression, and 6 are used to control the rotation of the face and body. The system consists of 5 neural networks and can be divided into two modules. Three networks constitute a module called "face converter", which is responsible for changing the character's facial expression. The face converter module is responsible for analyzing the features of 2D facial landmarks and reference images, integrating this information into the generation process to ensure that the generated facial movements are consistent with the input audio and reference images; the remaining two networks are called "low-resolution rotator" and "editor", which together constitute a module called "body rotator", whose responsibility is to calculate the overall image effect such as rotating the head and body with non-expression parameters based on the posture and appearance flow graph, and calculate a low-resolution image feature, such as Fig. 9 As shown, they are the original image, the alpha channel image, the extracted skeleton image, the mapped key point image, the converted action 1, and the converted action 2.

[0068] S2.3, the output of the posture converted image and appearance flow graph is enlarged and passed to the editor to improve the image quality, and finally returned to the user. The editor is also responsible for optimizing the details of the generated facial movements, such as the accuracy of lip synchronization and the naturalness of facial expressions, so that the final animation is more realistic and expressive. The specific calculation method is as follows:

[0069] 1) The image after the facial image feature points are transformed by perspective projection is I direct ;

[0070] 2) The image of the torso after the appearance flow graph changes is I wrapper ;

[0071] 3) The Mask of the Alpha channel is I alpha ;

[0072] The loss function is:

[0073] L total =∑(I wrapper -I wrapper ’ )+∑(I direct -I direct ’ );

[0074] The final image after processing through the body rotator and editor is:

[0075] I total (x,y)=λI alpha I direct (x,y)+(1-λ)I alpha I wrapper (x,y).

[0076] The present invention is mainly divided into two stages:

[0077] The first stage focuses on feature extraction of audio and constructing facial feature sequences and posture sequences. Audio input usually contains rich information, such as the rhythm, intensity and intonation of speech, which is crucial for accurately predicting and generating head and body postures. In this stage, key features are extracted from the audio using a specific model structure, and a detailed posture sequence is generated based on these features. This posture sequence contains the head posture and body posture information required for each frame of animation, laying a solid foundation for subsequent video generation steps. It can complete real-time perception of user emotions, give emotional responses, improve the quality of conversations, and better meet the emotional needs of users.

[0078] The second stage is to accurately map the facial features and posture information generated in the first stage to the character features on any given two-dimensional character image. Through this step, the two-dimensional character image can be quickly and refined, so that the originally static image can show rich expressions and movements in real time according to the audio input. This not only enhances the expressiveness of the virtual character, but also provides more innovative possibilities for virtual game characters, online education and other fields. It can adapt to any image and can achieve feature matching without the need to independently establish a character skeleton model to achieve real-time driving of different emotional expressions of the character image.

[0079] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0080] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An efficient method for driving expressions and actions in two-dimensional images based on audio. It is characterized in that The following steps are involved: S1, extract audio data and construct facial feature sequence and posture sequence; S1.1, audio extraction features; S1.2, convert the audio extracted features into 3D facial mesh to obtain the facial feature sequence; S1.3, extract posture from audio and obtain posture sequence; S1.3, build a material library and introduce reference information; S2, converting the facial feature sequence and posture sequence in S1 into input, and then driving the two-dimensional image photo to complete the two-dimensional image action sequence; S2.1, input the picture, obtain the second dimension face feature points through face recognition and feature extraction; S2.2, input the facial feature sequence and pose sequence calculated in S1, after obtaining the mesh and pose sequences, convert them into a 2D facial landmarks sequence to obtain a 45-dimensional pose vector, and then drive the pose vector through the two-dimensional expression system to generate a new image with a new pose; S2.3, the posture-converted image and appearance flow graph are output, enlarged, and passed to the editor to improve the image quality, and finally returned to the user.

2. According to claim 1, an efficient method for driving two-dimensional image expressions and actions based on audio, Features: In step S1, a transformer-based model is used to extract a 3D facial mesh and head pose sequence from the audio input and project it into a 2D facial landmark.

3. According to claim 1, an efficient method for driving two-dimensional image expressions and actions based on audio, Features: In step S1.1, the pre-trained wav2vec 2.0 is used to extract the audio feature H 1:T =(h 1 , ..., h T ), and perform corresponding emotion recognition to obtain the emotional feature C 1:T =(c 1 , ..., c T ), the audio features and emotional features are encoded by the encoder to obtain the new audio features A 1:T , the specific calculation method is as follows: A 1:T =∑(αH 1:T βC 1:T )。 4. The efficient method for driving two-dimensional image expressions and actions based on audio according to claim 1, Features: In step S1.2, the extracted audio feature A 1:T =(a 1 , ..., a T ) through a 34-layer ResNet, followed by a single architecture consisting of two fc layers to convert the extracted audio features into a 3D facial mesh M 1:T =(m 1 , ..., m T ), the training loss is L1loss, and the specific calculation method is as follows:

5. The efficient method for driving two-dimensional image expressions and actions based on audio according to claim 1, Features: In step S1.3, the extracted audio feature A 1:T =(a 1 , ..., a T ) to infer the action P of the head posture 1:T =(p 1 , ..., p T ), build an independent 34-layer ResNet and use transformer as decoder to decode the pose sequence. Using transformer will increase the generation speed; The audio features are integrated into the decoder through the cross-attention mechanism. The improved model structure consists of three parts: a shared encoder, a CTC decoder, and an attention decoder. The shared encoder is an independent 34-layer ResNet. The CTC decoder is a fully connected layer and a softmax layer. The attention decoder contains multiple transformer layers. The loss function formula is: THE combin (x,y)=λL ctc (x,y)+(1-λ)L aed (x,y)。 6. The efficient method for driving two-dimensional image expressions and actions based on audio according to claim 1, Features: In step S2.1, the input image is a "half-length photo" of a humanoid cartoon character and is processed into a 512×512 image. The character image can be obtained through image generation technology or can be a copyrighted self-drawn image.

7. The efficient method for driving two-dimensional image expressions and actions based on audio according to claim 1, Features: In the step S2.2, the facial feature sequence and the pose sequence calculated in S1 are inputted, and after obtaining the mesh and pose sequences, they are converted into a 2D facial landmarks sequence by perspective projection to obtain a 45-dimensional pose vector. Among the 45 parameters, 39 are used to control the expression of the character, and 6 are used to control the rotation of the face and body. Among them, the system consists of 5 neural networks and can be divided into two modules. Three networks constitute the face converter module, and the remaining two networks constitute the "low-resolution rotator" and "editor" respectively.

8. According to claim 7, an efficient method for driving two-dimensional image expressions and actions based on audio, Features: The face converter module is responsible for changing the facial expressions of the character. The face converter module is responsible for analyzing the features of 2D facial landmarks and reference images, integrating this information into the generation process, and ensuring that the generated facial movements are consistent with the input audio and reference images.

9. The efficient method for driving two-dimensional image expressions and actions based on audio according to claim 7, Features: The low-resolution rotator and editor together constitute the body rotator module, which is responsible for calculating the overall image effect such as rotating the head and torso with non-expression parameters based on the posture and appearance flow graph. Compute a low-resolution image feature.

10. The efficient method for driving two-dimensional image expressions and actions based on audio according to claim 9, Features: The editor is also responsible for optimizing the details of the generated facial movements, such as the accuracy of lip sync and the naturalness of facial expressions, making the final animation more realistic and expressive; Among them, the image after the facial image feature points are changed by perspective projection is I direct , the image of the torso after the appearance flow graph changes is I wrapper , the Mask of the Alpha channel is I alpha ; The loss function is: L total =∑(I wrapper -I wrapper ’)+∑(I direct -I direct ’); The final image after processing through the body rotator and editor is: I total (x,y)=λI alpha I direct (x,y)+(1-λ)I alpha I wrapper (x,y)。

Citation Information

Cited By

  • Picture digital human generation method based on deep learning

    CN120070683A