Character animation generation method and device based on voice driving, equipment and medium

Abstract

The embodiment of the invention provides a character animation generation method and device based on voice driving, equipment and a medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: acquiring an audio signal input by a user, a target human portrait picture and a target scene reference video; encoding the audio signal to obtain an audio code; generating a three-dimensional face model motion coefficient according to the portrait picture of the target person and the audio code; extracting blink information and head posture information from the target scene reference video; correcting the motion coefficient of the three-dimensional face model according to the blinking information and the head posture information to obtain a corrected motion coefficient of the three-dimensional face model; and taking a three-dimensional deformable head model as a potential vector, mapping the three-dimensional deformable head model to the corrected three-dimensional face model motion coefficient frame by frame, converting the three-dimensional face model motion coefficient into a video frame, and generating a head speech video of the target person. According to the embodiment of the invention, the quality and reality of character animation generation driven by voice can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of financial technology (Fintech), and in particular to a method and apparatus, device and medium for generating character animation based on voice drive. Background Art

[0002] With the rapid development of science and technology, the field of voice-driven character animation generation faces a series of problems and challenges. Existing methods mainly rely on single audio drive or single video drive to generate head speech videos. However, existing methods have significant deficiencies in achieving naturalness and realism, and existing methods do not fully consider the audio content and audio scenes, which leads to the generated video and audio are not fully aligned. The audience may feel that there is inconsistency and unnaturalness between the video and audio, resulting in the generated animation often lacking naturalness, and the overall look and feel appears stiff and unreal. In financial scenarios, digital human videos with low realism will affect customer perception and reduce customer trust. Based on this, how to improve the quality and realism of voice-driven character animation generation has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method and apparatus, device and medium for generating character animation based on voice-driven operation, which can generate character animation with coordinated video and audio, have high naturalness and good realism, and can realize customized generation of target character head speech video according to specific needs, so as to meet the specific requirements of different scenes and speakers, thereby improving the quality and realism of voice-driven character animation generation. In financial scenarios, generating high-quality target character head speech video can enhance customer perception, improve customer trust, help improve the quality of financial services, and thus enhance the competitiveness of financial enterprises.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for generating character animation based on voice drive, the method comprising:

[0005] Obtaining the audio signal input by the user, the target person portrait image and the target scene reference video;

[0006] Encoding the audio signal to obtain audio code;

[0007] Generate a three-dimensional face model motion coefficient according to the target person portrait picture and the audio code;

[0008] Extracting blink information and head posture information from the target scene reference video;

[0009] Correcting the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain a corrected motion coefficient of the three-dimensional face model;

[0010] The 3D deformable head model is used as a latent vector and mapped to the corrected 3D face model motion coefficients frame by frame.

[0011] The corrected 3D face model motion coefficients are converted into video frames to generate a target person's head speaking video.

[0012] In some embodiments, the motion coefficients of the three-dimensional face model include identity features, expression coefficients, and head posture, wherein the head posture includes head rotation and head translation.

[0013] In some embodiments, the step of correcting the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model includes:

[0014] Extracting expression coefficients of T×64 dimensions from the motion coefficients of the three-dimensional face model of the target person portrait image;

[0015] Inputting the expression coefficient of the T×64 dimension and the motion coefficient of the three-dimensional face model extracted frame by frame from the target scene reference video into a pre-trained first multi-layer perceptron to map to a potential expression vector of the T×64 dimension;

[0016] Merging the potential expression vector of the T×64 dimension and the motion vector of the target scene reference video to obtain a merged vector;

[0017] The merged vector and the three-dimensional face model motion coefficient of the target person portrait image are input into a pre-trained second multi-layer perceptron and mapped into a corrected three-dimensional face model motion coefficient of T×73 dimensions, where T is the total number of frames of the target scene reference video.

[0018] In some embodiments, extracting the expression coefficient of T×64 dimension from the motion coefficient of the three-dimensional face model of the target person portrait picture includes:

[0019] Extracting a first three-dimensional face model motion coefficient from the target person portrait picture;

[0020] Expanding the motion coefficient of the first three-dimensional face model to a total number of frames T;

[0021] Extract the expression coefficients of T×64 dimensions from the first three-dimensional face model motion coefficients.

[0022] In some embodiments, the first multi-layer perceptron is obtained by separate training using mean square error, and the second multi-layer perceptron uses a pre-trained weight network in the process of generating motion coefficients of a three-dimensional face model.

[0023] In some embodiments, the three-dimensional deformable head model is used as a potential vector and mapped frame by frame to the corrected three-dimensional face model motion coefficient, including:

[0024] The corrected 3D face model motion coefficients are input into the mapping network and mapped to the coefficients of the unsupervised 3D key point space;

[0025] The 3D deformable head model is used as the coefficients of the latent vector mapped to the unsupervised 3D keypoint space frame by frame;

[0026] The facial expression and head movement of the three-dimensional deformable head model are controlled by coefficients in an unsupervised 3D keypoint space.

[0027] In some embodiments, the method further comprises:

[0028] Inputting the video frame into an optical flow generator to generate a head posture animation of the target person;

[0029] A head speech video of the target person is obtained according to the head posture animation of the target person.

[0030] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a character animation generation device based on voice drive, the device comprising:

[0031] An acquisition module is used to acquire an audio signal input by a user, a target person portrait image, and a target scene reference video;

[0032] An encoding module, used for encoding the audio signal to obtain an audio code;

[0033] A generating module, used for generating motion coefficients of a three-dimensional face model according to the target person portrait picture and the audio coding;

[0034] An extraction module, used for extracting blink information and head posture information from the target scene reference video;

[0035] A correction module, used for correcting the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model;

[0036] A mapping module, used to map the three-dimensional deformable head model as a latent vector to the corrected three-dimensional face model motion coefficient frame by frame;

[0037] The conversion module is used to convert the corrected three-dimensional face model motion coefficients into video frames to generate a target person's head speech video.

[0038] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0039] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0040] The method, device, equipment and medium for generating character animation based on voice-driven driving proposed in the present application obtain the audio signal input by the user, the portrait picture of the target character and the reference video of the target scene; encode the audio signal to obtain the audio code; generate the motion coefficient of the three-dimensional face model according to the portrait picture of the target character and the audio code; extract the blink information and the head posture information from the reference video of the target scene; correct the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model; use the three-dimensional deformable head model as a potential vector, and map it frame by frame to the corrected motion coefficient of the three-dimensional face model; convert the corrected motion coefficient of the three-dimensional face model into a video frame to generate the head speech video of the target character. By adding reference videos of different scene types, the head posture information and blink information of the speaker videos of different styles are customized, and the motion coefficient of the three-dimensional face model is corrected to ensure that the generated head state of the character is consistent with the voice content of the audio, and the coordination between the generated video and the audio is guaranteed. Based on this, the embodiments of the present application can generate character animations that are coordinated between video and audio, have high naturalness and good realism, and can realize customized generation of target character head speaking videos according to specific needs, meeting the specific requirements of different scenarios and speakers, thereby improving the quality and realism of voice-driven character animation generation. In financial scenarios, generating high-quality target character head speaking videos can enhance customer perception, increase customer trust, and help improve the quality of financial services, thereby enhancing the competitiveness of financial enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a method for generating character animation based on voice drive provided in an embodiment of the present application;

[0042] Figure 2 yes Figure 1 Flow chart of step S105 in FIG.

[0043] Figure 3 This is a schematic diagram of the 3DMM motion coefficient correction process;

[0044] Figure 4 yes Figure 2 Flow chart of step S201 in FIG.

[0045] Figure 5 yes Figure 1 Flow chart of step S106 in FIG.

[0046] Figure 6 is a flowchart from step S601 to step S602;

[0047] Figure 7 is a flow chart of character animation generation based on voice drive provided by another embodiment of the present application;

[0048] Figure 8 It is a structural schematic diagram of a voice-driven character animation generation device provided in an embodiment of the present application;

[0049] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] First, some nouns involved in this application are analyzed:

[0054] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0055] 3DMM (3D Morphable Model): is a statistical model of face shape and appearance, i.e., a deformable face model. First, multiple sets of face 3D data are scanned together with high precision and aligned. Then, PCA is used to obtain a lower-dimensional subspace from these three-dimensional shape and color data. Variability is reflected in the ability to perform combined deformations in these PCA subspaces, transfer the characteristics of one face to another face, or generate a new face. The core idea of ​​the 3DMM model is that faces can be matched one by one in three-dimensional space, and can be obtained by weighted linear addition of many orthogonal bases of faces. This model is widely used in the industry in many fields such as face recognition, expression capture, and beauty. By fitting and projecting a 3D point cloud on the face, the function of restoring the three-dimensional shape from the two-dimensional face image is realized.

[0056] Optical flow generator: involves a two-dimensional vector field that describes the trajectory and speed of the movement of pixels over time in consecutive frame images. It is used to capture the movement of objects in the image and the displacement information between adjacent frames. The basic assumption of optical flow is that the pixels between adjacent frames are in continuous motion in a short period of time. By analyzing the changes in pixel brightness in adjacent frames, the optical flow algorithm can estimate the movement speed and direction of each pixel. In computer vision and image processing, optical flow is widely used in many fields, such as motion tracking, video compression, scene analysis, 3D reconstruction, etc. Optical flow can be used to infer the trajectory of moving objects, detect moving objects, and analyze the motion state of cameras or objects.

[0057] Audio-driven talking head video: is a technique that allows the movement of the head and mouth to be driven by an audio signal, generating a synthetic video in which the input facial image speaks in a realistic and coherent manner. The main goal of this technique is to generate a synthetic video for a given speech audio clip and a single facial image of an arbitrary individual, where the facial image moves as the audio is played, simulating a real speaking scene. The generated video should have a high degree of realism and naturalness, making it difficult for viewers to distinguish between real and synthetic. The implementation of this technique usually involves complex algorithms and model training, which aims to extract key information from the audio signal and map it to the movement of the face, including lip synchronization and changes in head posture. In this way, a highly realistic speaking face video can be generated, where the face moves as the audio is played, simulating the real speaking process. This technology is not only widely used in entertainment and film and television production, but also shows great potential in the fields of virtual character interaction and online education.

[0058] Multilayer Perceptron (MLP): It is a feedforward artificial neural network model that introduces one or more hidden layers based on a single-layer neural network, so that the neural network has multiple network layers. MLP transforms the output of the hidden layer through an activation function, which enables it to process and represent more complex nonlinear relationships.

[0059] With the rapid development of science and technology, the field of voice-driven character animation generation faces a series of problems and challenges. Existing methods mainly rely on single audio drive or single video drive to generate head speech videos. However, existing methods have significant deficiencies in achieving naturalness and realism, and existing methods do not fully consider the audio content and audio scenes, which leads to the generated video and audio are not fully aligned. The audience may feel that there is inconsistency and unnaturalness between the video and audio, resulting in the generated animation often lacking naturalness, and the overall look and feel appears stiff and unreal. In financial scenarios, digital human videos with low realism will affect customer perception and reduce customer trust. Based on this, how to improve the quality and realism of voice-driven character animation generation has become a technical problem that needs to be solved urgently.

[0060] Based on this, the embodiment of the present application provides a method and device for generating character animation based on voice drive, equipment and medium, which obtains the audio signal input by the user, the portrait picture of the target character and the reference video of the target scene; encodes the audio signal to obtain the audio code; generates the motion coefficient of the three-dimensional face model according to the portrait picture of the target character and the audio code; extracts the blink information and the head posture information from the reference video of the target scene; corrects the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model; uses the three-dimensional deformable head model as a potential vector, and maps it to the corrected motion coefficient of the three-dimensional face model frame by frame; converts the corrected motion coefficient of the three-dimensional face model into a video frame to generate the head speech video of the target character. By adding reference videos of different scene types, the head posture information and blink information of the speaker videos of different styles are customized, and the motion coefficient of the three-dimensional face model is corrected to ensure that the generated head state of the character is consistent with the voice content of the audio, and the coordination between the generated video and the audio is guaranteed. Based on this, the embodiments of the present application can generate character animations that are coordinated between video and audio, have high naturalness and good realism, and can realize customized generation of target character head speaking videos according to specific needs, meeting the specific requirements of different scenarios and speakers, thereby improving the quality and realism of voice-driven character animation generation. In financial scenarios, generating high-quality target character head speaking videos can enhance customer perception, increase customer trust, and help improve the quality of financial services, thereby enhancing the competitiveness of financial enterprises.

[0061] The voice-driven character animation generation method, device, equipment and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the voice-driven character animation generation method in the embodiments of the present application is described.

[0062] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0064] The voice-driven character animation generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The voice-driven character animation generation method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a voice-driven character animation generation method, etc., but is not limited to the above forms.

[0065] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0066] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0067] Figure 1 is an optional flow chart of the method for generating character animation based on voice drive provided in an embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S107.

[0068] Step S101, obtaining an audio signal input by a user, a target person portrait image, and a target scene reference video;

[0069] Step S102, encoding the audio signal to obtain audio code;

[0070] Step S103, generating motion coefficients of a 3D face model according to the target person portrait image and audio coding;

[0071] Step S104, extracting blink information and head posture information from the target scene reference video;

[0072] Step S105, correcting the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model;

[0073] Step S106, using the three-dimensional deformable head model as a potential vector, and mapping it to the corrected three-dimensional face model motion coefficient frame by frame;

[0074] Step S107, converting the corrected three-dimensional face model motion coefficients into video frames to generate a target person's head speaking video.

[0075] In step S101 of some embodiments, an audio signal input by the user, a portrait image of a target person, and a reference video of a target scene are obtained. The audio signal may be a voice input by the user. The portrait image of the target person is a portrait image of the target person that the user wants to generate. The reference video of the target scene is a reference video of the target person in the target scene that the user wants to generate. In a financial scenario, taking the introduction of financial products as an example, by obtaining an input of a pre-recorded voice introducing a financial product, and obtaining a portrait image of the spokesperson of the financial product and a reference video of the spokesperson in a financial business scenario, a head speech video introducing the financial product with the facial image of the spokesperson in a financial business scenario can be synthesized later, thereby improving customer trust.

[0076] In step S102 of some embodiments, the audio signal is encoded to obtain an audio code. The input audio signal is encoded and the obtained audio code is used as an input of a subsequent generation process to ensure that the voice features of the speaker can be effectively extracted from the audio signal.

[0077] In step S103 of some embodiments, a 3D face model motion coefficient is generated based on the target person portrait image and audio coding. The target person portrait image is input, and based on the audio coding, a real 3D face model motion coefficient, such as a 3DMM motion coefficient, is generated. The 3D face model motion coefficient includes but is not limited to different styles of head movement postures, expression changes, lip movements, and blinking. Taking the face 3DMM as an example, the face 3DMM expression is as follows:

[0078]

[0079] in, is the average shape of a 3D face, U id and U exp are the orthogonal basis of the identity and expression of the deformable model respectively. Identity feature coefficient Used to describe a person's identity, expression coefficient It is used to describe the facial expression of a person. In order to ensure the consistency of identity, only the expression coefficient β is used here, and the head posture parameter ρ = [r, t] is added, where r∈SO(3) and represents the head rotation and translation. This step ensures that the expression and head pose motion of each frame are generated separately to reduce uncertainty.

[0080] In step S104 of some embodiments, the blink information and head posture information are extracted from the target scene reference video. According to the reference videos of different scene types, the head posture information and blink information of the speaker videos of different styles can be customized, and the motion coefficients of the 3D face model can be corrected accordingly.

[0081] In step S105 of some embodiments, the motion coefficient of the 3D face model is corrected according to the blink information and the head posture information to obtain the corrected motion coefficient of the 3D face model. Correcting the motion coefficient of the 3D face model according to the blink information and the head posture information can ensure that the generated head state of the character is consistent with the voice content of the audio, and ensure the coordination between the generated video and the audio.

[0082] In step S106 of some embodiments, the three-dimensional deformable head model is used as a latent vector and mapped to the corrected three-dimensional face model motion coefficient frame by frame. For example, the head 3DMM is used as a latent vector and mapped to the motion coefficient in the 3D space frame by frame. The facial expression and head movement are controlled by unsupervised autonomous recognition of 3D key points, which ensures that the generated head state of the character is consistent with the speech content of the audio.

[0083] In step S107 of some embodiments, the corrected three-dimensional face model motion coefficients are converted into video frames to generate a head speech video of the target person. In the audio-driven head speech video generation method based on reference video correction, the coordinated processing of audio and video data ensures the coordination between the generated video and audio, so that customized generation can be achieved according to specific needs to meet the specific requirements of different scenes and speakers. By introducing reference videos of different scene types, the uncertainty and randomness of the generated character posture are significantly reduced, thereby improving the stability and consistency of the generated results. Since the audio and video data are comprehensively considered, the coordination between the generated video and audio is ensured through coordinated processing. Therefore, the correction of the reference video can provide higher customization, authenticity, stability, efficiency and coordination for the audio-driven head speech video generation, and provide a more outstanding generation solution for a variety of scenes and needs.

[0084] Steps S101 to S107 shown in the embodiment of the present application are to obtain the audio signal input by the user, the portrait image of the target person and the reference video of the target scene; encode the audio signal to obtain the audio code; generate the motion coefficient of the three-dimensional face model according to the portrait image of the target person and the audio code; extract the blink information and the head posture information from the reference video of the target scene; correct the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model; use the three-dimensional deformable head model as a potential vector, and map it to the corrected motion coefficient of the three-dimensional face model frame by frame; convert the corrected motion coefficient of the three-dimensional face model into a video frame to generate the head speech video of the target person. By adding reference videos of different scene types, the head posture information and blink information of the speaker videos of different styles are customized, and the motion coefficient of the three-dimensional face model is corrected to ensure that the generated head state of the person is consistent with the voice content of the audio, and the coordination between the generated video and the audio is guaranteed. Based on this, the embodiments of the present application can generate character animations that are coordinated between video and audio, have high naturalness and good realism, and can realize customized generation of target character head speaking videos according to specific needs, meeting the specific requirements of different scenarios and speakers, thereby improving the quality and realism of voice-driven character animation generation. In financial scenarios, generating high-quality target character head speaking videos can enhance customer perception, increase customer trust, and help improve the quality of financial services, thereby enhancing the competitiveness of financial enterprises.

[0085] See also Figure 2 In some embodiments, step S105 may include but is not limited to steps S201 to S204:

[0086] Step S201, extracting expression coefficients of T×64 dimensions from the motion coefficients of the three-dimensional face model of the target person portrait image;

[0087] Step S202, inputting the expression coefficients of T×64 dimensions and the motion coefficients of the three-dimensional face model extracted frame by frame from the target scene reference video into a pre-trained first multi-layer perceptron to map them to a potential expression vector of T×64 dimensions;

[0088] Step S203, merging the potential expression vector of T×64 dimension and the motion vector of the target scene reference video to obtain a merged vector;

[0089] Step S204, input the merged vector and the 3D face model motion coefficient of the target person portrait image into a pre-trained second multi-layer perceptron, and map them into corrected 3D face model motion coefficients of T×73 dimensions, where T is the total number of frames of the target scene reference video.

[0090] In some embodiments, Figure 3 As shown, the 3DMM motion coefficients are extracted from the input portrait image A, and the dimension is expanded to T (total number of frames). The 3DMM motion coefficients are extracted frame by frame from the input reference video B, and the T×64 dimension expression coefficient A is taken. 3de and B 3de The input is mapped to the potential expression vector by the first multi-layer perceptron MLP_1. This step is to ensure that the facial expression and mouth shape of the character are kept as much as possible in the original picture when the reference video is introduced. The potential expression vector of T×64 dimension is compared with the motion vector B of the reference video. 3dp The 3DMM motion coefficients are combined and input into the second multi-layer perceptron MLP_2 together with the 3DMM motion coefficients generated according to the portrait image A, and finally mapped into the corrected 3DMM motion coefficients of T×73 dimensions. The embodiment of the present application allows the influence of the reference video on the generated result to be freely controlled, further improving the quality of the animation.

[0091] In some embodiments, the first multi-layer perceptron MLP_1 can be trained separately using the mean square error, and the second multi-layer perceptron MLP_2 uses the pre-trained weight network in the process of generating the face 3DMM motion coefficients.

[0092] In some embodiments, the head 3DMM can be used as a latent vector and mapped to motion coefficients in 3D space frame by frame. The facial expressions and head movements are controlled by unsupervised autonomous recognition of 3D key points, which ensures that the generated head state of the character is consistent with the speech content of the audio.

[0093] See also Figure 4 In some embodiments, step S201 may include but is not limited to steps S401 to S403:

[0094] Step S401, extracting a first three-dimensional face model motion coefficient from a target person portrait image;

[0095] Step S402, expanding the motion coefficients of the first 3D face model to a total number of frames T;

[0096] Step S403: extracting expression coefficients of T×64 dimensions from the motion coefficients of the first three-dimensional face model.

[0097] In some embodiments, the 3DMM motion coefficients are extracted from the input portrait image A, and the dimension is expanded to the total number of frames T, and the 3DMM motion coefficients are extracted frame by frame from the input reference video B, and the expression coefficients A of the T×64 dimension are taken. 3de and B 3de The input is mapped to the potential expression vector by the first multi-layer perceptron MLP_1. This step is to ensure that the facial expression and mouth shape of the character remain as close to the original image as possible when the reference video is introduced.

[0098] See also Figure 5 In some embodiments, step S106 may include but is not limited to steps S501 to S503:

[0099] Step S501, inputting the corrected 3D face model motion coefficients into a mapping network, and mapping them to coefficients of an unsupervised 3D key point space;

[0100] Step S502, mapping the three-dimensional deformable head model as a latent vector to the coefficients of the unsupervised 3D key point space frame by frame;

[0101] Step S503, controlling the facial expression and head movement of the three-dimensional deformable head model through the coefficients of the unsupervised 3D key point space.

[0102] In some embodiments, explicit 3DMM motion coefficients (such as facial expressions and head movements) are mapped to coefficients of an unsupervised 3D key point space, and the relationship between the explicit 3DMM motion coefficients and the unsupervised 3D key point space is learned, so that facial rendering can be driven by the 3DMM coefficients. Facial expressions and head movements are controlled by unsupervised autonomous recognition of 3D key points, which ensures that the generated head state of the character is consistent with the speech content of the audio.

[0103] See also Figure 6 In some embodiments, the method for generating character animation based on voice-driven operation in the embodiment of the present application includes but is not limited to steps S601 to S602:

[0104] Step S601, inputting the video frame into an optical flow generator to generate a head posture animation of the target person;

[0105] Step S602, obtaining a head speech video of the target person according to the head posture animation of the target person.

[0106] In some embodiments, the corrected 3DMM coefficients are converted into video frames and input into an optical flow generator to generate a head posture animation of the target character. The optical flow generator is responsible for generating animations that only contain head postures, which ensures the naturalness and coordination of the animation. The optical flow generator consists of an optical flow estimation and a super-resolution network. First, the flow is estimated for adjacent frames of the input video, the alignment relationship between frames is learned, and then super-resolution is performed to help the super-resolution network generate details, which not only utilizes the timing information between frames, but also plays a role in aligning adjacent frames, so that the continuity between frames is guaranteed. The embodiment of the present application introduces an optical flow generator to further enhance the realism of the animation.

[0107] The following is a further explanation of the voice-driven character animation generation method of the present application in conjunction with the accompanying drawings and specific embodiments.

[0108] like Figure 7 As shown, Figure 7 This is a flow chart of the method for generating character animation based on voice drive. The specific implementation steps are as follows:

[0109] ① First, the input audio is encoded and used as the input for the subsequent generation process. This step ensures that the speaker's voice features can be effectively extracted from the audio.

[0110] ② Input a specific person portrait image, and based on audio coding, we generate the real face 3DMM (three-dimensional shape and texture model) motion coefficients, including different styles of head movement postures, expression changes, lip movements and blinking, etc. The face 3DMM expression is as follows:

[0111]

[0112] in, is the average shape of a 3D face, U id and U exp are orthogonal bases for the identity and expression of the deformable model, respectively. The coefficients Used to describe a person's identity, coefficient It is used to describe the facial expression of a person. In order to ensure the consistency of identity, only the expression coefficient β is used here, and the motion parameter ρ = [r, t] is added, where r∈SO(3) and represents the head rotation and translation. This step ensures that the expression and head pose motion of each frame are generated separately to reduce uncertainty.

[0113] ③Add reference videos to extract blinking and head posture information to correct the generated parameters. The specific process is as follows Figure 3 As shown. The MLP_1 module is trained separately using the mean square error, while MLP_2 uses the pre-trained weight network in the process of generating the 3DMM motion coefficients of the face. The 3DMM motion coefficients are extracted from the input portrait image A, and the dimension is expanded to the T (total number of frames) dimension. The 3DMM motion coefficients are extracted frame by frame from the input reference video B, and the T×64-dimensional expression coefficients A3de and B3de are input into MLP_1 to map to the potential expression vector. This step is to ensure that the facial expression and mouth shape of the character are kept as much as possible in the original image when the reference video is introduced. The T×64-dimensional potential expression vector is merged with the motion vector B3dp of the reference video, and is input into MLP_2 together with the 3DMM motion coefficient generated according to the portrait image A, and finally mapped to the corrected T×73-dimensional 3DMM motion coefficient. This allows the influence of the reference video on the generated results to be freely controlled, further improving the quality of the animation.

[0114] ④ The head 3DMM is used as a latent vector and mapped to the motion coefficient in 3D space frame by frame. The facial expression and head movement are controlled by unsupervised autonomous recognition of 3D key points, which ensures that the generated head state of the character is consistent with the speech content of the audio.

[0115] ⑤An optical flow generator can also be introduced to further enhance the realism of the animation. The optical flow generator is responsible for generating animations that only contain head postures, which ensures the naturalness and coordination of the animation.

[0116] The voice-driven character animation generation method proposed in the embodiment of the present application can customize the head movement posture and blinking of the speaker video of different styles by adding reference videos of different scene types. The coordination between the generated video and audio is ensured by co-processing audio and video data, so that customized generation can be achieved according to specific needs to meet the specific requirements of different scenes and speakers. The method supports the introduction of reference videos of different scene types, significantly reduces the uncertainty and randomness of the generated posture, thereby improving the stability and consistency of the generated results. The method comprehensively considers the audio and video data, and ensures the coordination between the generated video and audio through co-processing. The method can provide higher customization, authenticity, stability, efficiency and synergy for audio-driven head speech video generation through the correction of reference videos, and provides a more outstanding generation solution for a variety of scenes and needs. Based on this, the voice-driven character animation generation method of the present application can improve the quality and realism of character animation generation.

[0117] See also Figure 8The embodiment of the present application further provides a voice-driven character animation generation device, which can implement the voice-driven character animation generation method, and the device includes:

[0118] An acquisition module 810 is used to acquire an audio signal input by a user, a target person portrait image, and a target scene reference video;

[0119] The encoding module 820 is used to encode the audio signal to obtain audio code;

[0120] A generating module 830 is used to generate motion coefficients of a 3D face model according to a target person portrait image and audio coding;

[0121] An extraction module 840 is used to extract blink information and head posture information from a target scene reference video;

[0122] A correction module 850 is used to correct the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model;

[0123] A mapping module 860, for mapping the three-dimensional deformable head model as a potential vector to the corrected three-dimensional face model motion coefficients frame by frame;

[0124] The conversion module 870 is used to convert the corrected three-dimensional face model motion coefficients into video frames to generate a target person's head speaking video.

[0125] In some embodiments of the present application, the acquisition module 810 acquires an audio signal input by a user, a portrait image of a target person, and a reference video of a target scene; the encoding module 820 encodes the audio signal to obtain an audio code; the generation module 830 generates a three-dimensional face model motion coefficient based on the portrait image of the target person and the audio code; the extraction module 840 extracts blink information and head posture information from the reference video of the target scene; the correction module 850 corrects the three-dimensional face model motion coefficient based on the blink information and head posture information to obtain the corrected three-dimensional face model motion coefficient; the mapping module 860 uses the three-dimensional deformable head model as a latent vector and maps it frame by frame to the corrected three-dimensional face model motion coefficient; the conversion module 870 converts the corrected three-dimensional face model motion coefficient into a video frame to generate a head speaking video of the target person.

[0126] In some embodiments of the present application, an audio signal input by a user, a portrait image of a target person, and a reference video of a target scene are obtained. The audio signal may be a voice input by the user. The portrait image of the target person is a portrait image of the target person that the user wants to generate. The reference video of the target scene is a reference video of the target person in the target scene that the user wants to generate. In a financial scenario, taking the introduction of financial products as an example, by obtaining an input of a pre-recorded voice introducing a financial product, and obtaining a portrait image of the spokesperson of the financial product and a reference video of the spokesperson in a financial business scenario, it is possible to subsequently synthesize a head speech video introducing the financial product with the facial image of the spokesperson in a financial business scenario, thereby improving customer trust.

[0127] In some embodiments of the present application, the audio signal is encoded to obtain the audio code. The input audio signal is encoded and the obtained audio code is used as the input of the subsequent generation process to ensure that the voice features of the speaker can be effectively extracted from the audio signal.

[0128] In some embodiments of the present application, a 3D face model motion coefficient is generated based on a target person portrait image and audio coding. A target person portrait image is input, and based on the audio coding, a real 3D face model motion coefficient, such as a 3DMM motion coefficient, is generated. The 3D face model motion coefficient includes but is not limited to different styles of head movement postures, expression changes, lip movements, and blinking. Taking the face 3DMM as an example, the face 3DMM expression is as follows:

[0129]

[0130] in, is the average shape of a 3D face, U id and U exp are the orthogonal basis of the identity and expression of the deformable model respectively. Identity feature coefficient Used to describe a person's identity, expression coefficient It is used to describe the facial expression of a person. In order to ensure the consistency of identity, only the expression coefficient β is used here, and the head posture parameter ρ = [r, t] is added, where r∈SO(3) and represents the head rotation and translation. This step ensures that the expression and head pose motion of each frame are generated separately to reduce uncertainty.

[0131] In some embodiments of the present application, the blink information and head posture information are extracted from the target scene reference video. According to the reference videos of different scene types, the head posture information and blink information of the speaker videos of different styles can be customized, and the motion coefficients of the three-dimensional face model can be corrected accordingly.

[0132] In some embodiments of the present application, the motion coefficient of the three-dimensional face model is corrected according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model. Correcting the motion coefficient of the three-dimensional face model according to the blink information and the head posture information can ensure that the generated head state of the character is consistent with the voice content of the audio, and ensure the coordination between the generated video and the audio.

[0133] In some embodiments of the present application, a three-dimensional deformable head model is used as a latent vector and mapped frame by frame to the corrected three-dimensional face model motion coefficient. For example, the head 3DMM is used as a latent vector and mapped frame by frame to the motion coefficient in the 3D space. The facial expression and head movement are controlled by unsupervised autonomous recognition of 3D key points, which ensures that the generated head state of the character is consistent with the speech content of the audio.

[0134] In some embodiments of the present application, the corrected three-dimensional face model motion coefficients are converted into video frames to generate a head speech video of the target person. Based on the audio-driven head speech video generation method corrected by the reference video, the coordinated processing of audio and video data ensures the coordination between the generated video and audio, so that customized generation can be achieved according to specific needs to meet the specific requirements of different scenes and speakers. By introducing reference videos of different scene types, the uncertainty and randomness of the generated character posture are significantly reduced, thereby improving the stability and consistency of the generated results. Since the audio and video data are comprehensively considered, the coordination between the generated video and audio is ensured through collaborative processing. Therefore, it is possible to provide higher customization, authenticity, stability, efficiency and coordination for audio-driven head speech video generation through the correction of the reference video, and provide a more outstanding generation solution for a variety of scenes and needs.

[0135] Based on this, in the voice-driven character animation generation device of the embodiment of the present application, the acquisition module 810 acquires the audio signal input by the user, the target character portrait picture and the target scene reference video; the encoding module 820 encodes the audio signal to obtain the audio code; the generation module 830 generates the three-dimensional face model motion coefficient according to the target character portrait picture and the audio code; the extraction module 840 extracts the blink information and the head posture information from the target scene reference video; the correction module 850 corrects the three-dimensional face model motion coefficient according to the blink information and the head posture information to obtain the corrected three-dimensional face model motion coefficient; the mapping module 860 uses the three-dimensional deformable head model as a latent vector and maps it to the corrected three-dimensional face model motion coefficient frame by frame; the conversion module 870 converts the corrected three-dimensional face model motion coefficient into a video frame to generate a head speaking video of the target character. The present application obtains the audio signal input by the user, the portrait image of the target person and the reference video of the target scene; encodes the audio signal to obtain the audio code; generates the motion coefficient of the three-dimensional face model according to the portrait image of the target person and the audio code; extracts the blink information and the head posture information from the reference video of the target scene; corrects the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model; uses the three-dimensional deformable head model as a potential vector, and maps it to the corrected motion coefficient of the three-dimensional face model frame by frame; converts the corrected motion coefficient of the three-dimensional face model into a video frame to generate the head speech video of the target person. By adding reference videos of different scene types, the head posture information and blink information of the speaker videos of different styles are customized, and the motion coefficient of the three-dimensional face model is corrected to ensure that the generated head state of the person is consistent with the voice content of the audio, and the coordination between the generated video and the audio is guaranteed. Based on this, the embodiments of the present application can generate character animations that are coordinated between video and audio, have high naturalness and good realism, and can realize customized generation of target character head speaking videos according to specific needs, meeting the specific requirements of different scenarios and speakers, thereby improving the quality and realism of voice-driven character animation generation. In financial scenarios, generating high-quality target character head speaking videos can enhance customer perception, increase customer trust, and help improve the quality of financial services, thereby enhancing the competitiveness of financial enterprises.

[0136] The specific implementation of the voice-driven character animation generation device is basically the same as the specific implementation of the voice-driven character animation generation method described above, and will not be repeated here.

[0137] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned voice-driven character animation generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0138] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0139] The processor 901 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in the embodiments of the present application.

[0140] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 is called to execute the voice-driven character animation generation method of the embodiment of this application, that is, by obtaining the audio signal input by the user, the target character portrait picture and the target scene reference video; encoding the audio signal to obtain the audio code; generating a three-dimensional face model motion coefficient according to the target character portrait picture and the audio code; extracting blink information and head posture information from the target scene reference video; correcting the three-dimensional face model motion coefficient according to the blink information and the head posture information to obtain the corrected three-dimensional face model motion coefficient; using the three-dimensional deformable head model as a potential vector, mapping it frame by frame to the corrected three-dimensional face model motion coefficient; converting the corrected three-dimensional face model motion coefficient into a video frame to generate a head speech video of the target person. By adding reference videos of different scene types, customizing the head posture information and blinking information of speakers' videos of different styles, and using this to correct the motion coefficients of the three-dimensional face model, it is ensured that the generated head state of the character is consistent with the voice content of the audio, and the coordination between the generated video and audio is guaranteed. Based on this, the embodiment of the present application can generate character animations that are coordinated between video and audio, with high naturalness and good realism, and can realize customized generation of target character head speech videos according to specific needs, meeting the specific requirements of different scenes and speakers, thereby improving the quality and realism of voice-driven character animation generation. In financial scenarios, generating high-quality target character head speech videos can enhance customer perception and customer trust, help improve the quality of financial services, and thus enhance the competitiveness of financial enterprises.

[0141] The input / output interface 903 is used to implement information input and output.

[0142] The communication interface 904 is used to realize the communication interaction between this device and other devices. Communication can be realized through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WIFI, Bluetooth, etc.).

[0143] The bus transmits information between various components of the device (eg, the processor 901 , the memory 902 , the input / output interface 903 , and the communication interface 904 ).

[0144] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus.

[0145] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned voice-driven character animation generation method is implemented.

[0146] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The voice-driven character animation generation method, voice-driven character animation generation device, electronic device and storage medium provided in the embodiment of the present application obtain the audio signal input by the user, the portrait picture of the target character and the reference video of the target scene; encode the audio signal to obtain the audio code; generate the three-dimensional face model motion coefficient according to the portrait picture of the target character and the audio code; extract the blink information and the head posture information from the reference video of the target scene; correct the three-dimensional face model motion coefficient according to the blink information and the head posture information to obtain the corrected three-dimensional face model motion coefficient; use the three-dimensional deformable head model as a potential vector, and map it frame by frame to the corrected three-dimensional face model motion coefficient; convert the corrected three-dimensional face model motion coefficient into a video frame to generate the target character's head speech video. By adding reference videos of different scene types, the head posture information and blink information of the speaker videos of different styles are customized, and the three-dimensional face model motion coefficient is corrected to ensure that the generated character's head state is consistent with the voice content of the audio, and the coordination between the generated video and the audio is guaranteed. Based on this, the embodiments of the present application can generate character animations that are coordinated between video and audio, have high naturalness and good realism, and can realize customized generation of target character head speaking videos according to specific needs, meeting the specific requirements of different scenarios and speakers, thereby improving the quality and realism of voice-driven character animation generation. In financial scenarios, generating high-quality target character head speaking videos can enhance customer perception, increase customer trust, and help improve the quality of financial services, thereby enhancing the competitiveness of financial enterprises.

[0148] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as a computer-readable program, a data structure, a program module or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally contain computer-readable programs, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0149] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0150] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0151] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0153] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0154] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0155] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0156] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0159] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating character animation based on speech drive, characterized in that: The method comprises: Obtaining the audio signal input by the user, the target person portrait image and the target scene reference video; Encoding the audio signal to obtain audio code; Generate a three-dimensional face model motion coefficient according to the target person portrait picture and the audio code; Extracting blink information and head posture information from the target scene reference video; Correcting the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain a corrected motion coefficient of the three-dimensional face model; The 3D deformable head model is used as a latent vector and mapped to the corrected 3D face model motion coefficients frame by frame. The corrected 3D face model motion coefficients are converted into video frames to generate a target person's head speaking video.

2. The method according to claim 1, characterized in that The motion coefficients of the three-dimensional face model include identity features, expression coefficients and head posture, wherein the head posture includes head rotation and head translation.

3. The method according to claim 2, characterized in that The step of correcting the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model includes: Extracting expression coefficients of T×64 dimensions from the motion coefficients of the three-dimensional face model of the target person portrait image; Inputting the expression coefficient of the T×64 dimension and the motion coefficient of the three-dimensional face model extracted frame by frame from the target scene reference video into a pre-trained first multi-layer perceptron to map to a potential expression vector of the T×64 dimension; Merging the potential expression vector of the T×64 dimension and the motion vector of the target scene reference video to obtain a merged vector; The merged vector and the three-dimensional face model motion coefficient of the target person portrait image are input into a pre-trained second multi-layer perceptron and mapped into a corrected three-dimensional face model motion coefficient of T×73 dimensions, where T is the total number of frames of the target scene reference video.

4. The method according to claim 3, characterized in that The step of extracting the expression coefficient of T×64 dimension from the motion coefficient of the three-dimensional face model of the target person portrait picture includes: Extracting a first three-dimensional face model motion coefficient from the target person portrait picture; Expanding the motion coefficient of the first three-dimensional face model to a total number of frames T; Extract the expression coefficients of T×64 dimensions from the first three-dimensional face model motion coefficients.

5. The method according to claim 3, characterized in that: The first multi-layer perceptron is obtained by separate training using mean square error, and the second multi-layer perceptron uses a pre-trained weight network in the process of generating motion coefficients of a three-dimensional face model.

6. The method according to claim 2, characterized in that The three-dimensional deformable head model is used as a potential vector and mapped frame by frame to the corrected three-dimensional face model motion coefficient, including: The corrected 3D face model motion coefficients are input into the mapping network and mapped to the coefficients of the unsupervised 3D key point space; The 3D deformable head model is used as the coefficients of the latent vector mapped to the unsupervised 3D keypoint space frame by frame; The facial expression and head movement of the three-dimensional deformable head model are controlled by coefficients in an unsupervised 3D keypoint space.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Inputting the video frame into an optical flow generator to generate a head posture animation of the target person; A head speech video of the target person is obtained according to the head posture animation of the target person.

8. A character animation generation device based on voice drive, characterized in that: The device comprises: An acquisition module is used to acquire an audio signal input by a user, a target person portrait image, and a target scene reference video; An encoding module, used for encoding the audio signal to obtain an audio code; A generating module, used for generating motion coefficients of a three-dimensional face model according to the target person portrait picture and the audio coding; An extraction module, used for extracting blink information and head posture information from the target scene reference video; A correction module, used for correcting the motion coefficient of the three-dimensional face model according to the blink information and the head posture information to obtain the corrected motion coefficient of the three-dimensional face model; A mapping module, used to map the three-dimensional deformable head model as a latent vector to the corrected three-dimensional face model motion coefficient frame by frame; The conversion module is used to convert the corrected three-dimensional face model motion coefficients into video frames to generate a target person's head speech video.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the voice-driven character animation generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating character animation based on voice-driven operation according to any one of claims 1 to 7 is implemented.

Images