Video generation method, device, electronic device and storage medium

Abstract

The embodiment of the present invention relates to the field of computer vision, and discloses a video generation method, device, electronic equipment, and storage medium. The video generation method of the present invention includes: extracting the audio feature of each frame of the target audio data according to the video frame rate corresponding to the pre-trained video generation model; acquiring the human body posture vector of the person in the target video data synchronized with the target audio data; Using the audio feature and the human body pose vector corresponding to the audio feature through a pre-trained video generation model to obtain a portrait video synchronized with the target audio data, wherein the portrait video includes multiple frames of character images, and the multiple The frame of person image contains the mapping relationship between the audio feature and the human body pose vector. Applied to the process of voice-driven video generation, the generated video voice and portrait are strictly matched and synchronized.

Description

technical field [0001] Embodiments of the present invention relate to the field of computer vision, and in particular, to a video generation method, device, electronic equipment, and storage medium. Background technique [0002] In fields such as artificial intelligence and computer vision, digital or virtual humans that simulate real-life prototypes have been used more and more. The generation of digital human or virtual human mainly utilizes voice-driven video generation technology to generate visual effects as realistic as the original video by estimating the expression, movement and speaking style of the human face at this moment. At present, voice-driven video generation is mostly achieved through processing methods such as reconstruction of 3D faces, efficient regression of expression coefficients, or 2D facial key points. [0003] However, most of the above processing methods do not take into account the changes in the body movements of the characters, resulting in r...

AI Technical Summary

Problems solved by technology

[0003] However, most of the above processing methods do not take into account the changes in the body movements of the characters, resulting in rigid and stiff videos.

In addition, due to various analysis processes on the face, these intermediate analysis processes have caused the loss of face information, so that the generated video voice and face image are not completely matched and synchronized.

Images