Video synthesis method based on lip language synchronization and miracle adaptation effect enhancement

Abstract

The invention discloses a video synthesis method based on lip language synchronization and miracle adaptation effect enhancement. According to the method, the portrait and the audio stream to be synthesized are directly and integrally coded; a cyclic decoder network retaining original face information is used to decode converted abstract features into an image sequence, and then five discriminatornetworks are used to carry out adversarial training on the synthesized image sequence according to a real image sequence, so that a total reconstruction error is minimized. Compared with an existingvideo synthesis method, the method has the advantages that the continuity of face change between the front frame and the rear frame is guaranteed, the definition of face pictures in the frames is improved, meanwhile, under the action of the lip language synchronous discriminator and the miracle adaptation discriminator, the synthesized video appears more natural, and the authenticity of the visualeffect is greatly enhanced. The method has high practical value in the aspect of improving the user experience of virtual live broadcast and man-machine interaction.

Description

technical field [0001] The present invention relates to the field of artificial intelligence, in particular to the field of artificial intelligence virtual audio and video synthesis, in particular to a video synthesis method based on lip synchronization and expression adaptation effect enhancement. Background technique [0002] At present, with the continuous improvement of video quality captured by cameras and the rise of various online video platforms, the pressure on online video data storage is increasing day by day; in addition, due to the increasing number of online video viewing users, excellent web anchors are gradually in short supply. Case. In order to solve the above two difficulties, a product that synthesizes portrait video through text and audio data has emerged in the industry. Through this product, the video data is compressed into text and audio data for storage, and live webcasting instead of real people. However, the current virtual video synthesis method...

AI Technical Summary

Problems solved by technology

The patent "Virtual anchor implementation method and device" (application number 201811320949.4) uses the speech synthesis model and the overall state synthesis model of the face to synthesize speech sequences and image sequences, but does not process the details of the mouth shape and expression, resulting in changes in the mouth shape and expression of the portrait not obvious

The patent "A Method of Audio and Video Synthesis" (application number 201910912787.1) uses a variational self-encoding network to achieve end-to-end audio and video synthesis, and further considers the relationship between the front and rear frame images, but there are still details such as mouth shapes that do not change significantly The problem

Images