Method, system and device for converting voice into lip shape and storage medium

Abstract

The invention discloses a method, system and device for converting voice into a lip shape and a storage medium. The method comprises the steps: acquiring a voice sequence; receiving and processing thevoice sequence by using a trained generative adversarial network model; and obtaining a lip-shaped image output by the trained generative adversarial network model. According to the method, the generative adversarial network model (GAN) is trained, and the trained generative adversarial network model is utilized to convert the voice into the lip shape, so that the lip-shaped image with high quality and high resolution can be obtained; the generative adversarial network model is trained in an unsupervised learning mode, so that the voice quality can be obviously improved, the voice distortionis reduced, and the robustness of the system is enhanced; when changed voice is continuously input, a dynamic lip-shaped image can be finally output, and a smooth visual effect can be provided; and meanwhile, the generated lip-shaped image is combined with the voice, so that a high-quality face speaking video can be synthesized. The method, system and device are widely applied to the technical field of voice data.

Description

technical field [0001] The invention relates to the technical field of voice data, in particular to a method, system, device and storage medium for converting voice into lip shapes. Background technique [0002] After years of exploration and development, computer vision has been applied in many fields such as digital entertainment, medical health, and security monitoring. Synthesizing photorealistic visual content is not only of great commercial value, but also has been desired by the industry. Many movie special effects would also not be possible without computer-generated integrated visual effects. At present, there are already a large number of artificially synthesized videos on the Internet. In addition, speech recognition and text-to-speech technologies have also been widely used in chat robots. However, the existing technologies generally have shortcomings such as huge amount of calculation and time-consuming. Most researches on audio-based speech and face video gen...

AI Technical Summary

Problems solved by technology

In addition, speech recognition and text-to-speech technologies have also been widely used in chat robots. However, the existing technologies generally have disadvantages such as large amount of calculation and time-consuming. Most researches on audio-based speech and face video generation are based on Graphics methods, but such methods usually require training and modeling of a large number of videos of specific target objects; existing deep learning-based methods use the Image-to-Image method to generate the entire face from a single image Speaking videos, this method is enough to get a lip shape that matches the provided voice well, but the quality of the generated image is greatly reduced. The generated result is not only low in resolution, but the facial features of the characters may even be lost or Chromatic aberration and other problems

Images