Voice-based mouth shape and/or expression simulation method and device

Abstract

The invention discloses a voice-based mouth shape and / or expression simulation method and device. The method comprises the steps that audio signals are acquired; the audio signals are converted into the spectrum data corresponding to the audio signals; the frequency distribution data are determined according to the spectrum data; the mouth shape simulation parameter and / or the expression simulation parameter are / is determined according to the frequency distribution data; and the corresponding mouth shape is simulated according to the mouth shape simulation parameter and / or the corresponding expression is simulated according to the expression simulation parameter. The technical problems of high workload of mouth shape or expression simulation in the virtual reality application in the related technology can be solved.

Description

technical field [0001] The present invention relates to the technical field of computer graphics processing, in particular to a voice-based mouth shape and / or facial expression simulation method and device. Background technique [0002] In related technologies, most of the real-time voice-based lip and facial expression simulation solutions require complex audio processing or animation lip libraries as assistance. For example, in the related art, by analyzing the syllables of the voice, matching the mouth shape of the animated character in real time for the voice, it first needs the support of the animation mouth shape library and the syllable mouth shape library, how to obtain and calibrate these two libraries is relatively troublesome. In addition, Some application scenarios (such as mobile virtual reality applications) are not suitable for packaging several libraries with the application, which will increase the size of the package. And for obtaining speech data in the r...

AI Technical Summary

Problems solved by technology

[0004] 1. Many schemes require a lot of preparatory work in the early stage, such as establishing a voice library, manually marking various voice tags, etc., the process is cumbersome

[0005] 2. The amount of calculation is large, especially in virtual reality applications. The platform itself has high performance requirements, and there are often multiple people on the same screen who need to simulate lip movements and facial expressions at the same time. Searching and matching in the speech database will cause performance problems

[0006] 3. For many implementation schemes based on neural networks, the final implementation effect is directly related to the accuracy of the collected training data, so the quality requirements for the collected data are high, and the workload and technology required for this requirement The means are more complicated

Moreover, due to the fact that in virtual reality, the user cannot really know what kind of mouth shape and expression the other party has made in reality.

Therefore, wasting too much computing resources on improving the simulation accuracy is less cost-effective in virtual reality applications.

Images