Speech lip shape fitting method, system and storage medium

Abstract

The invention relates to a voice lip shape fitting method, comprising the following contents: collecting image data and voice data of a video data set of a target character; extracting a lip shape feature vector of the target character in the image data; extracting the target character in the voice data The character's voice feature vector; with the voice feature vector as input and the lip shape feature vector as output, train the multi-scale fusion convolutional neural network; input the voice feature vector of the target person to be fitted into the multi-scale fusion convolutional neural network, and multi-scale A fitted lip shape feature vector is generated by fusing the convolutional neural network and outputted, and the lip shape is fitted based on the lip shape feature vector.

Description

technical field

[0001] The present invention relates to the technical field of voice signals, and more specifically, to a voice lip shape fitting method, system and storage medium. Background technique

[0002] The voice lip shape fitting technology that generates the corresponding lip shape according to the voice is one of the basic technologies for applications such as virtual anchors, virtual image robots, and animation character mouth shape design. Accurately and smoothly fitting the lip shape according to the voice is the difficulty of this technology.

[0003] At present, lip shape fitting based on speech is technically realized through the following two solutions:

[0004] 1) According to phoneme theory and basic lip shape theory, after using Bayesian estimation, hidden Markov model, BP neural network, etc. to classify lip shape, the method of generating lip shape sequence by interpolation estimation.

[0005] 2) Lip shape estimation method using LSTM, RNN and other...

Examples

Embodiment 1

[0053] Such as figure 1 As shown, it is a flow chart of the speech lip shape fitting method based on multi-scale fusion convolutional neural network in this embodiment.

[0054] The speech lip shape fitting method based on the multi-scale fusion convolutional neural network of the present embodiment comprises the following steps:

[0055] S1: Collect the image data and voice data of the video data set of the target person. In this step, the image data and voice data of the target person video data set need to be collected at the same time and at the same frame rate, and the image data of the target person video data set needs to be collected using a three-dimensional structured light depth camera. In this embodiment, MacOS and ARKit are used to write a face tracking program, run on an IphoneX device, and use its front camera to collect the video image data at a frame rate of 60 frames per second.

[0056]S2: Extracting a lip shape feature vector of the target person in the i...

Embodiment 2

[0070] This embodiment provides a system for applying the voice lip shape fitting method of Embodiment 1, and its specific scheme is as follows:

[0071] Including data acquisition module, lip shape feature vector extraction module, voice feature vector extraction module, multi-scale fusion convolutional neural network training module and voice lip shape fitting module;

[0072] Wherein the data collection module is used to collect the image data and voice data of the target figure video data set;

[0073] The lip shape feature vector extraction module is used to extract the lip shape feature vector of the target person in the image data;

[0074] The voice feature vector extraction module is used to extract the voice feature vector of the target person in the voice data;

[0075] The multi-scale fusion convolutional neural network training module is used to train the multi-scale fusion convolutional neural network with the speech feature vector as input and the lip shape fea...

Embodiment 3

[0078] This embodiment provides a storage medium, and a program is stored in the storage medium, and the method steps of the voice lip shape fitting method in the first embodiment are executed when the program is running.

technical field

[0001] The present invention relates to the technical field of voice signals, and more specifically, to a voice lip shape fitting method, system and storage medium. Background technique

[0002] The voice lip shape fitting technology that generates the corresponding lip shape according to the voice is one of the basic technologies for applications such as virtual anchors, virtual image robots, and animation character mouth shape design. Accurately and smoothly fitting the lip shape according to the voice is the difficulty of this technology.

[0003] At present, lip shape fitting based on speech is technically realized through the following two solutions:

[0004] 1) According to phoneme theory and basic lip shape theory, after using Bayesian estimation, hidden Markov model, BP neural network, etc. to classify lip shape, the method of generating lip shape sequence by interpolation estimation.

[0005] 2) Lip shape estimation method using LSTM, RNN and other...

Examples

Embodiment 1

[0053] Such as figure 1 As shown, it is a flow chart of the speech lip shape fitting method based on multi-scale fusion convolutional neural network in this embodiment.

[0054] The speech lip shape fitting method based on the multi-scale fusion convolutional neural network of the present embodiment comprises the following steps:

[0055] S1: Collect the image data and voice data of the video data set of the target person. In this step, the image data and voice data of the target person video data set need to be collected at the same time and at the same frame rate, and the image data of the target person video data set needs to be collected using a three-dimensional structured light depth camera. In this embodiment, MacOS and ARKit are used to write a face tracking program, run on an IphoneX device, and use its front camera to collect the video image data at a frame rate of 60 frames per second.

[0056]S2: Extracting a lip shape feature vector of the target person in the i...

Embodiment 2

[0070] This embodiment provides a system for applying the voice lip shape fitting method of Embodiment 1, and its specific scheme is as follows:

[0071] Including data acquisition module, lip shape feature vector extraction module, voice feature vector extraction module, multi-scale fusion convolutional neural network training module and voice lip shape fitting module;

[0072] Wherein the data collection module is used to collect the image data and voice data of the target figure video data set;

[0073] The lip shape feature vector extraction module is used to extract the lip shape feature vector of the target person in the image data;

[0074] The voice feature vector extraction module is used to extract the voice feature vector of the target person in the voice data;

[0075] The multi-scale fusion convolutional neural network training module is used to train the multi-scale fusion convolutional neural network with the speech feature vector as input and the lip shape fea...

Embodiment 3

[0078] This embodiment provides a storage medium, and a program is stored in the storage medium, and the method steps of the voice lip shape fitting method in the first embodiment are executed when the program is running.

Images