Three-dimensional face personalized expression generation method and system based on speech recognition

Abstract

The invention discloses a three-dimensional face personalized expression generation method and system based on voice recognition, and the method comprises the steps: obtaining a voice signal inputted by a user, carrying out the data preprocessing, obtaining voice input data, carrying out the feature extraction of the voice input data through a voice model HuBERT, and obtaining voice feature information; inputting a sentiment classification model for training to recognize a sentiment classification result, inputting a basic expression generation model for training according to the voice feature information and the sentiment classification result to obtain expression data, and performing personalized expression fine adjustment on the expression data and preset personalized features to obtain a personalized expression generation model, personalized emotional expressions are generated based on the emotion classification result and the personalized expression generation model, the robustness of the expression generation model to different voice timbres is improved, personalized adjustment is performed on the basic expression generation model through a fine adjustment mode, and therefore unique expression styles and speaking habits of specific users are learned, and the user experience is improved. And emotional expressions can be accurately generated through voices of different emotions.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to a method and system for generating personalized expressions of three-dimensional faces based on speech recognition. Background Art

[0002] At present, in the field of 3D expression generation, existing technologies mainly focus on predicting and generating corresponding facial expressions by analyzing speech signals. The following are some key aspects: 1. Speech feature extraction: Traditional methods usually include the use of acoustic models (such as Mel-frequency cepstral coefficients, MFCC) to extract features from speech. These features can express the basic properties of speech, such as pitch, rhythm, intensity, etc., which have a certain correlation with expression changes; 2. Expression parameter mapping: Map the extracted speech features to facial action units (Facial Action Coding System, FACS) or other predefined facial expression parameters. This process may use simple linear models or complex machine learning algorithms, such as support vector machines (SVM), random forests (Random Forests) or neural networks to build a voice-to-expression conversion model; 3. Deep learning methods: Deep learning, especially convolutional neural networks (CNN) and recurrent neural networks (RNN), are widely used in natural language processing tasks and have been introduced into voice-driven facial expression generation. These models can learn more complex and abstract voice-expression mapping relationships from large amounts of data; 4. End-to-end systems: Some studies have explored end-to-end methods to generate facial expressions directly from voice signals, eliminating the need for explicit feature extraction steps. This method uses a network structure similar to a sequence-to-sequence model, which can simultaneously learn the intrinsic connection between voice features and expression dynamics.

[0003] However, the above-mentioned three-dimensional facial expression generation has the following technical problems: (1) Inaccurate voice-expression matching: The existing technology still faces challenges in the precise matching between voice and expression. The mapping of voice to facial expression is a non-deterministic mapping, which is difficult to accurately model, resulting in the generated facial expressions being inaccurate or unnatural; (2) Insufficient expression details: In some cases, the existing technology may lack fine control over details when generating facial expressions. Specific facial expression details, such as the angle of a smile or changes in the eyes, may not be fully considered and expressed. Therefore, it is necessary to provide a method and system for generating personalized three-dimensional facial expressions based on voice recognition to solve the above-mentioned technical problems. Summary of the invention

[0004] In view of this, the present invention provides a method and system for generating personalized expressions of three-dimensional faces based on speech recognition, which can process the voice input of any user to generate corresponding expression parameters with emotions, and improve the robustness of the expression generation model for different voice timbres, so as to solve the above-mentioned technical problems, and specifically adopt the following technical solutions to achieve this.

[0005] In a first aspect, the present invention provides a method for generating personalized expressions of three-dimensional faces based on speech recognition, comprising the following steps:

[0006] Acquire a voice signal input by a user and perform data preprocessing on the voice signal to obtain voice input data, wherein the data preprocessing includes performing noise filtering, data enhancement and timbre migration on the voice signal;

[0007] Using the speech model HuBERT to extract features from the speech input data to obtain speech feature information, and inputting the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data;

[0008] Inputting the trained basic expression generation model according to the speech feature information and the emotion classification result to perform training to obtain expression data, wherein the basic expression generation model is obtained by training the diffusion model;

[0009] The expression data and preset personalized features are fine-tuned to obtain a personalized expression generation model, and a personalized emotional expression corresponding to the voice signal is generated based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

[0010] As a preferred embodiment of the above technical solution, the timbre migration includes:

[0011] Acquire first sound data at a first preset time;

[0012] Training the first sound data to obtain a voice changer corresponding to the first sound data;

[0013] The input sound is transformed into a target sound according to the voice transformer.

[0014] As a preferred embodiment of the above technical solution, the training process of the basic expression generation model includes:

[0015] Collecting second sound data of different emotions and target expression data corresponding to the second sound data, wherein the second sound data and the target expression data are aligned, the second sound data are collected by a microphone, and the target expression data are collected by livelinkface software;

[0016] The second sound data and the target expression data are segmented to obtain segmented data, wherein a long voice in the second sound data is segmented into a plurality of short voices of a second preset time, and the target expression data is segmented accordingly;

[0017] The segmented data is used to train a diffusion model to obtain a basic expression generation model.

[0018] As a preferred embodiment of the above technical solution, the process of training the style encoder includes:

[0019] Collecting video data of multiple people of different genders and ages speaking, wherein the duration of the video data of each person is within a third preset time, and the video data belongs to the preset audio data;

[0020] Processing the video data to obtain time-series three-dimensional motion data of the person;

[0021] The time series 3D motion data is input into the time series convolutional neural network to obtain the style feature e, and the auxiliary style classifier A is used to classify the style features of different characters. The corresponding loss function is: Among them, A e is a style classifier implemented by a training weight matrix, p is a unique label for the identity class, N e is the number of person identities in the training set;

[0022] A style encoder is obtained based on the time series three-dimensional motion data, the temporal convolutional network and the style classifier training.

[0023] As a preferred embodiment of the above technical solution, the process of extracting style features includes:

[0024] Collect the target person’s speaking video;

[0025] Performing three-dimensional reconstruction on the target person's speaking video to obtain three-dimensional vertex motion sequence data of the target person, wherein the three-dimensional vertex motion sequence data includes facial motion of the target person when speaking, which reflects the specific identity and speaking style of the target person;

[0026] Performing difference calculation on the three-dimensional vertex motion sequence data to obtain difference sequence data, wherein the difference calculation is used to obtain the motion change between each frame, and the difference sequence data is used to describe the motion change of the target person's facial vertices between adjacent frames to reflect the dynamic characteristics of the target person's speaking style;

[0027] The difference sequence data is used as the input of the style encoder to extract the style features of the target person, and the style features are used to fine-tune the basic expression generation model to obtain a personalized expression generation model.

[0028] As a preferred embodiment of the above technical solution, the face of the preset target person is composed of N vertices. Then, in the tth frame, the position of each vertex can be expressed as a three-dimensional coordinate of The three-dimensional vertex motion sequence data of the speaking video can be represented as a matrix S: S = V1, V2, V3, ..., V T}, where T is the total number of frames in the video, V t is the vertex coordinate vector of the tth frame;

[0029] The difference calculation process includes: ΔV t =V t+1 -V t ,t=1,2,...,T-1, where ΔV t represents the vertex motion difference between the t-th frame and the t-1 frame, and the result will form a new difference sequence data ΔS: ΔS = {ΔV1, ΔV2, ..., ΔV T-1};

[0030] The input of the style encoder is the difference sequence data ΔS output, and a coding vector e representing the style feature is preset: e=TCN(ΔS), where e is a feature vector of fixed dimension, which represents the extracted facial movement style of the specific target person.

[0031] As a preferred embodiment of the above technical solution, the process of model fine-tuning includes:

[0032] The parameters of the basic expression generation model except the last fully connected layer remain unchanged to obtain a fixed network;

[0033] The style features are fused with the output of the fixed network and input into the fully connected layer to obtain the target basic expression generation model;

[0034] The time series three-dimensional motion data is used to fine-tune the parameters of the last fully connected layer of the target basic expression generation model to obtain a personalized expression generation model. The voice is input and processed by the voice changer and the emotional result is obtained through the emotion classifier. Based on the personalized expression generation model and the emotional result, expressions with personalized characteristics are generated.

[0035] As a preferred embodiment of the above technical solution, the style features are fused with the output of the fixed network and input into the fully connected layer to obtain the target basic expression generation model, including:

[0036] Get the style feature e obtained by the style encoder, where e is a feature vector of fixed dimension;

[0037] Get the basic expression feature x' output by the fixed network, where x' is a feature vector of fixed dimension, and its dimension is equal to e;

[0038] The style feature e is combined with the basic expression feature x' to obtain the personalized expression feature x, x = x' + e;

[0039] Input the personalized expression feature x into the fully connected layer.

[0040] As a preferred embodiment of the above technical solution, the first preset time is five to ten minutes, the second preset time is five seconds, the third preset time is between half an hour and one hour, and the emotion classification result includes at least one of happiness, disgust, fear, neutrality, sadness, surprise, contempt or anger.

[0041] In a second aspect, the present invention further provides a three-dimensional face personalized expression generation system based on speech recognition, comprising:

[0042] A data preprocessing unit, used to obtain a voice signal input by a user and perform data preprocessing on the voice signal to obtain voice input data, wherein the data preprocessing includes performing noise filtering, data enhancement and timbre migration on the voice signal;

[0043] An emotion discrimination unit is used to extract features from the speech input data using a speech model HuBERT to obtain speech feature information, and input the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data;

[0044] An expression data generating unit, used for inputting a trained basic expression generation model according to the speech feature information and the emotion classification result for training to obtain expression data, wherein the basic expression generation model is obtained by training a diffusion model;

[0045] A personalized expression generation unit is used to perform personalized expression fine-tuning on the expression data and preset personalized features to obtain a personalized expression generation model, and generate a personalized emotional expression corresponding to the voice signal based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

[0046] The present invention provides a method and system for generating personalized expressions of three-dimensional human faces based on speech recognition. The method comprises the following steps: acquiring a speech signal input by a user and performing data preprocessing on the speech signal to obtain speech input data; using a speech model HuBERT to perform feature extraction on the speech input data to obtain speech feature information; inputting the speech feature information into a trained emotion classification model for training to identify an emotion classification result corresponding to the speech input data; inputting the speech feature information and the emotion classification result into a trained basic expression generation model for training to obtain expression data; performing personalized expression fine-tuning on the expression data and preset personalized features to obtain a personalized expression generation model; generating personalized emotional expressions corresponding to the speech signal based on the emotion classification result and the personalized expression generation model; processing speech input of any user and generating corresponding expression parameters with emotion; performing timbre migration processing on the speech data to well improve the robustness of the expression generation model for different speech timbres; performing personalized adjustment on the existing basic expression generation model by fine-tuning, thereby learning the unique expression style and speaking habits of a specific user; and accurately generating corresponding emotional expressions through speech with different emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flow chart of a method for generating personalized expressions of three-dimensional faces based on speech recognition provided by the present invention;

[0049] Figure 2 A simplified flow chart of the method for generating personalized facial expressions based on speech recognition for a three-dimensional human face provided by the present invention;

[0050] Figure 3 This is a structural block diagram of the three-dimensional face personalized expression generation system based on speech recognition provided by the present invention. DETAILED DESCRIPTION

[0051] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0052] See also Figure 1 and Figure 2 The present invention provides a method for generating personalized expressions of three-dimensional faces based on speech recognition, comprising the following steps:

[0053] S1: Acquire a voice signal input by a user and perform data preprocessing on the voice signal to obtain voice input data, wherein the data preprocessing includes performing noise filtering, data enhancement and timbre migration on the voice signal;

[0054] S2: Using the speech model HuBERT to extract features from the speech input data to obtain speech feature information, and inputting the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data;

[0055] S3: inputting the trained basic expression generation model into the speech feature information and the emotion classification result for training to obtain expression data, wherein the basic expression generation model is obtained by training the diffusion model;

[0056] S4: Fine-tune the expression data with preset personalized features to obtain a personalized expression generation model, and generate a personalized emotional expression corresponding to the voice signal based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

[0057] In this embodiment, the timbre migration includes: obtaining first sound data of a first preset time; training the first sound data to obtain a voice changer corresponding to the first sound data; and converting the input sound into a target sound according to the voice changer. The training process of the basic expression generation model includes: collecting second sound data of different emotions and target expression data corresponding to the second sound data, wherein the second sound data and the target expression data are aligned, the second sound data are collected by a microphone, and the target expression data are collected by livelinkface software; segmenting the second sound data and the target expression data to obtain segmented data, wherein the long voice in the second sound data is segmented into multiple short voices of a second preset time, and the target expression data is segmented accordingly; and the diffusion model is trained using the segmented data to obtain a basic expression generation model. The first preset time is five to ten minutes, the second preset time is five seconds, and the third preset time is between half an hour and one hour. The emotion classification result includes at least one of happiness, disgust, fear, neutrality, sadness, surprise, contempt or anger. Speech signal denoising refers to a signal processing technology that removes noise interference from speech signals to make the signals clearer and easier to identify. Commonly used denoising algorithms include spectral subtraction, wavelet denoising algorithm or Wiener filtering. Data enhancement is to make small changes to existing data sets, such as rotation, shift and other small changes, to obtain more data.

[0058] It should be noted that the process of the method for generating personalized expressions of three-dimensional faces based on speech recognition provided by the present invention can be simplified as follows: (1) speech input, (2) preprocessing, (3) feature extraction, (4) emotion discrimination, (5) generation of emotional expressions, (6) personalized data input, (7) personalized fine-tuning and (8) personalized expression generation. The above (1)-(4) can be collectively classified as input. The above (1) to (4) can be collectively classified as an input module, which inputs the user's speech signal and performs noise filtering, data enhancement, timbre migration and other processing operations on the speech signal to improve the accuracy of subsequent feature extraction and the robustness of the expression generation model to the sound timbre. The detailed process of timbre migration is as follows: prepare five to ten minutes of target voice data; train the target voice changer; input any voice, and transform the input voice into the target voice through the trained voice changer; use the speech model HuBERT to extract features from the speech signal and convert the speech signal into a vector with semantic information. HuBERT is a self-supervised learning model used to learn meaningful representations from unlabeled speech input data; input the extracted features into the pre-trained emotion classification model; the classification model is trained based on a neural network and is used to identify the emotion categories contained in the speech (such as happiness, disgust, fear, neutrality, sadness, surprise, contempt, anger, etc.). The above (5) is an expression generation module, which combines the original speech features and the emotion classification results as conditional inputs, uses a basic expression generation model, which is trained based on a diffusion model, and outputs the corresponding general emotion-bearing expression.

[0059] The detailed steps of model training are as follows: collect sounds of different emotions and the corresponding expression data of the sounds, and each sound data and expression data must be strictly aligned; use python to write software, record sound and expression data at the same time, use microphone to collect sound, use Apple's livelinkface software to collect expression data, and collect 60 frames of expression data per second; divide the sound data, divide the long speech into multiple five-second short speech, and divide the corresponding expression data in this way; use the processed data to train the diffusion model to obtain a basic expression generation model; the above (6) to (8) are personalized expression fine-tuning and personalized expression generation modules, collect specific target person speech video samples, use style encoder to extract features from the video, and extract the style features of the specific person, that is, personalized features. The style encoder is trained by the temporal convolutional neural network. Among them, the program collects expression data and sound data synchronously, strictly collecting 16,000 audio sampling points and 60 frames of expression per second, so there will be no misalignment. If misalignment occurs, you can only choose to re-collect, so that the data is the cleanest. The purpose of collecting video features of a specific target person speaking is to extract style features. The video data contains the motion information of the person speaking, and the style features of the person speaking are extracted from the motion information of the person speaking.

[0060] It should be understood that the speech input data is obtained by obtaining the speech signal input by the user and performing data preprocessing on the speech signal, the speech input data is feature extracted using the speech model HuBERT to obtain speech feature information, the speech feature information is input into the trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, the speech feature information and the emotion classification result are input into the trained basic expression generation model for training to obtain expression data, the expression data and the preset personalized features are fine-tuned for personalized expression to obtain a personalized expression generation model, and the personalized emotional expression corresponding to the speech signal is generated based on the emotion classification result and the personalized expression generation model, and a basic three-dimensional facial expression parameter generation model is established, which can process the speech input of any user and generate corresponding expression parameters with emotion; for any speech input, the speech is first processed for timbre migration, which can well improve the robustness of the expression generation model for different speech timbres. When it is necessary to generate personalized expressions, the existing basic expression generation model is personalized by fine-tuning, so as to learn the unique expression style and speaking habits of specific users. In this way, natural and personalized 3D facial expression animation can be achieved, whether in general scenarios or for specific users. The entire system is end-to-end and can generate expressions in real time.

[0061] Optionally, the process of training a style encoder includes:

[0062] Collecting video data of multiple people of different genders and ages speaking, wherein the duration of the video data of each person is within a third preset time, and the video data belongs to the preset audio data;

[0063] Processing the video data to obtain time-series three-dimensional motion data of the person;

[0064] The time series 3D motion data is input into the time series convolutional neural network to obtain the style feature e, and the auxiliary style classifier A is used to classify the style features of different characters. The corresponding loss function is: Among them, A e is a style classifier implemented by a training weight matrix, p is a unique label for the identity class, N e is the number of person identities in the training set;

[0065] A style encoder is obtained based on the time series three-dimensional motion data, the temporal convolutional network and the style classifier training.

[0066] In this embodiment, more than 150 speaking video data of people of different genders and ages are collected, and the video data of each person is about half an hour to one hour; the original video data is processed to obtain the time series three-dimensional motion data of the person, which is the training data set; the three-dimensional motion data is input into the time series convolutional neural network to obtain style features, and the style features of different subjects, i.e., people, are classified using an auxiliary style classifier; through the training data set, the time series convolutional neural network and the style classifier, a style encoder is finally trained. The process of extracting style features includes: collecting a target person's speaking video; performing three-dimensional reconstruction on the target person's speaking video to obtain three-dimensional vertex motion sequence data of the target person, wherein the three-dimensional vertex motion sequence data includes facial motion of the target person when speaking to reflect the target person's specific identity and speaking style; performing difference calculation on the three-dimensional vertex motion sequence data to obtain difference sequence data, wherein the difference calculation is used to obtain the motion change between each frame, and the difference sequence data is used to describe the motion change of the target person's facial vertices between adjacent frames to reflect the dynamic characteristics of the target person's speaking style; using the difference sequence data as the input of a style encoder to extract the style features of the target person, and using the style features to fine-tune a basic expression generation model to obtain a personalized expression generation model.

[0067] It should be noted that the face of the preset target person is composed of N vertices. In the tth frame, the position of each vertex can be expressed as a three-dimensional coordinate: The three-dimensional vertex motion sequence data of the speaking video can be represented as a matrix S: S = {V1, V2, V3, ..., V T}, where T is the total number of frames in the video, V t is the vertex coordinate vector of the tth frame;

[0068] The difference calculation process includes: ΔV t =V t+1 -V t ,t=1,2,...,T-1, where ΔV t represents the vertex motion difference between the t-th frame and the t-1 frame, and the result will form a new difference sequence data ΔS: ΔS = {ΔV1, ΔV2, ..., ΔV T-1};

[0069] The input of the style encoder is the difference sequence data ΔS output, and a coding vector e representing the style feature is preset: e=TCN(ΔS), where e is a feature vector of fixed dimension, which represents the extracted facial movement style of the specific target person.

[0070] Specifically, the detailed process of extracting style features is as follows: collect 3-5 minutes of target person's speaking video; obtain 3D vertex motion sequence data: firstly, the target person's speaking video is 3D reconstructed by 3D reconstruction technology to obtain the target person's 3D vertex motion sequence data, which records the target person's facial movement during the speaking process and can reflect the specific identity and speaking style of the person; 3D vertex motion difference calculation: in order to capture the dynamic changes of facial vertices in time, the vertex sequence is further difference calculated. The goal of difference calculation is to obtain the motion change between each frame, which helps to extract the dynamic facial expression characteristics of the target person. The difference sequence data ΔS describes the motion change of the target person's facial vertices between adjacent frames, which can reflect the dynamic characteristics of the person's speaking style; style feature extraction: the 3D vertex motion difference data ΔS is used as the input of the style encoder to extract the specific style features of the target person. The style encoder adopts the Temporal Convolutional Network (TCN) structure. The network can capture the temporal dependency of the input data in the time dimension, so as to better extract the style features of facial motion. The collected style features are used to fine-tune the general expression generation model (diffusion model) to obtain a personalized expression generation model (diffusion model) to adapt to the expression characteristics and language style of personalized users. Among them, the general expression generation model is the basic expression generation model.

[0071] Optionally, the process of model fine-tuning includes:

[0072] The parameters of the basic expression generation model except the last fully connected layer remain unchanged to obtain a fixed network;

[0073] The style features are fused with the output of the fixed network and input into the fully connected layer to obtain the target basic expression generation model;

[0074] The time series three-dimensional motion data is used to fine-tune the parameters of the last fully connected layer of the target basic expression generation model to obtain a personalized expression generation model. The voice is input and processed by the voice changer and the emotional result is obtained through the emotion classifier. Based on the personalized expression generation model and the emotional result, expressions with personalized characteristics are generated.

[0075] In this embodiment, the style feature is fused with the output of the fixed network and input into the fully connected layer to obtain the target basic expression generation model, including: obtaining the style feature e obtained by the style encoder, where e is a feature vector of fixed dimension; obtaining the basic expression feature x' output by the fixed network, where x' is a feature vector of fixed dimension and the dimension is equal to e; fusing the style feature e with the basic expression feature x' to obtain the personalized expression feature x, x=x'+e; and inputting the personalized expression feature x into the fully connected layer.

[0076] 1) It should be noted that the detailed process of model fine-tuning includes: fixing the parameters of the basic model network except the last fully connected layer, and the basic model network has other layers except the last fully connected layer. The parameters outside here refer to the network parameters of other layers, such as the transformer structure, which has parameters; the style features are fused with the output of the fixed network and then input into the fully connected layer; the training data is used to fine-tune the parameters of the last fully connected layer of the basic model to obtain a personalized expression generation model; the voice is input, the voice changer processes the voice, and the emotion result is obtained through the emotion classifier, and finally the personalized expression generation model generates an expression with personalized characteristics. Among them, the basic model network is the basic expression generation model, and the classifier is used to judge the emotion of the sound, such as happy, neutral, sad, etc., and output the emotion label, which is used as a condition for expression generation.

[0077] See also Figure 3 The present invention also provides a three-dimensional face personalized expression generation system based on speech recognition, comprising:

[0078] A data preprocessing unit, used to obtain a voice signal input by a user and perform data preprocessing on the voice signal to obtain voice input data, wherein the data preprocessing includes performing noise filtering, data enhancement and timbre migration on the voice signal;

[0079] An emotion discrimination unit is used to extract features from the speech input data using a speech model HuBERT to obtain speech feature information, and input the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data;

[0080] An expression data generating unit, used for inputting a trained basic expression generation model according to the speech feature information and the emotion classification result for training to obtain expression data, wherein the basic expression generation model is obtained by training a diffusion model;

[0081] A personalized expression generation unit is used to perform personalized expression fine-tuning on the expression data and preset personalized features to obtain a personalized expression generation model, and generate a personalized emotional expression corresponding to the voice signal based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

[0082] In this embodiment, a neural network is used to train a voice changer to convert the timbre of the input voice into a target timbre, and the robustness of the model to timbre differences is improved by timbre migration, and the interference of timbre differences on the model generation results is reduced. The voice changer is trained, the input voice is converted to timbre, and the converted voice is used to drive the expression generation model to generate expressions, and the timbre migration is used to improve the robustness of the model to different voice inputs. The speech emotion recognizer is trained, the input voice is emotionally recognized, and the expression generation model is driven by the emotion label and the voice signal to generate emotional expressions. The speech is emotionally classified, and the speech emotion category is used to guide the model to generate corresponding emotional expressions. The speech emotion recognizer is trained by a neural network, and the input voice is emotionally recognized to obtain the emotion category. The emotion category is used as a condition and the input voice as the input of the expression generation model to drive the expression generation model to generate emotional expressions. In addition, by collecting a target style speech video, using a neural network model to extract the target style features, and using the target style features to fine-tune the general expression generation model to obtain a personalized expression generation model, the model can be fine-tuned for a specific target person, and the expression generated by the model conforms to the style of the target person, so as to achieve personalized expression generation.

[0083] It should be noted that the present invention can generate expressions through voice in real time, use diffusion model to improve the stability of non-deterministic mapping, improve the accuracy of expressions and enrich the details of expressions. During personalized data preprocessing, the specific target person style is vectorized by encoding, and the corresponding emotional expressions can be accurately generated through voices with different emotions.

[0084] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0085] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0086] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for generating personalized facial expressions based on speech recognition, characterized in that: The following steps are involved: Acquire a voice signal input by a user and perform data preprocessing on the voice signal to obtain voice input data, wherein the data preprocessing includes performing noise filtering, data enhancement and timbre migration on the voice signal; Using the speech model HuBERT to extract features from the speech input data to obtain speech feature information, and inputting the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data; Inputting the trained basic expression generation model according to the speech feature information and the emotion classification result to perform training to obtain expression data, wherein the basic expression generation model is obtained by training the diffusion model; The expression data and preset personalized features are fine-tuned to obtain a personalized expression generation model, and a personalized emotional expression corresponding to the voice signal is generated based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

2. The method for generating personalized facial expressions based on speech recognition according to claim 1, characterized in that: The timbre migration includes: Acquire first sound data at a first preset time; Training the first sound data to obtain a voice changer corresponding to the first sound data; The input sound is transformed into a target sound according to the voice transformer.

3. The method for generating personalized facial expressions based on speech recognition according to claim 1, characterized in that: The training process of the basic expression generation model includes: Collecting second sound data of different emotions and target expression data corresponding to the second sound data, wherein the second sound data and the target expression data are aligned, the second sound data are collected by a microphone, and the target expression data are collected by livelinkface software; The second sound data and the target expression data are segmented to obtain segmented data, wherein a long voice in the second sound data is segmented into a plurality of short voices of a second preset time, and the target expression data is segmented accordingly; The segmented data is used to train a diffusion model to obtain a basic expression generation model.

4. The method for generating personalized facial expressions based on speech recognition according to claim 1, characterized in that: The process of training a style encoder involves: Collecting video data of multiple people of different genders and ages speaking, wherein the duration of the video data of each person is within a third preset time, and the video data belongs to the preset audio data; Processing the video data to obtain time-series three-dimensional motion data of the person; The time series 3D motion data is input into the time series convolutional neural network to obtain the style feature e, and the auxiliary style classifier A is used to classify the style features of different characters. The corresponding loss function is: in, is a style classifier implemented by a training weight matrix, p is a unique label for the identity class, is the number of person identities in the training set; A style encoder is obtained based on the time series three-dimensional motion data, the temporal convolutional network and the style classifier training.

5. The method for generating personalized facial expressions based on speech recognition according to claim 4, characterized in that: The process of extracting style features includes: Collect target person’s speaking video; Performing three-dimensional reconstruction on the target person's speaking video to obtain three-dimensional vertex motion sequence data of the target person, wherein the three-dimensional vertex motion sequence data includes facial motion of the target person when speaking, which reflects the specific identity and speaking style of the target person; Performing difference calculation on the three-dimensional vertex motion sequence data to obtain difference sequence data, wherein the difference calculation is used to obtain the motion change between each frame, and the difference sequence data is used to describe the motion change of the target person's facial vertices between adjacent frames to reflect the dynamic characteristics of the target person's speaking style; The difference sequence data is used as the input of the style encoder to extract the style features of the target person, and the style features are used to fine-tune the basic expression generation model to obtain a personalized expression generation model.

6. The method for generating personalized facial expressions based on speech recognition according to claim 5, characterized in that: The face of the target person is composed of N vertices. At the tth frame, the position of each vertex can be expressed as a three-dimensional coordinate: The three-dimensional vertex motion sequence data of the speaking video can be represented as a matrix Where T is the total number of frames in the video, is the vertex coordinate vector of the tth frame; The difference calculation process includes: in, Represents the vertex motion difference between the tth frame and the t-1th frame. The result will form a new difference sequence data ; The input of the style encoder is the difference sequence data Output and preset an encoding vector representing style features in, is a fixed-dimensional feature vector that represents the extracted facial movement style of a specific target person.

7. The method for generating personalized facial expressions based on speech recognition according to claim 6, characterized in that: The process of model fine-tuning includes: The parameters of the basic expression generation model except the last fully connected layer remain unchanged to obtain a fixed network; The style features are fused with the output of the fixed network and input into the fully connected layer to obtain the target basic expression generation model; The time series three-dimensional motion data is used to fine-tune the parameters of the last fully connected layer of the target basic expression generation model to obtain a personalized expression generation model. The voice is input and processed by the voice changer and the emotional result is obtained through the emotion classifier. Based on the personalized expression generation model and the emotional result, expressions with personalized characteristics are generated.

8. The method for generating personalized facial expressions based on speech recognition according to claim 7, characterized in that: The style features are fused with the output of the fixed network and input into the fully connected layer to obtain the target basic expression generation model, including: Get the style feature e obtained by the style encoder, where e is a feature vector of fixed dimension; Get the basic expression features of the fixed network output in, is a fixed-dimensional feature vector, and its dimension is equal to e; The style feature e and the basic expression feature The personalized expression feature x is obtained by fusion. ; Input the personalized expression feature x into the fully connected layer.

9. The method for generating personalized facial expressions based on speech recognition according to claim 1, characterized in that: The first preset time is five to ten minutes, the second preset time is five seconds, the third preset time is between half an hour and one hour, and the emotion classification result includes at least one of happiness, disgust, fear, neutrality, sadness, surprise, contempt or anger.

10. A three-dimensional face personalized expression generation system based on speech recognition, characterized in that: The method for generating personalized expressions of three-dimensional human faces based on speech recognition as claimed in any one of claims 1 to 9 comprises: A data preprocessing unit, used to obtain a voice signal input by a user and perform data preprocessing on the voice signal to obtain voice input data, wherein the data preprocessing includes performing noise filtering, data enhancement and timbre migration on the voice signal; An emotion discrimination unit is used to extract features from the speech input data using a speech model HuBERT to obtain speech feature information, and input the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data; An expression data generating unit, used for inputting a trained basic expression generation model according to the speech feature information and the emotion classification result for training to obtain expression data, wherein the basic expression generation model is obtained by training a diffusion model; A personalized expression generation unit is used to perform personalized expression fine-tuning on the expression data and preset personalized features to obtain a personalized expression generation model, and generate a personalized emotional expression corresponding to the voice signal based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

Images