A method, system, apparatus and storage medium for voice driven images
By extracting speech features and predicting lip movements, and using convolutional neural networks and transformer neural networks to generate 3D character lip expression animations, the problems of high manpower and time costs and poor stability in existing technologies are solved, and efficient and stable 3D animation generation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU SAILINGLI TECH CO LTD
- Filing Date
- 2023-03-30
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies require significant manpower and time to create 3D character lip-syncing animations, and their stability is poor.
By using a speech feature extraction model and a lip-shape expression prediction model, audio feature vectors are extracted using convolutional neural networks and bidirectional long short-term memory networks, and lip-shape expression offsets are predicted using a transformer neural network to generate 3D facial lip-shape expression animations.
It enables efficient and stable generation of 3D animations including lip shapes and facial expressions, reducing manpower and time costs and improving generation efficiency and stability.
Smart Images
Figure CN116597857B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, and particularly relates to a voice-driven image method, system, device and storage medium. BACKGROUND
[0002] With the continuous growth of the diversity of 3D video content and the rapid development of digital virtual human application scenarios, higher quality and higher efficiency are required for the creation of 3D digital virtual human related content. The lip shape and facial expression of the 3D digital virtual human generated by rapid output can help the audience understand the dialogue content more vividly. The dual-modal information fusion expression mode of visual animation and auditory sound can not only improve the understanding of the user to the content, but also provide a more accurate experience in the scene that needs to be interacted, and improve the artistic and ornamental degree of the 3D virtual digital human.
[0003] The current technical scheme for making 3D character lip expression animation includes the following types: the first is to listen to the audio content by a professional animator, and to make key frame animation matching the sound and character lip shape by manual method; the second is to capture the facial lip expression of a professional actor by a motion capture device, and to make secondary adjustment to the captured data by manual method, and finally to import a rendering engine to drive the character facial lip expression movement. The above two schemes need to consume a large amount of human and time cost, and different people and devices have an impact on the stability of the final output. SUMMARY
[0004] Therefore, the embodiment of the present application provides a voice-driven image method, system, device and storage medium, which can generate a three-dimensional animation containing lip shape and expression according to input voice-driven image, has high efficiency and good stability.
[0005] In a first aspect, the embodiment of the present application provides a voice-driven image method, comprising the following steps:
[0006] Obtaining audio data, and extracting an audio feature vector corresponding to the audio data by a voice feature extraction model;
[0007] Predicting the audio feature vector by a lip expression prediction model and a determined expression emotion vector to obtain a lip expression offset sequence;
[0008] Obtaining a three-dimensional face basic model, and performing synthesis processing on the three-dimensional face basic model and the lip expression offset sequence to obtain a three-dimensional face lip expression animation.
[0009] Optionally, the speech feature extraction model comprises a convolutional neural network and a bidirectional long short-term memory network, and the audio feature vector corresponding to the audio data is extracted through the speech feature extraction model, specifically comprising:
[0010] The one-dimensional vector corresponding to the audio data is input into the convolutional neural network to obtain high-level speech features;
[0011] The high-level speech features are input into the bidirectional long short-term memory network to obtain the audio feature vector.
[0012] Optionally, the training process of the speech feature extraction model comprises:
[0013] Obtain speech sample data and corresponding real speech sample feature vectors;
[0014] Input the speech sample data into an initial model to extract predicted speech sample feature vectors;
[0015] Adjust the model parameters of the initial model according to the error between the predicted speech sample feature vectors and the real speech sample feature vectors until the error between the predicted speech sample feature vectors output by the initial model and the real speech sample feature vectors meets the training requirements, and obtain the speech feature extraction model.
[0016] Optionally, the lip expression prediction model comprises a transformer neural network model, and the transformer neural network model comprises an encoder network and a decoder network, and the audio feature vector is predicted through the lip expression prediction model and a determined expression emotion vector to obtain a lip expression offset sequence, specifically comprising:
[0017] Input the audio feature vector into the encoder network to obtain an audio information representation vector sequence;
[0018] Input the audio information representation vector sequence and the determined expression emotion vector into the decoder network to obtain the lip expression offset sequence.
[0019] Optionally, the training process of the lip expression prediction model comprises:
[0020] Obtain video sample data of multiple perspectives of a speaker, establish a three-dimensional point cloud face sequence according to the video data, and determine real face lip expression offsets according to the three-dimensional point cloud face sequence;
[0021] Extract speech sample data from the video sample data, and match and label the three-dimensional point cloud face sequence and the speech sample data to form a sample data pair;
[0022] The speech sample data in the sample data pair is input into the encoder network to obtain the audio sample information representation vector;
[0023] The audio sample information representation vector, the 3D point cloud face sequence in the sample data pair, and the randomly generated expression vector are input into the decoder network to obtain the predicted face lip shape expression offset.
[0024] The loss value between the actual facial lip shape shift and the predicted facial lip shape shift is calculated based on the target loss function. The encoder network, decoder network and target loss function are then updated based on the loss value to obtain the transformer neural network model.
[0025] Optionally, the formula for calculating the target loss function is as follows:
[0026] Loss=S l ×L lip +S f ×L face +S r ×L reg
[0027] Where Loss represents the loss value, L lip S represents the loss value of the lip region. l L represents the influence coefficient of the lip region. face S represents the loss value of the facial expression region outside the lip area. f L represents the influence coefficient of facial expression regions outside the lip area. reg S represents the loss value of the expression regularization term. r This represents the influence coefficient of the facial expression regularization term.
[0028] Optionally, the facial expression vector is obtained in the following way:
[0029] The facial emotion vector learned during the training of the lip-shape prediction model is determined as the facial emotion vector.
[0030] Alternatively, obtain facial expression information and determine the facial emotion vector based on the facial expression information.
[0031] In a second aspect, embodiments of the present invention provide a voice-driven image system, comprising:
[0032] The first module is used to acquire audio data and extract the audio feature vector corresponding to the audio data through a speech feature extraction model.
[0033] The second module is used to predict the audio feature vector using a lip-shape expression prediction model and a determined facial emotion vector to obtain a lip-shape expression offset sequence.
[0034] The third module is used to obtain a three-dimensional facial base model, and to synthesize the three-dimensional facial base model and the lip expression offset sequence to obtain a three-dimensional facial lip expression animation.
[0035] Thirdly, embodiments of the present invention provide a speech-driven image apparatus, comprising:
[0036] At least one processor;
[0037] At least one memory for storing at least one program;
[0038] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.
[0039] Fourthly, embodiments of the present invention provide a storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the methods described above.
[0040] The implementation of this invention includes the following beneficial effects: This embodiment extracts audio feature vectors corresponding to audio data through a speech feature extraction model, so that the lip-shape expression prediction model can adapt to different languages. Then, the lip-shape expression prediction model and the determined expression emotion vector are used to predict the audio feature vectors to obtain the lip-shape expression offset sequence, thereby obtaining the changes in lip shape and surface. Then, based on the three-dimensional face base model and the lip-shape expression offset sequence, a three-dimensional face lip-shape expression animation is obtained, thereby realizing the generation of three-dimensional animation containing lip shape and expression based on speech-driven images, which is highly efficient and stable. Attached Figure Description
[0041] Figure 1 This is a flowchart illustrating the steps of a voice-driven image method provided in an embodiment of the present invention;
[0042] Figure 2 This is a flowchart illustrating the steps of another voice-driven image method provided in an embodiment of the present invention;
[0043] Figure 3 This is a structural block diagram of a language feature extraction model provided in an embodiment of the present invention;
[0044] Figure 4 This is a structural block diagram of a lip-shape expression prediction model provided in an embodiment of the present invention;
[0045] Figure 5 This is a structural block diagram of a voice-driven image system provided in an embodiment of the present invention;
[0046] Figure 6This is a structural block diagram of a voice-driven image device provided in an embodiment of the present invention. Detailed Implementation
[0047] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
[0048] See Figure 1 and Figure 2 As shown, this embodiment of the invention provides a method for voice-driven image processing, including the following steps:
[0049] S100. Acquire audio data and extract the audio feature vector corresponding to the audio data through a speech feature extraction model.
[0050] Audio data refers to speech data used to drive images. Speech data includes various languages, such as Chinese or English. Audio feature vectors are used to characterize the audio features of language data. Speech feature extraction models are used to obtain output audio feature vectors from input audio data.
[0051] Those skilled in the art will understand that the specific type of audio feature vector is determined according to the actual application, and this embodiment does not impose specific limitations. For example, the audio feature vector is a PPG (phonetic posterior grams) feature vector. PPG audio feature vectors can extract richer audio feature information, thereby improving the adaptability to different languages in the subsequent process of predicting 3D facial lip movements through speech.
[0052] It should be noted that the specific structure of the speech feature extraction model is determined based on the actual application, and this embodiment does not impose specific limitations. See also Figure 3In a specific implementation, the speech feature extraction model includes a Convolutional Neural Network (CNN) and a Bi-directional Long Short-Term Memory (BiLSTM) network. The input to the speech feature extraction model is the speech signal, and the output is the corresponding speech feature vector. The speech signal is a one-dimensional vector obtained by sampling audio data at certain time intervals. Specifically, the speech signal is input into a 1D-CNN (one-dimensional convolutional neural network), passing through three 1D-CNN network layers and a pooling layer to extract high-level speech features. Then, the output of the CNN layer is used as input, and the BiLSTM captures the time-series information of the audio signal to further extract speech features. The final network layer uses a fully connected layer as the output layer, mapping the output of the BiLSTM layer to the PPG feature vector.
[0053] Optionally, the training process of the speech feature extraction model includes:
[0054] S101. Obtain speech sample data and the corresponding real speech sample feature vectors;
[0055] S102. Input the speech sample data into the initial model and extract the predicted speech sample feature vector;
[0056] S103. Based on the error between the predicted speech sample feature vector and the real speech sample feature vector, adjust the model parameters of the initial model until the error between the predicted speech sample feature vector and the real speech sample feature vector output by the initial model meets the training requirements, thereby obtaining the speech feature extraction model.
[0057] Speech sample data includes sample data from multiple languages, and the feature vectors of real speech samples are the feature vectors of the speech sample data. The initial model refers to the speech feature extraction model whose parameters are yet to be determined. Specifically, the speech sample data is first input into the speech feature extraction model to obtain predicted speech sample feature vectors. Then, the model parameters of the initial model are adjusted based on the error between the predicted and real speech sample feature vectors. During modulation, the error between the predicted and real speech sample feature vectors is reduced. When the error between the predicted and real speech sample feature vectors output by the initial model meets the training requirements, the initial model with the corresponding model parameters is used as the speech feature extraction model.
[0058] Optionally, the speech feature extraction model includes a convolutional neural network and a bidirectional long short-term memory network. The step of extracting the audio feature vector corresponding to the audio data using the speech feature extraction model specifically includes:
[0059] S110. Input the one-dimensional vector corresponding to the audio data into the convolutional neural network to obtain high-level speech features;
[0060] S120. Input the high-level speech features into the bidirectional long short-term memory network to obtain the audio feature vector.
[0061] Specifically, see Figure 3 First, the one-dimensional vector corresponding to the audio data is input into the CNN (Convolutional Neural Network) in the speech feature extraction model, and high-level speech features are extracted. Then, the high-level speech features are input into the BiLSTM (Bidirectional Long Short Memory Network) in the speech feature extraction model, and audio feature vectors are extracted.
[0062] S200. The audio feature vector is predicted using the lip-shape expression prediction model and the determined expression emotion vector to obtain the lip-shape expression offset sequence.
[0063] A lip-shape expression prediction model is used to predict lip-shape shift sequences based on audio feature vectors and facial emotion vectors. The lip-shape shift characterizes the degree of deviation between the lip shape and expression based on point clouds of silent and expressionless faces. It should be noted that the lip-shape expression prediction model is determined according to the actual application, and this embodiment does not impose specific limitations. In a specific embodiment, see [reference needed]. Figure 4 The lip expression prediction model includes an encoder and a decoder. The encoder includes a forward propagation layer, several overlapping multi-head self-attention mechanisms, and a linear projection layer. The decoder includes a forward propagation layer, a multi-head self-attention mechanism, and a linear projection layer. The input of the encoder is an audio feature vector, and the output of the encoder is the input of the decoder. The output of the decoder is a 3D facial lip expression offset value.
[0064] Optionally, the training process of the lip-shape expression prediction model includes:
[0065] S201. Acquire video sample data from multiple perspectives of the speaker, establish a three-dimensional point cloud face sequence based on the video data, and determine the actual lip shape expression offset of the face based on the three-dimensional point cloud face sequence.
[0066] The video sample data includes multi-view video data of different people speaking, acquired through a multi-view array camera. The audio data in the videos consists of mixed audio from multiple speakers and multiple languages, while the image data comprises multi-view facial data of different people speaking. The video resolution for each viewpoint is preferably above 1080p. Each frame of the acquired video's multi-view facial data is reconstructed using 3D point cloud alignment to obtain a 3D point cloud face sequence. Simultaneously, a 3D face model of a person speaking in a natural, silent state (with mouth closed) is selected from the reconstructed data and saved as the base model for natural facial expression. The offset between the 3D point cloud face sequence and the base model for natural facial expression is used as the offset for the real face's lip shape.
[0067] S202. Extract the audio sample data from the video sample data, and match and label the three-dimensional point cloud face sequence with the audio sample data to form a sample data pair.
[0068] The audio data from the video sample data is extracted as audio sample data. The speaker's 3D face point cloud sequence is matched and labeled with the corresponding audio. Each audio segment is labeled with the corresponding speaker's 3D face point cloud sequence. The data is segmented into audio and speaker's 3D face point cloud matching data pairs through labeling, and finally the sequence segments are formed and saved.
[0069] S203. Input the speech sample data from the sample data pair into the encoder network to obtain the audio sample information representation vector.
[0070] See Figure 4 The audio sample feature vectors corresponding to the speech sample data in the sample data pairs are extracted, and then input into the encoder network to obtain the audio sample information representation vector. It should be noted that the number of overlapping multi-head self-attention mechanisms and forward propagation layers in the encoder network is determined according to the actual application, and this embodiment does not impose specific limitations. For example, the number of overlapping multi-head self-attention mechanisms and forward propagation layers is 5.
[0071] S204. Input the audio sample information representation vector, the 3D point cloud face sequence in the sample data pair, and the randomly generated expression emotion vector into the decoder network to obtain the predicted face lip shape expression offset.
[0072] During training, the facial expression emotion vector is an N-dimensional vector sampled from a random Gaussian distribution. Since the training data includes vocal expression data pairs with different emotions from different speakers, this emotion-laden training data is input into the lip-sync prediction model. Through training, the loss function is calculated, and backpropagation automatically learns the facial expression emotion vectors implied by different emotions. Finally, the facial expression emotion vectors learned from different emotion data are combined to form an facial expression emotion vector matrix and saved. (See also...) Figure 4The encoder inputs the audio sample information representation vector, the 3D point cloud face sequence in the sample data pair, and the randomly generated expression emotion vector into the decoder network. The output of the decoder network is the predicted lip shape expression offset.
[0073] S205. Calculate the loss value between the actual facial lip shape shift and the predicted facial lip shape shift according to the target loss function, and update the encoder network, decoder network and target loss function according to the loss value to obtain the transformer neural network model.
[0074] The target loss function is used to calculate the error between the model's predicted value and the true target value. The specific form of the target loss function is determined according to the actual application, and this embodiment does not impose specific restrictions. The smaller the loss value calculated by the target loss function, the more accurate the model parameters of the obtained transformer neural network model.
[0075] Optionally, the formula for calculating the target loss function is as follows:
[0076] Loss=S l ×L lip +S f ×L face +S r ×L reg
[0077] Where Loss represents the loss value, L lip S represents the loss value of the lip region. l L represents the influence coefficient of the lip region. face S represents the loss value of the facial expression region outside the lip area. f L represents the influence coefficient of facial expression regions outside the lip area. reg S represents the loss value of the expression regularization term. r This represents the influence coefficient of the facial expression regularization term.
[0078] Specifically, the influence coefficient S of the lip region l The value of the influence coefficient S on the facial expression area outside the lip area f The value of S is adjusted according to the weight in the actual application. l With S f After adjustment, the influence coefficient S of the facial expression regularization term is also adjusted. r This allows the model to focus on facial expression changes over a longer period during training, avoiding drastic changes in the model's predicted expressions within a short timeframe, resulting in more natural facial expressions. During training, the model iteratively minimizes the loss value of the target loss function while adjusting S... l S f and Sr The coefficients enable the generation of more accurate and natural 3D facial lip expression animations.
[0079] Optionally, the lip-shape expression prediction model includes a transformer neural network model, which comprises an encoder network and a decoder network. The step of predicting the audio feature vector using the lip-shape expression prediction model and a determined expression emotion vector to obtain a lip-shape expression offset sequence specifically includes:
[0080] S210. Input the audio feature vector into the encoder network to obtain an audio information representation vector sequence;
[0081] S220. Input the audio information representation vector sequence and the determined facial expression vector into the decoder network to obtain the lip shape expression offset sequence.
[0082] The encoder network is mainly used to encode and extract context-related audio representation information from audio features. The input data is the audio feature vector, and the output is the context-related audio information representation vector. The decoder network is used to decode the context-related audio information representation vector and the facial expression vector output by the encoder network. The input of the decoder network is the context-related audio information representation vector output by the encoder network, the 3D point cloud face, and the determined facial expression vector. The output of the decoder network is the lip expression offset sequence.
[0083] Optionally, the facial expression vector is obtained in the following way:
[0084] S221. The facial emotion vector learned during the training of the lip-shape expression prediction model is determined as the facial emotion vector.
[0085] S222, or, obtain facial expression information and determine the facial expression emotion vector based on the facial expression information.
[0086] Specifically, during the prediction process, the facial expression and emotion vector input to the decoder network can be the facial expression and emotion vector already learned during training, or a new facial expression and emotion vector can be formed by the linear superposition and combination of multiple facial expression and emotion vectors as input, controlling the emotion in the output 3D face lip expression vertex animation.
[0087] S300. Obtain a three-dimensional facial basic model, and synthesize the three-dimensional facial basic model and the lip expression offset sequence to obtain a three-dimensional facial lip expression animation.
[0088] A 3D face base model represents a silent, expressionless 3D face model, while lip shape offsets represent the vertex offsets of the lip shape and expression on the 3D face. By overlaying the 3D face base model and the lip shape offset sequence, a 3D face lip expression animation is obtained.
[0089] The implementation of this invention includes the following beneficial effects: This embodiment extracts audio feature vectors corresponding to audio data through a speech feature extraction model, so that the lip-shape expression prediction model can adapt to different languages. Then, the lip-shape expression prediction model and the determined expression emotion vector are used to predict the audio feature vectors to obtain the lip-shape expression offset sequence, thereby obtaining the changes in lip shape and surface. Then, based on the three-dimensional face base model and the lip-shape expression offset sequence, a three-dimensional face lip-shape expression animation is obtained, thereby realizing the generation of three-dimensional animation containing lip shape and expression based on speech-driven images, which is highly efficient and stable.
[0090] See Figure 5 This invention provides a voice-driven image system, comprising:
[0091] The first module is used to acquire audio data and extract the audio feature vector corresponding to the audio data through a speech feature extraction model.
[0092] The second module is used to predict the audio feature vector using a lip-shape expression prediction model and a determined facial emotion vector to obtain a lip-shape expression offset sequence.
[0093] The third module is used to obtain a three-dimensional facial base model, and to synthesize the three-dimensional facial base model and the lip expression offset sequence to obtain a three-dimensional facial lip expression animation.
[0094] It is evident that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0095] See Figure 6 This invention provides a voice-driven image apparatus, comprising:
[0096] At least one processor;
[0097] At least one memory for storing at least one program;
[0098] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.
[0099] It is evident that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented in this device embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0100] Furthermore, this application also discloses a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the described method. Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0101] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A method for voice-driven image processing, characterized in that, include: Acquire audio data and extract the audio feature vector corresponding to the audio data using a speech feature extraction model; The audio feature vector is predicted using a lip-shape prediction model and a determined facial expression vector to obtain a lip-shape offset sequence. A three-dimensional facial base model is obtained, and the three-dimensional facial base model and the lip expression offset sequence are synthesized to obtain a three-dimensional facial lip expression animation. The lip-shape expression prediction model includes a transformer neural network model, which comprises an encoder network and a decoder network. The step of predicting the audio feature vector using the lip-shape expression prediction model and a determined expression emotion vector to obtain a lip-shape expression offset sequence specifically includes: The audio feature vector is input into the encoder network to obtain an audio information representation vector sequence; The audio information representation vector sequence and the determined facial expression vector are input into the decoder network to obtain the lip shape expression offset sequence; The calculation formula for the target loss function in the training process of the lip-shape expression prediction model is as follows: in, Loss Indicates the loss value. L lip This represents the loss value in the lip region. S l This represents the influence coefficient of the lip region. L face This represents the loss value for the facial expression region outside the lip area. S f This represents the influence coefficient of facial expression regions outside the lip area. L reg This represents the loss value of the facial expression regularization term. S r This represents the influence coefficient of the facial expression regularization term.
2. The method according to claim 1, characterized in that, The speech feature extraction model includes a convolutional neural network and a bidirectional long short-term memory network. The extraction of the audio feature vector corresponding to the audio data using the speech feature extraction model specifically includes: The one-dimensional vector corresponding to the audio data is input into the convolutional neural network to obtain high-level speech features; The high-level speech features are input into the bidirectional long short-term memory network to obtain audio feature vectors.
3. The method according to claim 2, characterized in that, The training process of the speech feature extraction model includes: Obtain speech sample data and the corresponding real speech sample feature vectors; The speech sample data is input into the initial model to extract the feature vector of the predicted speech sample. Based on the error between the predicted speech sample feature vector and the real speech sample feature vector, the model parameters of the initial model are adjusted until the error between the predicted speech sample feature vector and the real speech sample feature vector output by the initial model meets the training requirements, thus obtaining the speech feature extraction model.
4. The method according to claim 1, characterized in that, The training process of the lip-shape expression prediction model includes: Acquire video sample data from multiple perspectives of the speaker, establish a three-dimensional point cloud face sequence based on the video sample data, and determine the real face lip expression offset based on the three-dimensional point cloud face sequence. Extract audio sample data from the video sample data, and match and label the 3D point cloud face sequence with the audio sample data to form sample data pairs; The speech sample data in the sample data pair is input into the encoder network to obtain the audio sample information representation vector; The audio sample information representation vector, the 3D point cloud face sequence in the sample data pair, and the randomly generated expression vector are input into the decoder network to obtain the predicted face lip shape expression offset. The loss value between the actual facial lip shape shift and the predicted facial lip shape shift is calculated based on the target loss function. The encoder network, decoder network and target loss function are then updated based on the loss value to obtain the transformer neural network model.
5. The method according to claim 1, characterized in that, The facial expression vector is obtained through the following method: The facial emotion vector learned during the training of the lip-shape prediction model is determined as the facial emotion vector. Alternatively, obtain facial expression information and determine the facial emotion vector based on the facial expression information.
6. A voice-driven image system, characterized in that, include: The first module is used to acquire audio data and extract the audio feature vector corresponding to the audio data through a speech feature extraction model. The second module is used to predict the audio feature vector using a lip-shape prediction model and a determined emotional expression vector to obtain a lip-shape offset sequence. The lip-shape prediction model includes a transformer neural network model, which includes an encoder network and a decoder network. Specifically, it includes: inputting the audio feature vector into the encoder network to obtain an audio information representation vector sequence; and inputting the audio information representation vector sequence and the determined emotional expression vector into the decoder network to obtain a lip-shape offset sequence. The third module is used to obtain a three-dimensional facial base model, and to synthesize the three-dimensional facial base model and the lip expression offset sequence to obtain a three-dimensional facial lip expression animation. The calculation formula for the target loss function in the training process of the lip-shape expression prediction model is as follows: in, Loss Indicates the loss value. L lip This represents the loss value in the lip region. S l This represents the influence coefficient of the lip region. L face This represents the loss value for the facial expression region outside the lip area. S f This represents the influence coefficient of facial expression regions outside the lip area. L reg This represents the loss value of the facial expression regularization term. S r This represents the influence coefficient of the facial expression regularization term.
7. A device for voice-driven image processing, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-5.
8. A storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-5.