An apparatus and method for generating speech animation from an audio signal

Abstract

The application discloses a device and method for generating speech animation from an audio signal, comprising: receiving an audio signal; and converting the received audio signal into a frequency domain audio feature; inputting the frequency domain audio feature into a neural network to perform a neural network processing procedure to identify phonemes, wherein the neural network is trained using a phoneme data set, the audio data set comprising audio signals with corresponding accurate phoneme labels; and generating speech animation from the identified phonemes.

Description

Technical Field

[0001] This invention relates to the field of animation generation, and more particularly to an apparatus and method for generating voice-over animation. Background Technology

[0002] Virtual characters are now incredibly popular in streaming video, games, and virtual reality-related applications. These characters interact with each other in the virtual world, and sometimes they even interact with viewers or players in the real world. Humans are extremely sensitive to any unnatural facial features, as well as uncoordinated or asynchronous behavior in virtual characters, making facial animation, especially voice animation, very challenging because animation involves both sound and mouth movements.

[0003] In real-world voice-over animation, artists seek the most immersive experience between themselves and highly anthropomorphic virtual characters. However, the high costs of production and the massive data requirements—including audio, video, and possibly 3D models—significantly limit its scalability. Take, for example, online meetings using avatars. In applications where realism is less critical but scalability is crucial, audio data may be the only available medium. This necessitates that the virtual character function correctly through mouth movements that seamlessly match the voice input—what we call believable voice-over animation. "Believable" voice-over animation requires algorithms, within realistic resource constraints, to serve all possible virtual faces with various 3D models and generate synthetic videos with sufficient realism to make the user at the remote end feel comfortable.

[0004] Real-time speech animation is a compelling yet challenging topic, where multimodal inputs (such as audio, video, or depth information) can be used to maximize performance. However, when audio is the only available input, the quality of the results largely depends on real-time phoneme recognition, such as accuracy and latency, due to resource constraints or privacy concerns. Balancing latency and accuracy is crucial for real-time applications. The current challenge is to find a phoneme recognition solution that enables real-time animation to achieve the highest accuracy with low latency. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a phoneme recognition solution to enable real-time animation to achieve the highest accuracy with low latency.

[0006] This invention discloses a method for generating speech animation from audio signals. The method includes: receiving an audio signal; converting the received audio signal into frequency domain audio features; inputting the frequency domain audio features into a neural network to perform a neural network processing procedure to identify phonemes, wherein the neural network is trained with a phoneme dataset to perform the neural network processing procedure, the phoneme dataset including audio signals with corresponding accurate phoneme annotations; and generating speech animation based on the identified phonemes.

[0007] In one implementation of this embodiment, converting the audio signal into frequency domain audio features includes: calculating the Mel-frequency cepstral coefficients (MFCCs) of the audio signal; calculating the first-order derivative components and the second-order derivative components of the MFCCs, and combining the first-order derivative components and the second-order derivative components into a single audio feature frame; generating the frequency domain audio features based on the single audio feature frame, the forward context vector of the single audio feature frame, and the backward context vector of the single audio feature frame.

[0008] In one implementation of this embodiment, the step of inputting the frequency domain audio features into a neural network to perform neural network processing to identify phonemes includes: inputting the frequency domain audio features into a convolutional neural network (CNN) filter to generate multiple sets of intermediate features, each set of intermediate features corresponding to a CNN output channel; inputting the intermediate features into multiple long short-term memory (LSTM) networks for parallel temporal modeling to generate LSTM stack block (LSB) output features, wherein each LSTM network inputs a set of intermediate features to generate a set of LSB output features; inputting the LSB output features into a unified LSTM network for unified temporal modeling to generate unified LSTM output features; inputting the unified LSTM output features into a fully connected layer (FCL) to generate a hidden Markov model (HMM) shared state; and inputting the HMM shared state into an HMM triphone decoder to generate identified phoneme annotations.

[0009] In one implementation of this embodiment, the method further includes: storing the audio features in a first storage buffer; storing the HMM shared state in a second storage buffer; and storing the identified phonemes in a third storage buffer; wherein the first storage buffer, the second storage buffer, and the third storage buffer are fixed-size first-in-first-out (FIFO) buffers.

[0010] In one implementation of this embodiment, the method further includes: generating a first combined feature based on the single audio feature frame and the LSB output feature; and inputting the first combined feature into the unified LSTM network to generate the unified LSTM output feature.

[0011] In one implementation of this embodiment, the method further includes: combining the LSB output features with the unified LSTM output features to generate a second combined feature; and inputting the second combined feature into the FCL to generate the HMM shared state.

[0012] In one implementation of this embodiment, generating a speech animation based on the identified phonemes includes: generating a two-dimensional (2D) visual pixel domain, dividing the 2D visual pixel domain into multiple blocks, wherein each identified phoneme corresponds to a block of the 2D visual pixel domain.

[0013] Another aspect of the present invention discloses an apparatus for generating speech animation from audio signals. The apparatus includes a memory storing computer-executable instructions; a processor coupled to the memory, and, when executing the computer-executable instructions, configured to: receive the audio signal; convert the received audio signal into frequency-domain audio features; input the frequency-domain audio features into a neural network to perform a neural network processing procedure to identify phonemes, wherein the neural network is trained with a phoneme dataset to perform the neural network processing procedure, the phoneme dataset comprising audio signals with corresponding accurate phoneme annotations; and generate speech animation based on the identified phonemes.

[0014] In one implementation of this embodiment, the processor is further configured to: calculate the Mel-frequency cepstral coefficients (MFCCs) of the audio signal; calculate the first-order derivative components and the second-order derivative components of the MFCCs, and combine the first-order derivative components and the second-order derivative components into a single audio feature frame; and generate the frequency domain audio features based on the single audio feature frame, the forward context vector of the single audio feature frame, and the backward context vector of the single audio feature frame.

[0015] In one implementation of this embodiment, the processor is further configured to: input the frequency domain audio features into a convolutional neural network (CNN) filter to generate multiple sets of intermediate features, each set of intermediate features corresponding to a CNN output channel; input the intermediate features into multiple long short-term memory (LSTM) networks for parallel temporal modeling to generate LSTM stack block (LSB) output features, wherein each LSTM network inputs a set of intermediate features to generate a set of LSB output features; input the LSB output features into a unified LSTM network for unified temporal modeling to generate unified LSTM output features; input the unified LSTM output features into a fully connected layer (FCL) to generate a hidden Markov model (HMM) shared state; and input the HMM shared state into an HMM triphone decoder to generate identified phoneme annotations.

[0016] In one implementation of this embodiment, the processor is further configured to: store the audio features in a first storage buffer; store the HMM shared state in a second storage buffer; and store the identified phonemes in a third storage buffer; wherein the first storage buffer, the second storage buffer, and the third storage buffer are fixed-size first-in-first-out (FIFO) buffers.

[0017] In one implementation of this embodiment, the processor is further configured to: combine the single audio feature frame with the LSB output feature to generate a first combined feature; and input the first combined feature into the unified LSTM network to generate the unified LSTM output feature.

[0018] In one implementation of this embodiment, the processor is further configured to: combine the LSB output features with the unified LSTM output features to generate a second combined feature; and input the second combined feature into the FCL to generate the HMM shared state.

[0019] In one implementation of this embodiment, the processor is further configured to: generate a two-dimensional (2D) pixel domain, divide the 2D pixel domain into multiple blocks, wherein each identified phoneme corresponds to a block of the 2D pixel domain.

[0020] Another aspect of the present invention discloses a non-transitory computer-readable storage medium. The storage medium stores a plurality of instructions, wherein, when a processor executes the instructions, the instructions cause the processor to: receive the audio signal; convert the received audio signal into frequency domain audio features; input the frequency domain audio features into a neural network to perform a neural network processing procedure to identify phonemes, wherein the neural network processing procedure is performed using the neural network trained with a phoneme dataset, the phoneme dataset comprising audio signals with corresponding accurate phoneme annotations; and generate a speech animation based on the identified phonemes.

[0021] In one implementation of this embodiment, the instructions further instruct the processor to: calculate the Mel-frequency cepstral coefficients (MFCCs) of the audio signal; calculate the first-order derivative components and the second-order derivative components of the MFCCs, and combine the first-order derivative components and the second-order derivative components into a single audio feature frame; and generate the frequency domain audio features based on the single audio feature frame, the forward context vector of the single audio feature frame, and the backward context vector of the single audio feature frame.

[0022] In one implementation of this embodiment, the instructions further instruct the processor to: input the frequency domain audio features into a convolutional neural network (CNN) filter to generate multiple sets of intermediate features, each set of intermediate features corresponding to a CNN output channel; input multiple long short-term memory (LSTM) networks for parallel temporal modeling to generate LSTM stack block (LSB) output features, wherein each LSTM network inputs a set of intermediate features to generate a set of LSB output features; input the LSB output features into a unified LSTM network for unified temporal modeling to generate unified LSTM output features; input the unified LSTM output features into a fully connected layer (FCL) to generate a hidden Markov model (HMM) shared state; and input the HMM shared state into an HMM triphone decoder to generate identified phoneme annotations.

[0023] In one implementation of this embodiment, the instructions further instruct the processor to: store the audio features in a first storage buffer; store the HMM shared state in a second storage buffer; and store the identified phonemes in a third storage buffer; wherein the first storage buffer, the second storage buffer, and the third storage buffer are fixed-size first-in-first-out (FIFO) buffers.

[0024] In one implementation of this embodiment, the instructions further instruct the processor to: combine the single audio feature frame with the LSB output feature to generate a first combined feature; and input the first combined feature into the unified LSTM network to generate the unified LSTM output feature.

[0025] In one implementation of this embodiment, the instructions further instruct the processor to: combine the LSB output features with the unified LSTM output features to generate a second combined feature; and input the second combined feature into the FCL to generate the HMM shared state.

[0026] Compared with the prior art, the present invention provides an apparatus and method for generating speech animation from audio signals and a non-transitory computer-readable storage medium. The apparatus and method can generate real-time reliable speech animation and are simple to execute, thereby enabling real-time animation to achieve the highest accuracy with low latency. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments are briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention. Based on these drawings, those skilled in the art can obtain other drawings.

[0028] Figure 1This is a schematic diagram of the voice animation generation device in an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of a speech animation generation method for generating speech animation from input audio signals in an embodiment.

[0030] Figure 3 This is a schematic diagram summarizing a speech animation generation method that generates speech animation from an input audio signal, where the input audio signal corresponds to the phoneme stream {S, OW, IY} of the word / slowly / .

[0031] Figure 4 This is a schematic diagram of the phoneme recognition process with a buffer in an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the audio feature transformation time consumption distribution in an embodiment of the present invention;

[0033] Figure 6 Example diagrams showing the visual pixels corresponding to the phoneme at different frames during the pronunciation of the phoneme / o / ;

[0034] Figure 7 This is an example diagram of phoneme selection for generating 2D visual pixel domain animation curves in an embodiment of the present invention;

[0035] Figure 8 This is a schematic diagram illustrating the division of a 2D view pixel domain into 20 blocks using a block index.

[0036] Figure 9 This is a schematic diagram of the overall network architecture of RealPRNet in the embodiment;

[0037] Figure 10 This is a schematic diagram of the audio features output from the CNN output channel in the embodiment.

[0038] Figure 11A This is a schematic diagram of the process in the LSB in the embodiment;

[0039] Figure 11B This is a schematic diagram of the process in the LSB in another embodiment;

[0040] Figure 12 This is a schematic diagram illustrating the steps of the neural network processing for predicting phonemes based on audio features in the embodiment.

[0041] Figure 13 This is a diagram comparing the real-time performance of LSTM, CLDNN, and RealPRNet.

[0042] Figure 14 A diagram illustrating the performance comparison of phoneme prediction methods based on phoneme error rates;

[0043] Figure 15This is a diagram illustrating the performance comparison of phoneme prediction methods based on block distance error.

[0044] Figure 16 A visual comparison diagram of speech animation generation methods; and

[0045] Figure 17 This is a schematic diagram of the output phoneme buffer occupancy rate in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of embodiments of the present invention are described below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0047] Recent works have demonstrated that believable speech animation is achievable. In some works, the audio input is divided into small segments called frames, and basic frequency features are extracted from each frame. Phonemes are perceptually distinct units in the phonation of a particular language, distinguishing words from one another. Phonemes are predicted by identifying vowels and basic fricative consonants from these features, and then these phonemes are mapped to corresponding static animations (called visual pixels, i.e., phoneme copies in the visual domain). In this frame-based processing mechanism, latency is negligible, as it is almost equal to the processing time of a single frame. However, the lack of consideration for neighborhood contextual information in this process means the system can only recognize basic phonemes, greatly limiting the accuracy of phoneme recognition, which directly leads to low-quality generated animations. In other works, word-based processing mechanisms are employed to achieve higher-quality animations. A forced alignment method is used, corresponding to audio blocks containing multiple words, to extract phoneme transcriptions with high accuracy. The problem of so-called visual pixel co-pronunciation is addressed by considering the natural transition between adjacent phonemes. Furthermore, this method considers a relatively rich set of phonemes (approximately 39 different phonemes) and compares it to the basic phonemes (less than 10 different phonemes) in some works, which directly improves the quality of the generated animation. However, a drawback is that the delay in word-level duration is unacceptable for real-time applications. Therefore, finding a middle ground between frame-based and word-based methods might solve the problem of reliable real-time speech animation, for example, by utilizing a sliding window of frames during processing and a phoneme recognition algorithm with a richer phoneme set and acceptable accuracy.

[0048] Recent advances in deep learning have re-examined the topic of phoneme recognition using novel approaches. Hidden Markov Models (HMMs) (with some works showing an error rate of 26.61%) and HMM / Conditional Random Fields (CRFs) (with some works showing an error rate of 25.24%), based on deep learning neural networks (DNNs), can reduce the error rate for predicting phonemes by 10% compared to traditional models. In some works, a feedforward deep neural network model has been proposed, achieving an error rate of 23.71%. The system uses 39 Mel-frequency cepstral coefficients (MFCCs) from 11 consecutive frames (each standard audio frame is 25ms, with a 15ms overlap) as input to predict each phoneme label. In another work, a feedforward deep neural network architecture has been reported to achieve an even lower error rate of 16.49%, while a standard four-layer Long Short-Term Memory (LSTM) network has achieved an even lower error rate of 15.02%. In some works, a network architecture called CLDNN has been proposed, which combines convolutional neural networks (CNNs), LSTMs, and fully connected DNNs. Compared to LSTM, it can further improve performance by 4-6%.

[0049] For real-time applications, the design principles of Recurrent Neural Networks (RNNs) and LSTMs demonstrate that finding a balance between latency and accuracy is crucial. The temporal correlation between adjacent audio frames plays a significant role in improving recognition accuracy. However, improving phoneme recognition accuracy by adding more adjacent frames to a sliding window comes at the cost of latency. Therefore, the current challenge is to find a phoneme recognition solution that achieves optimal accuracy with low latency in real-time animation.

[0050] This invention discloses a method and apparatus for generating speech animation from audio input containing speech signals. In embodiments, the apparatus and method disclosed herein employ a novel deep network architecture, called RealPRNet (RealPRNet), to address existing problems in generating speech animation based on audio signals. With a network structure design that considers temporal and spatial correlations, RealPRNet can predict phoneme streams for a sliding window of audio frame input. In embodiments, LSTM stack blocks (LSBs) can be used to maximize the learning efficiency of temporal-spatial patterns. The LSBs include multiple parallel LSTM layers. After the CNN filters learn the spatial correlations between frequency features in the input signal, temporal modeling can be performed through two modules: separate temporal modeling via LSBs and unified temporal modeling via unified LSTM layers. In embodiments using RealPRNet, the CNNs are not directly connected to the LSTMs. During training, different CNN filters can learn to reinforce different frequency-temporal patterns in the input features. The LSBs can further utilize temporal correlations and process intermediate output features from different CNN filters separately, the temporal correlations having different reinforced frequency-temporal patterns. Each output feature is fed into a separate LSTM network within the LSB. The outputs from the LSBs are then aggregated and fed into a subsequent unified LSTM network and output DNN layer. These strategies can improve the performance of speech animation generation methods.

[0051] Figure 1 A voice animation generation apparatus 100 according to an embodiment of the present invention is shown. For example... Figure 1 As shown, the voice animation generation device 100 can be used to receive audio signals. In an embodiment, the received audio signal can be a digitized voice signal. The voice audio signal can be obtained from a recording device, loaded from a storage module, or otherwise provided to the voice animation generation device 100. The voice animation generation device 100 can be used to perform a neural network processing procedure to convert the audio input into a recognized phoneme sequence and generate facial animation based on the recognized phonemes. In an embodiment, the parameters of the neural network can be generated through a training process that receives training data containing multiple audio signals and accurate phoneme annotations.

[0052] In an embodiment, the speech animation generation device 100 may be a computing device including a processor 102 and a storage medium 104. The speech animation generation device 100 may also include a display 106, a communication module 108, and additional peripheral devices 112. Some devices may be omitted, while others may be included. The processor 102 may include any suitable processor. In an embodiment, the processor 102 includes multiple cores for multi-threaded or parallel processing. The processor 102 can execute sequences of computer program instructions to implement various processing procedures, such as neural network processing procedures. The storage medium 104 may be a non-transitory computer-readable storage medium and may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), flash memory modules, erasable and rewritable memories, and mass storage devices such as CD-ROM (Compact Optical Disc Read-Only Memory), USB flash drives, and hard disks. The storage medium 104 may store computer programs and instructions for implementing various processing procedures, which, when executed by the processor 102, cause the processor to execute the steps of a neural network processing procedure for a speech animation generation method used to generate speech animation from an input audio signal. The communication module 108 may include a network device that establishes a connection via a network. The display 106 may include any suitable type of computer display device or electronic device display (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) based device, a touchscreen). In an embodiment, the peripheral device 112 may include an audio receiving and recording device, such as a microphone. The peripheral device 112 may further include additional input / output devices, such as a keyboard, mouse, etc. The processor 102 may be used to execute instructions stored on the storage medium 104 and perform various operations of the voice animation generation method as detailed below.

[0053] Figure 2 A speech animation generation method 200 for generating speech animation from input audio signals is illustrated in an embodiment. For example... Figure 3 and Figure 4 The image shows a portion of the speech animation generation method 200 described in this embodiment. In this embodiment, the speech animation generation method 200 may disclose a real-time process, which identifies a phoneme stream from the audio input, maps the phoneme stream to a corresponding parameter stream, and drives a 3D facial model.

[0054] Visual morphologies are visual mouth shapes that are widely used in speech recognition and animation because they can be directly mapped from phonemes (but not vice versa, since multiple phonemes can be mapped to the same visual morphology if they have similar mouth shapes during pronunciation, such as / b / and / p / ). It is important to recognize that, to date, there is no unified standard to regulate visual morphologies; for example, some systems use 20 different visual morphologies, while others use 26, and still others use 16, and so on.

[0055] like Figure 2 As shown, the speech animation generation method 200 may include the following steps as an example. Step S210 is receiving an audio input signal. In an embodiment, the audio input signal may be a digitized speech signal. The audio input signal may be obtained from a recording device, loaded from a storage module, or otherwise provided. In an embodiment, the speech animation generation method 200 may process the audio input signal in real time to generate a real-time speech animation synchronized with the speech signal.

[0056] Step S220 involves converting the audio input signal into frequency domain audio features. When the original audio input signal is received, it is converted into a frequency domain signal. In this embodiment, Mel-frequency cepstral coefficients (MFCCs) are calculated based on the audio input signal. MFCCs are widely used in audio processing. They collectively constitute a Mel-frequency cepstral spectrum (MFC), representing the short-term power spectrum of the audio signal. As can be seen, the human voice is a combination of sound waves at different frequencies. MFCCs can balance the differences in sound variations at different frequencies.

[0057] The audio input signal comprises multiple frames. For example, in one embodiment, the audio input signal has 100 frames per second, and each audio frame is 25 ms long, with a 15 ms overlap. In other embodiments, the number, length, and overlap duration of the audio frames can be varied and are not limited thereto in this invention.

[0058] For each audio frame, the first and second derivative components of MFCCs can be set into a vector representing a single audio feature frame f. The audio feature x at time t... t Audio features f that match time t t and f t Forward and backward context vectors, denoted as [f t-n ,…,f t ,…,f t+mThe value n represents the number of audio frames before time t, and the value m represents the number of audio frames in the future. The integrated audio feature xt is used to predict phoneme labeling at time t. Therefore, the choice of the value m directly results in a latency of 10 ms at a frame rate of 100 frames per second. The larger the value of m, the higher the recognition accuracy, but the longer the latency. When m = 0, future frames are not buffered to avoid introducing additional latency. In any case, the potential advantage of contextual information has been utilized to improve phoneme recognition performance.

[0059] Step S230 involves inputting the frequency domain audio features into a neural network for neural network processing to identify phonemes. In an embodiment, the parameters of the neural network can be generated through a training process that receives training data containing multiple audio signals and accurate phoneme annotations. In an embodiment, a three-phoneme decoder using a RealPRNet and a Hidden Markov Model (HMM) can be used for phoneme recognition. The RealPRNet can utilize different neural networks, such as CNN and LSTM. The output of the RealPRNet can be the shared state H of the HMM, which is used as the input to the decoder to calculate the identified phoneme P.

[0060] To generate smooth animation, the phoneme recognition process may need to output predicted phonemes at constant time intervals of A ms. Therefore, in this embodiment, a buffer mechanism can be employed. Figure 4 The diagram illustrates a phoneme recognition process with a buffer in an embodiment. The process utilizes and... Figure 4 The buffer shown is a fixed-size first-in-first-out (FIFO) buffer.

[0061] like Figure 4 As shown, the audio input signal is converted into frequency domain audio features f at time intervals of 1 ms (equal to the sampling interval). t After that, f t The input feature is stored in the input feature buffer (first storage buffer) B1. The calculation of the conversion process from the raw audio signal to audio features may result in a first delay d1+e1, where d1 is the median calculation time and e1 is the fluctuation time. According to experiments, the time consumption of the frequency domain audio feature transformation process is generally stable and 100% less than A ms. Figure 5 The time consumption distribution of audio feature transformation is shown. For example... Figure 5 As shown, the time consumption of the frequency domain audio feature transformation process is 100% lower than that of AMS (in Figure 5 The experiment shown is 10ms (less).

[0062] The audio features described in B1 can be used to construct an audio feature vector x. t It includes the audio features f at time t.t and f t Forward and backward context vectors, denoted as [f t-n ,…,f t ,…,f t+m The size of B1 depends on two factors: (1) the chosen values ​​m and n, and (2) the time consumption d2+e2 for each RealPRNet prediction, where d2 is the median prediction time and e2 is the prediction fluctuation time. To ensure a smooth output, the following inequality (Formula 1) must be satisfied:

[0063] d² + e² ≤ br·A, br = 1, 2, 3, ... Formula 1

[0064] Where br is the batch size used in the prediction progress, and A is the time interval constant. If br is a small value (i.e., br = 10), the neural network can be run once to predict the br output in parallel with minimal computational increase. This is because when br is small, the data size of the input features is relatively small compared to the model parameters. In the embodiment, the main time consumption is in data parsing and transmission. The RealPRNet acquires audio features from B1 every br·A ms and predicts the br output [h]. t ,h t-1 The predicted outputs can be stored in the HMM shared state buffer (second storage buffer) B2. Based on experiments, the value of br can be configured to 4, which ensures that formula (1) is satisfied in 99% of cases. Figure 5 The distribution of network prediction time consumption (dark gray) obtained from experiments is further illustrated. In some implementations, when there are br sub-buffers in B1, and each sub-buffer is of size m+n+1 and contains audio frame features f... t-n-i ,…,f t-i ,…,f t+m-i Since m forward audio frames are used to construct x, the processing latency may increase to m·A+br·A ms.

[0065] In this embodiment, since the combination of the neural network and the HMM decoder can further improve system performance, the phoneme recognition process does not directly predict phoneme annotations, but instead predicts the HMM shared state. The predicted HMM shared state in B2 can be used as input content for the HMM triphone decoder. For each time interval br·A, the decoder can acquire all the predicted data in B2 and calculate the corresponding phoneme [P]. t ,…,P t-br+1, and store the phoneme in buffer (third storage buffer) B3. The time for the calculation process is d3 + e3 (d3 is the median calculation time, e3 is the fluctuation time of the calculation), and B3 can depend on the number of previous states used to calculate [P t ,…,P t-br+1 . The HMM decoder can calculate the corresponding phoneme P t by using the previously predicted shared state and the current state h together t to improve the prediction result. In an embodiment, the decoder can use only the shared state as a reference, rather than relying on the shared state as in a speech recognition system. This is because there may not be as strong an internal logical relationship between phoneme and phoneme as there is between word and word. Therefore, the decoder can be constructed with a bigram phoneme model that focuses on the acoustic part (signal to phoneme) of the processing process rather than the linguistic part (phoneme to phoneme). According to the experimental results, the calculation time for decoding is usually stable and always less than br·A. If the delay is considered in the above discussion, the total delay from the original audio input signal to the corresponding output phoneme is (m + br)·A + D + e t , where D = d1 + d2 + d3 and e t = e1 + e2 + e3.

[0066] As an additional buffer, the third storage buffer B3 can be used to control the final output P of the phoneme recognition system. Store the output predicted phoneme with a timestamp into B3. The method can continuously obtain the first phoneme from B3 and input the first phoneme into the animation step every A ms. If the predicted phoneme with a timestamp has a negative e t , that is, D + e t < D, the method may have to wait A ms to output from B3. If the predicted phoneme P et with a timestamp has a positive e t , that is, D + e t > D and there are no more phonemes in the buffer, the method can output the last phoneme and store P et as the last phoneme in the buffer. If the next predicted phoneme also has a positive e t , P et can be output. If the next predicted phoneme does not have a positive e t , the method can discard P etThe next predicted phoneme is output after a time interval of A ms. In this way, the phoneme recognition process can have a stable output stream with a time interval of A ms, and the total delay from the original audio input signal to the corresponding output phoneme stream is (m+br)·A+D, where D=d1+d2+d3. The stable phoneme output stream can be stored in data buffer B4. The following pseudocode illustrates the processing in the embodiment. The buffer is B3, and the output d_P is the output phoneme to be stored in B4.

[0067]

[0068] Reference Figure 2 Step S240 of the speech animation generation method involves generating a speech animation based on the identified phonemes. In an embodiment, the method may acquire data from a data buffer B4 and use the data to generate a corresponding speech animation.

[0069] For real-time speech animation, the method disclosed in the embodiments may not directly use the output phoneme sequence from the phoneme recognition process, because the output phoneme sequence predicts phonemes for audio frames with a time interval of A. Therefore, B4 is used to select the appropriate next phoneme frame for the animation curve generation process. Figure 6 Examples of visual pixels corresponding to the phoneme / o / are shown in different frames when the phoneme is pronounced. Figure 7 An example of phoneme selection for generating 2D pixel-domain animation curves is shown in the embodiment. For example... Figure 7 As shown, the same phoneme can appear in different frames. The size of B4 can correspond to the average pronunciation time of a single phoneme in an audio frame. The predicted phoneme output by the phoneme recognition processing can first be stored in the data buffer B4. Phoneme pronunciation can be dynamic, meaning that the visual phoneme at the start of pronunciation may be different from the phoneme at the beginning of pronunciation. Figure 6 The lip shapes corresponding to the phonemes shown are not exactly the same. Therefore, an appropriate phoneme frame should be selected from the partial phoneme pronunciation frames to calculate the complete time of the visual pixel transition used in the animation curve generation process. For example, representing Figure 6 The rightmost visual pixel frame contains the phoneme frame in the / o / phoneme frame sequence. If the smallest recognizable phoneme pronunciation time in the dataset is pr... min Audio frames, and the length of consecutively predicted phonemes in the buffer is less than pr min Then, the corresponding phoneme can be replaced with the previous phoneme. Therefore, the upcoming phoneme can be selected according to the following rules:

[0070] (a) continuously at least pr min The same phoneme of the unit is added to the buffer.

[0071] (b) If the number of consecutive phonemes is greater than prmin The unit is smaller than the size of data buffer B4. Select the appropriate frame representing the phoneme from this part of the buffer.

[0072] (c) If the number of consecutive phonemes is greater than the size of B4, then select an appropriate frame from all phonemes in the buffer, and do not select a new frame until the next different phoneme is appended to the buffer.

[0073] In most procedural methods that use keyframe visuals to generate the final animation, including the JALI (jaw and lip) visual domain and procedural lip synchronization methods, phonemes are mapped to a fixed, static visual without any variation. However, based on observations of human speech behavior, the visual corresponding to a phoneme may differ slightly in different situations. This observation is correct even if the current phoneme has a sufficiently long articulation time to eliminate the co-articulation caused by the previous phoneme articulation. For example, under the same speaking style, the visual representing the phoneme / s / behaves differently in "things" and "false". Therefore, in embodiments of the present invention, the animation process can be implemented using a modified form of the jaw and lip (JALI) visual domain and a procedural lip synchronization method.

[0074] In embodiments of the present invention, the two-dimensional (2D) view domain is divided into different blocks. For example... Figure 8 The diagram shows the division of the 2D viewport into 20 blocks with block indices. (See reference...) Figure 8 Each phoneme can correspond to a range of blocks rather than a static point in a 2D domain.

[0075] In an embodiment, the neural network used in step S230 may include several components. Figure 9 The overall network architecture in the embodiment is shown. Figure 9 As shown, a network for phoneme recognition may include one or more convolutional neural network (CNN) layers, long short-term memory (LSTM) stacks, one or more unified LSTM layers, and fully connected layers (FCL). The CNN layers process the audio feature vector x... t As input, frequency modeling is applied to the audio features. The n-dimensional vector output of the CNN layer can be passed to the n-layer stack of the LSTM for parallel and discrete processing and temporal modeling. The output can be related to f t The input is combined with an additional LSTM layer for temporal modeling, and then passed through a fully connected layer. An HMM triphone decoder can be used to predict phoneme annotations.

[0076] The CNN layer can be used for frequency modeling to reduce frequency variations. Different people, even when saying the same thing, contain different frequency ranges in their speech; based on this observation, CNN layers can be used to improve the frequency variations in audio features.

[0077] CNN layers can also function in the time-space domain. Here, space refers to the detection and learning of frequency domain audio feature patterns. The input features contain frequency information because each coefficient in the Mel frequency cepstral is generated through a different set of frequency filters. The CNN layer can include CNN filters. Each CNN filter can learn different frequency patterns from the input features during training. In embodiments of the invention, the CNN output features of different CNN filters (CNN...) are used to... fout The CNN layers are connected to different LSTMs in the LSTM stack to select a network structure that enhances the learned frequency patterns in the input features. For example, in one embodiment, the first CNN layer includes a 9×9 frequency (spatial)-temporal filter, while the second CNN layer includes a 3×3 filter. The first CNN layer may have 256 output channels, and the second CNN layer may have 16 output channels. Since neither max pooling nor average pooling layers in the CNN layers help improve the performance of the neural network processing, pooling layers are not configured in the CNN layers in this embodiment.

[0078] After frequency modeling is applied to the CNN layer and the CNN filters learn acoustic features from the input features, each CNN output channel can generate intermediate features. Figure 10 The output from the CNN output channel is shown. Subsequently, the LSB can perform temporal modeling on the intermediate features output from the CNN output channel through a parallel processing procedure.

[0079] The output content of the CNN channel is CNN. fout This can be expressed by Formula 2, where C fout,i This represents the output content from the i-th channel. CNN fout This can be passed to multiple LSTM modules in the LSB. These LSTM modules can be referred to as LSTM tubes (LTs), and in cases such as... Figure 9 The LSB is shown in the diagram. The number of LTs within the LSB depends on the number of output channels of the last CNN layer, with one LT corresponding to one CNN output channel. LTs can be relatively small and independent LSTM networks. LSTM networks have storage units to remember information about previous features, making them very advantageous in temporal modeling. In the embodiment, each output C from a specific CNN channel is... fout,iConsidered as derived features of the original audio features, and possessing context within a sliding window. Therefore, separate temporal modeling can be applied to each different C. fout,i . Figure 11A and Figure 11B The processing procedure for the LSB in the embodiment is shown. For example... Figure 11A An example is shown showing C fout,i Intermediate features are passed to LTs inside the LSB, and the LTs generate the corresponding LT output content Lt. n For example, C fout,0 The LT0 located in the LSB is passed to the independent temporal modeling (as shown in Equation 3, where FCL(LSTM(...)) represents the fully connected layer processing of the LSTM module), where "0" represents the CNN filter with index 0, and the output feature is LT0. Figure 11B The LSB output characteristics of the set are shown. For example... Figure 11B As shown, the separate output features from LTs can be aggregated together and used as the input features of the next LSTM layer (as shown in Equation 4).

[0080] CNN fout ={C fout,1 C fout,2 ,…,C fout,n} Formula 2

[0081] Lt n =FCL(LSTM(C fout,i )) Formula 3

[0082] LSB (CNN) fout )={Lt1,Lt2,…,Lt n} Formula 4

[0083] In embodiments of the present invention, the LSB may comprise 16 LTs, and each LT includes two LSTM layers and a fully connected layer (FCL). Each LSTM layer may be configured to have 512 hidden units and a dropout rate of 0.3, while the FCL may have 128 output units.

[0084] It has been demonstrated that LSTM layers are advantageous in extracting temporal patterns from the input feature space. The gates in an LSTM can be used to control the flow of information within the LSTM. For example, the input gate can decide what information to add to the cell state. The forget gate can decide which information needs to be removed from the cell state, and the output gate can decide which information can be used in the output. Therefore, LSTM networks can selectively “remember” temporal features.

[0085] In embodiments of the invention, after the LSB performs parallel and separate timing modeling, the output can be passed to an additional unified LSTM layer for unified timing modeling. In one embodiment, four additional LSTM layers can be configured. Each additional LSTM layer may have 1024 hidden units, 512 output units, and a dropout rate of 0.2.

[0086] The output from the additional LSTM layer is passed to a fully connected layer (FCL), which has the ability to mix the output signals from all neurons in the previous layers and shape the output probabilities for each class so that the target class has a higher probability. In an embodiment, the FCL can be implemented using a softmax function activation. In an embodiment, the FCL can have 1024 hidden units and output an 1896-dimensional vector, which represents the probability of 1896 shared states in an HMM-GMM (Hidden Markov Model-Gaussian Mixture Model). The HMM triphone decoder takes the vector from the FCL as input to predict the final phoneme labeling.

[0087] In an embodiment, in order to summarize the network components used for predicting phonemes during neural network processing, step S230 of the speech animation generation method may include several steps. Figure 12 The steps in the embodiments are shown. (Refer to...) Figure 12 Step S232 involves inputting audio features into a CNN filter to generate multiple sets of intermediate features, each set of intermediate features corresponding to a CNN output channel; Step S234 involves inputting the intermediate features into multiple LSTM networks for parallel temporal modeling to generate LSB output features, wherein each LSTM network inputs a set of intermediate features to generate a set of LSB output features; Step S236 involves inputting the LSB output features into a unified LSTM network for unified temporal modeling to generate unified LSTM output features; Step S238 involves inputting the unified LSTM output features into an FCL to generate an HMM shared state; Step S239 involves inputting the HMM shared state into an HMM triphone decoder to generate recognized phoneme annotations.

[0088] In neural networks, each layer can focus on different input content. In automatic speech recognition (ASR) tasks, lower layers (e.g., CNN layers in RealPRNet) may focus more on speaker adaptability, while higher layers (e.g., additional LSTM layers in RealPRNet) may focus more on discriminative ability. Therefore, the input features of different layers can complement each other to improve the performance of the neural network. In the embodiments of this invention, two feature addition strategies are employed. Figure 9Two additional strategies are shown in the figure by lines (1) and (2), where (1) represents the original frame feature x. t f in t (2) represents the combination of LSB output content and additional LSTM layer output content.

[0089] The first set of features explored short-term features f t And supplementary information on long-term features output from the LSB (which can be understood as x) t (Higher-order representation). x t You can use f t and its contextual features [f t-n ,…,f t-1 ] and [f t+1 ,…,f m However, LSTM networks may not consider f when making predictions. t And its different values ​​of contextual features. Conversely, the LSTM network can... t All f are considered as continuous features and x is considered equally. t All f in the equation. By using the first combined feature, it is possible to predict Pt in x. t Strengthening f t The importance of.

[0090] The second set of combined features can examine the output of the LSB and the additional LSTM layers, including separate and unified supplementary temporal feature information. This is because the layers of a neural network may focus on x. t Different information in the data can be used to view the features as x with different pattern complementarities. t The representation of higher-order features. Additional features at multiple scales demonstrate a positive impact on neural network performance.

[0091] The performance of the methods disclosed in the embodiments of the present invention has been evaluated. In particular, the performance of RealPRNet has been evaluated in three aspects: (1) phoneme recognition accuracy, (2) subjective and objective speech animation quality, and (3) buffer occupancy dynamics for real-time applications.

[0092] Experimental results were presented using the TIMIT dataset. TIMIT is widely used for phoneme recognition evaluation. It contains 6300 sentences, each spoken by 630 speakers from eight major dialect regions in the United States, delivering 10 sentences each. Following the standard TIMIT setup, the neural network was trained using the standard TIMIT training set (containing 3696 utterances from 462 speakers). The performance of the neural network was then evaluated on the TIMIT Core test set, which contains 192 utterances.

[0093] TIMIT's S5 formula computes phoneme durations in each utterance using a forced alignment technique and generates a shared-state triphone model of the Hidden Markov Model (HMM), with 1896 shared states and their properties (i.e., state probabilities, transition probabilities, corresponding phonemes, etc.). Subsequently, a triphone decoder with a binary phoneme model is used at the end of the neural network structure, taking the shared-state stream as input and outputting a predicted phoneme stream. The output accuracy y corresponding to the input feature x in the training dataset is the index of the HMM shared state. Forced alignment from audio to HMM shared states is enabled. For network training, 10 full training iterations are set as the minimum training epoch, and early stopping is enabled during training (if the loss change to full training is verified to be less than 0.001). The Adam optimizer is used in the first epoch, while the Momentum Stochastic Gradient Descent (MSGD) optimizer is used in the remaining epochs. The batch size for the first epoch is 256, and the batch size for the remaining epochs is 128. The learning rates for the first four epochs were 0.01, 0.001, 0.0005, and 0.0001, respectively, while the learning rate for the remaining epochs was 0.0001. The parameters of the CNN layers were initialized with weights, while all LSTM layers used a dropout rate of 0.3. ReLU activation was applied to most layers except for the final fully connected layer using the softmax function. The performance of the neural networks with different values ​​of m in the input features was evaluated. In the experiments, x... t The value n is set to m+1. Table 1 shows the network parameters during execution.

[0094] Table 1 Neural Network Parameters

[0095] Hidden Unit Output Channel Output unit Drop rate Window size (k) conv0 256 channels 9 conv1 16 channels 3 lsb0 1024 128 0.3 lstm0 1024 512 0.2 lstm1 1024 512 0.2 lstm2 1024 512 0.2 lstm3 1024 512 0.2 fcl0 1024 1896

[0096] The standard TIMIT phoneme recognition evaluation metric is used to evaluate the phoneme error rate (PER) of RealPRNet. The original TIMIT dataset uses 60 phonemes to create a dictionary. During evaluation, the process first maps the 60 phonemes to 39 phonemes, then calculates the "Levenshtein Distance" between the recognized phoneme sequence and the accurate phoneme sequence. The edit distance is a method to measure the difference between two sequences. It calculates the minimum number of single-character edits (including insertions, deletions, and replacements) required to transform one sequence into another. First, the number of edits required to change the recognized phoneme sequence to the accurate phoneme sequence is calculated, then the ratio between the minimum number of edits and the length of the entire phoneme sequence is calculated. This ratio is the PER. In real-time, phonemes are predicted for each audio frame. Therefore, in our evaluation process, PER is calculated based on the phoneme sequence at the audio frame level.

[0097] The block distance error is measured as the average distance between the trajectory curve generated by the phoneme sequence identified in the visual pixel domain and the curve generated by the accurate phoneme sequence. The basic unit used here is the short side of a 2D visual pixel domain block. For each audio frame, corresponding points on the two curves and the absolute distance between these two points, also known as the Euclidean distance, are calculated. Then, the average distance between these two curves at each time t is calculated.

[0098] In the experiments, RealPRNet was compared with two other phoneme recognition systems: (1) a 4-layer LSTM with a feedforward deep neural network structure; and (2) a CLDNN that combines CNN, LSTM, and a fully connected deep learning neural network (DNN). Table 2 shows the performance of offline phoneme recognition evaluation using the TIMIT dataset. RealPRNet outperformed the other methods in terms of PER, reducing it by up to 7.7%.

[0099] Reference Figure 13 The real-time performance of LSTM, CLDNN, and RealPRNet was compared. The x-axis in the figure represents a single x... t The number of frames used, with the y-axis representing the phoneme error rate. For example... Figure 13 As shown, when there is insufficient temporal context information (i.e., using fewer than 10 f's), s To set x t At that time, LSTM performance was very poor, but LSTM performance increased with x. t The value increases until x t The feature of 20 audio frames is continuously improving. For example... Figure 13 As shown, RealPRNet outperforms LSTM by up to 20% and CLDNN by up to 10%. The combination of frequency modeling and temporal modeling enables RealPRNet to outperform x t It exhibits smooth performance across different frame rate options.

[0100] Table 2 Offline Phoneme Recognition Error Rate

[0101] Neural Networks Offline PER LSTM 18.63 CLDNN 18.30 RealPRNet 17.20

[0102] To further understand the advantages of RealPRNet, we considered the following two other variants: (1) RealPRNet without LSTM layers, and (2) CLDNN and RealPRNet without adding long short-term features.

[0103] The RealPRNet applies two different types of temporal modeling to x. tThis includes separate temporal modeling performed by LSBs and unified temporal modeling performed by unified LSTM layers. The differences between these two temporal modeling methods, which affect neural network performance, can be compared. In one experiment, the unified LSTM layers were removed from the RealPRNet, and the resulting neural network was termed "RealPRNet-Separate." By comparing RealPRNet-Separate and CLDNN separately, the following results were obtained: Figure 14 The result in the middle. Figure 14 This demonstrates that a single, separate temporal model cannot outperform a unified temporal model. Therefore, RealPRNet's performance benefits when both temporal models are used together. Figure 14 The per-percentage (PER) of RealPRNet is shown under different latency scenarios (different values ​​of m and n), demonstrating that RealPRNet outperforms any single network architecture used for temporal modeling.

[0104] As mentioned above, the performance of neural networks can be further improved through multi-scale feature addition techniques. This technique can be demonstrated using RealPRNet, and its performance can be compared with the strongest baseline model CLDNN with feature addition structures and a previous network without added features. Figure 14 As shown, the performance of RealPRNet with multi-scale feature addition (RealPRNet-Addition) and CLDNN with multi-scale feature addition (CLDNN-Addition) are both improved. The added short-term features provide supplementary information to the intermediate input features, which force the neural network to focus its attention on the current frame f. t Therefore, RealPRNet outperforms other neural networks in most latency scenarios. In particular, it achieves an additional 4% relative improvement in real-time PER evaluation.

[0105] The quality of the output speech animation generated by the method described in this invention was also evaluated. The trajectory curves generated by the identified phoneme streams predicted by various reference neural networks and RealPRNet in the visual pixel domain were compared with the trajectory curves generated by the accurate model. The shorter side length of the block was used as the distance unit to calculate the difference between the curve generated by the identified phonemes and the accurate curve at each time point.

[0106] The comparison results are in Figure 15 As shown in the image. Figure 15 In this diagram, the x-axis represents the number of frames used in the input features, and the y-axis represents the block distance error in the basic unit. For example... Figure 15 As shown, RealPRNet can achieve the fewest errors under different latency scenarios, reducing them by up to 27% compared to other networks.

[0107] For subjective evaluation, the visual differences between this invention and other prior art realistic speech animation methods were compared. Believable speech animation was generated using the same audio, and video was recorded. Videos from different methods were aligned with the same audio signal. Then, screenshots of the visual pixels were simultaneously acquired to visually compare the similarity of the pixels. Figure 16 As shown, all results were similar. The subjective test involved inviting participants to watch the output animated video side-by-side and receive the same feedback.

[0108] The buffer mechanism is used to stabilize the system output and ensure that the output delay is within tolerance. In the experiment, the occupancy state of B3 during runtime is... Figure 17 As shown. Ideally, every br·A ms, [P t ,…,P t-br+1 Input B3 is queued, and B3 outputs the first value in the queue every A ms. After each output, the occupancy status of B3 is recorded. In the experiment, the value of br was 4, therefore the occupancy of B3 should be evenly distributed in [0, 1, 2, 3]. In fact, 90% of the test cases were within this range, indicating that the method disclosed in this invention can work in real-time. Most errors in B3 occur at 0 and 4 because the computer program cannot use precise A at every step during operation (e.g., A fluctuates between 10.3 ms and 11 ms). Other cases are caused by computational fluctuations in other parts of the phoneme recognition subsystem.

[0109] This invention discloses an apparatus and method for generating believable audio-driven speech animations, the framework featuring a novel phoneme recognition neural network called RealPRNet. RealPRNet demonstrates improved performance in phoneme recognition against strong benchmark models. The framework can be easily implemented in most existing virtual avatars without creating new ones. Creating realistic human-like speech animations requires significant effort in pre-training using video and audio data, and even post-production editing by artists. The apparatus and method described in this invention focus on generating believable real-time speech animations with a simple execution method. These disclosed features also allow users to edit with online post-production effects.

[0110] The methods and apparatus in the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the embodiments above are only for the purpose of helping to understand the present invention. Furthermore, those skilled in the art can make improvements to the specific embodiments and application scope based on the ideas of the present invention. In conclusion, the content of this specification should not be construed as limiting the present invention.

[0111] This invention contains copyrighted material. Copyright is the property of the copyright holder. The copyright holder does not object to the reproduction of patent documents or the publication of patents in the official records and documents of the State Intellectual Property Office.

Claims

1. A method for generating speech animation from audio signals, characterized in that, include: Receive audio signals; The received audio signal is converted into frequency domain audio features; The frequency domain audio features are input into a neural network to perform neural network processing to identify phonemes, wherein the neural network is trained using a phoneme dataset, the phoneme dataset comprising audio signals with corresponding accurate phoneme annotations; Generate speech animation based on the identified phonemes; The step of inputting the frequency domain audio features into a neural network to perform neural network processing to identify phonemes includes: The audio features are input into a convolutional neural network filter to generate multiple sets of intermediate features, each set of intermediate features corresponding to a convolutional neural network output channel. The intermediate features are input into multiple long short-term memory networks for parallel time-series modeling to generate long short-term memory stack block output features, wherein each long short-term memory network inputs a set of intermediate features to generate a set of long short-term memory stack block output features. The output features of the long short-term memory stack block are input into a unified long short-term memory network for unified temporal modeling to generate unified long short-term memory output features. The unified long short-term memory output features are input into the fully connected layer to generate the shared state of the hidden Markov model; and The shared state of the Hidden Markov Model is input into the Hidden Markov Model triphone decoder to generate the recognized phoneme annotations.

2. The method for generating speech animation from audio signals according to claim 1, characterized in that, The step of converting the audio signal into frequency domain audio features includes: Calculate the Mel-frequency cepstral coefficients of the audio signal; Calculate the first and second derivative components of the Mel frequency cepstral coefficients, and combine the first and second derivative components into a single audio feature frame; The frequency domain audio features are generated based on the single audio feature frame, the forward context vector of the single audio feature frame, and the backward context vector of the single audio feature frame.

3. The method for generating speech animation from audio signals according to claim 1, characterized in that, The method further includes: The audio features are stored in the first storage buffer; The shared state of the hidden Markov model is stored in a second storage buffer; and The identified phonemes are stored in a third storage buffer; The first storage buffer, the second storage buffer, and the third storage buffer are fixed-size first-in-first-out (FIFO) buffers.

4. The method for generating speech animation from audio signals according to claim 2, characterized in that, The method further includes: A first combined feature is generated based on the single audio feature frame and the output feature of the long short-term memory stack block; and The first combined features are used as input to the unified long short-term memory network to generate the unified long short-term memory output features.

5. The method for generating speech animation from audio signals according to claim 2, characterized in that, The method further includes: The long short-term memory stack block output features are combined with the unified long short-term memory output features to generate a second combined feature; and The second combined feature is used as input to the fully connected layer to generate the shared state of the hidden Markov model.

6. The method for generating speech animation from audio signals according to claim 1, characterized in that, The process of generating speech animation based on the identified phonemes includes: A two-dimensional visual pixel domain is generated and the two-dimensional visual pixel domain is divided into multiple blocks, wherein each identified phoneme corresponds to a block of the two-dimensional visual pixel domain.

7. An apparatus for generating speech animation from audio signals, characterized in that, include: Memory stores executable instructions for a computer; and The processor, coupled to the memory, and when executing the computer-executable instructions, is used for: Receive audio signals; The received audio signal is converted into frequency domain audio features; The frequency domain audio features are input into a neural network to perform neural network processing to identify phonemes, wherein the neural network is trained using a phoneme dataset, the phoneme dataset comprising audio signals with corresponding accurate phoneme annotations; and Generate speech animation based on the identified phonemes; The step of inputting the frequency domain audio features into a neural network to perform neural network processing to identify phonemes includes: The audio features are input into a convolutional neural network filter to generate multiple sets of intermediate features, each set of intermediate features corresponding to a convolutional neural network output channel. The intermediate features are input into multiple long short-term memory networks for parallel time-series modeling to generate long short-term memory stack block output features, wherein each long short-term memory network inputs a set of intermediate features to generate a set of long short-term memory stack block output features. The output features of the long short-term memory stack block are input into a unified long short-term memory network for unified temporal modeling to generate unified long short-term memory output features. The unified long short-term memory output features are input into the fully connected layer to generate the shared state of the hidden Markov model; and The shared state of the Hidden Markov Model is input into the Hidden Markov Model triphone decoder to generate the recognized phoneme annotations.

8. A non-transitory computer-readable storage medium storing a plurality of instructions, characterized in that, When the processor executes the instruction, the instruction causes the processor to: Receive audio signals; The received audio signal is converted into frequency domain audio features; The frequency domain audio features are input into a neural network to perform neural network processing to identify phonemes, wherein the neural network is trained using a phoneme dataset, the phoneme dataset comprising audio signals with corresponding accurate phoneme annotations; and Generate speech animation based on the identified phonemes; The step of inputting the frequency domain audio features into a neural network to perform neural network processing to identify phonemes includes: The audio features are input into a convolutional neural network filter to generate multiple sets of intermediate features, each set of intermediate features corresponding to a convolutional neural network output channel. The intermediate features are input into multiple long short-term memory networks for parallel time-series modeling to generate long short-term memory stack block output features, wherein each long short-term memory network inputs a set of intermediate features to generate a set of long short-term memory stack block output features. The output features of the long short-term memory stack block are input into a unified long short-term memory network for unified temporal modeling to generate unified long short-term memory output features. The unified long short-term memory output features are input into the fully connected layer to generate the shared state of the hidden Markov model; and The shared state of the Hidden Markov Model is input into the Hidden Markov Model triphone decoder to generate the recognized phoneme annotations.

Images