Digital human expression generation method and system based on voice features and facial features

Abstract

The application provides an expression generation method and system of a digital person based on voice features and facial features; the expression generation method comprises the following steps: positioning a facial expression area of a plurality of continuous frames of to-be-processed images according to a YOLO algorithm, and extracting an LBP feature vector of the facial expression area; identifying the LBP feature vector by using a random forest recognizer to obtain a facial emotion recognition type; performing pre-emphasis and frame windowing processing on continuous voice data to obtain a preprocessed voice signal; extracting linear and nonlinear voice features by using an MFCC feature extraction algorithm and a nonlinear feature extraction algorithm respectively; inputting the linear and nonlinear voice features into an SVM classifier to obtain a voice emotion recognition type; performing decision judgment on the facial emotion recognition type and the voice emotion recognition type to obtain a final emotion recognition type result; and generating an expression animation of the digital person according to the final emotion recognition type result. The technical scheme of the application can solve the problems of low recognition accuracy and poor recognition effect in the prior art.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a method and system for generating facial expressions in digital humans based on voice features and facial features. Background Technology

[0002] AR headsets are wearable devices that can be worn on the head to display information. These devices overlay virtual information onto the real world, allowing both to be displayed in real-time on the same screen. With the rapid development of AR technology and the widespread adoption of smart devices, AR technology has been widely applied in AR-based cultural and tourism scenarios.

[0003] Emotion recognition in AR-based cultural tourism scenarios is a challenging task, but it can enhance the cultural tourism experience and provide service providers with crucial information about tourists' emotions and reactions. This often utilizes "digital human" technology. "Digital humans" typically refer to virtual characters or digitized human figures. In the context of metaverse cultural tourism, "digital humans" can enrich the travel experience, providing personalized experiences for each tourist based on their needs and interests, offering services such as guidance, entertainment, interaction, emotional support, and cultural exchange.

[0004] In AR-based cultural and tourism scenarios, while "digital humans" can simulate emotions and interactions, they lack the genuine emotions and interpersonal skills of humans. Therefore, emotion recognition is crucial in AR-based cultural and tourism scenarios. Common emotion recognition mechanisms include facial expression analysis and voice analysis. However, relying solely on either facial expression or voice analysis cannot accurately identify a user's emotions, leading to the development of multimodal emotion analysis. Multimodal emotion analysis combines information from multiple perceptual modalities, such as facial expressions, voice, and physiological signals, to improve the accuracy of emotion recognition. Multimodal emotion analysis technology utilizes deep learning methods to integrate information from different modalities.

[0005] However, due to the complexity of various signal data, data preprocessing under different modes and feature fusion under different modes are quite complicated and have poor fusion capabilities, resulting in low recognition accuracy and poor final recognition effect.

[0006] Application content

[0007] This application provides a digital human expression generation scheme based on voice features and facial features, which can solve the problems of low recognition accuracy and poor recognition effect in the prior art.

[0008] To address the aforementioned problems, according to a first aspect of this application, this application provides a method for generating facial expressions in digital humans based on speech features and facial features, comprising:

[0009] Based on the YOLO algorithm, the regions of facial expressions related to emotion in multiple consecutive frames of images to be processed are located, and the LBP operator is used to extract the LBP feature vectors of the facial expression regions.

[0010] The LBP feature vector is identified using a random forest recognizer to obtain the facial emotion recognition type.

[0011] Pre-emphasis and frame-segmented windowing are performed on the continuous speech data corresponding to multiple consecutive frames of images to be processed to obtain a pre-processed speech signal.

[0012] The MFCC feature extraction algorithm and the nonlinear feature extraction algorithm are used to extract linear speech features and nonlinear speech features corresponding to the preprocessed speech signal, respectively.

[0013] Linear and nonlinear speech features are input into an SVM classifier to perform speech emotion recognition and obtain the speech emotion recognition type.

[0014] Based on the feature fusion algorithm based on the decision layer, decision judgments are made on the facial emotion recognition type and the voice emotion recognition type to obtain the final emotion recognition type result;

[0015] Based on the final emotion recognition type results, generate digital human facial animations.

[0016] Preferably, in the above-mentioned digital human expression generation method, the step of locating emotion-related facial expression regions in multiple consecutive frames of images to be processed using the YOLO algorithm includes:

[0017] For multiple consecutive frames of images to be processed, each frame of the image to be processed is divided into multiple square grids;

[0018] For multiple square grids, the YOLO algorithm is used to predict multiple bounding boxes for each square grid, resulting in multiple target windows;

[0019] Using a nonmaximum suppression algorithm, redundant windows in multiple target windows are removed based on multiple borders to obtain the facial expression region.

[0020] Preferably, in the above-mentioned digital human expression generation method, the step of extracting the LBP feature vector of the facial expression region using the LBP operator includes:

[0021] Define the LBP radius and neighborhood in the facial expression region;

[0022] The LBP value of each pixel in the neighborhood is set according to the gray values ​​of the neighboring pixels and the center pixel, so as to obtain the LBP feature vector of the image to be processed.

[0023] Calculate the average Euclidean distance between the LBP feature vectors of each frame of the image to be processed and all other frames in a series of consecutive frames of the image to be processed.

[0024] A predetermined number of frames with the smallest average Euclidean distance are selected as the keyframe images for facial expression recognition.

[0025] The LBP feature vectors of the keyframe images are used as the LBP feature vectors of the facial expression regions and input into the random forest recognizer.

[0026] Preferably, in the above-mentioned digital human expression generation method, the step of performing pre-emphasis and frame-by-frame windowing processing on the continuous speech data corresponding to multiple consecutive frames of images to be processed to obtain a preprocessed speech signal includes:

[0027] Using a first-order high-pass filter, the calculation formula based on the pre-emphasized signal is as follows:

[0028]

[0029] Continuous speech data is pre-emphasized to obtain a pre-emphasized speech signal, wherein... For the nth voice data, Let y[n] be the (n-1)th speech data, y[n] be the nth preprocessed speech signal, and a be the filter coefficients used for pre-emphasis.

[0030] The pre-emphasized speech signal is segmented into frames to obtain multiple frames of speech signal.

[0031] Using a Hamming window as the window function, the calculation formula for the window function is as follows:

[0032]

[0033] The calculated window function is used to window the multi-frame speech signal to obtain a periodic speech signal; where L represents the number of points in the speech signal and w[n] represents the window function.

[0034] Preferably, in the above-mentioned digital human expression generation method, the steps of using the MFCC feature extraction algorithm and the nonlinear feature extraction algorithm to extract the linear speech features and nonlinear speech features corresponding to the preprocessed speech signal respectively include:

[0035] The windowed periodic speech signal is subjected to a fast Fourier transform to obtain the Fourier transformed speech signal.

[0036] Using the Mel filter, the center frequency of the Mel filter is calculated according to the formula:

[0037]

[0038] Calculate the center frequency of the Mel filter, where, and These represent the highest and lowest frequencies within the application range of the Mel filter, respectively, and N represents the number of points in the periodic speech signal when performing a Fast Fourier Transform. This represents the sampling frequency, and M is the number of triangular filters in the Mel filter. This represents the center frequencies of m Mel filters. represent The inverse function;

[0039] Using the center frequency of the Mel filter, the frequency response is calculated using the formula:

[0040]

[0041] Calculate the frequency response of the Mel filter; where, This represents the center frequency of the m-th Mel filter. This represents the center frequency of the (m-1)th Mel filter. This represents the center frequency of the (m+1)th Mel filter;

[0042] Using the frequency response of the Mel filter and the speech signal after Fourier transform, the logarithmic energy is calculated according to the formula:

[0043]

[0044] Calculate the logarithmic energy of the output of the m-th filter; where, This represents the logarithmic energy of the output of the m-th filter. This represents the speech signal after Fourier transform. This represents the frequency response of the Mel filter;

[0045] Using the logarithmic energy of the m-th filter output, the MFCC coefficients of the speech signal are obtained by transforming it according to the discrete cosine transform formula;

[0046] Using the MFCC coefficients of the speech signal, the calculation formula is based on the differential parameters:

[0047]

[0048] Calculate the dynamic difference parameters of the preprocessed speech signal as the linear speech features corresponding to the preprocessed speech signal; where Ct represents the t-th cepstral coefficient; Q represents the order of the cepstral coefficient; and K represents the time difference of the first derivative.

[0049] Preferably, in the above-mentioned digital human expression generation method, the steps of using the MFCC feature extraction algorithm and the nonlinear feature extraction algorithm to extract the linear speech features and nonlinear speech features corresponding to the preprocessed speech signal respectively include:

[0050] Using the approximate entropy algorithm, for an N-point pre-emphasized speech signal sequence {y(n)} composed of pre-emphasized speech signals, the formula for calculating the pre-emphasized speech signal sequence using a w-dimensional vector is:

[0051]

[0052] The combination yields a w-dimensional vector, where X(n) represents the w-dimensional vector, w represents the dimension, and n represents the nth point of the pre-emphasized speech signal;

[0053] Calculate the relative Euclidean distance between any w-dimensional vector X(n) corresponding to the N-point pre-emphasized speech signal sequence and all remaining vectors X(k);

[0054] Count the number of w-dimensional vectors whose relative Euclidean distance is less than or equal to a predetermined distance threshold;

[0055] The number of vectors using the w-dimensional approach is calculated using the following formula:

[0056]

[0057] Calculate the proximity between multiple w-dimensional vectors, where, This represents the number of w-dimensional vectors, where W represents the dimension of the vectors and N represents the total number of points in the speech sequence. This represents the degree of proximity between w-dimensional vectors;

[0058] Using the proximity between w-dimensional vectors and the proximity between w+1-dimensional vectors, the formula for calculating approximate entropy is obtained:

[0059]

[0060] The approximate entropy is calculated and used as the nonlinear speech feature corresponding to the preprocessed speech signal, where, Represents the degree of proximity between w-dimensional vectors. It represents the degree of proximity between w+1 dimensional vectors.

[0061] Preferably, in the above-mentioned digital human expression generation method, the step of inputting linear and nonlinear speech features into an SVM classifier for speech emotion recognition to obtain the speech emotion recognition type includes:

[0062] The data samples corresponding to linear speech features and the data samples corresponding to nonlinear speech features are divided into training data samples and test data samples respectively.

[0063] Using a 10-fold cross-validation method, the SVM classifier was trained using training data samples and tested using test data samples, respectively, to obtain the recognition accuracy of various speech emotion recognition types.

[0064] The recognition accuracy of various voice emotion recognition types is averaged to obtain the average recognition accuracy as the final voice emotion recognition type.

[0065] Preferably, in the above-mentioned digital human expression generation method, the step of making a decision judgment on the facial emotion recognition type and the voice emotion recognition type based on the feature fusion algorithm based on the decision layer to obtain the final emotion recognition type result includes:

[0066] Obtain the recognition probability corresponding to the facial emotion recognition type and the recognition probability corresponding to the voice emotion recognition type;

[0067] Using the recognition probabilities corresponding to facial emotion recognition types and the recognition probabilities corresponding to voice emotion recognition types, the recognition probabilities corresponding to m classification results are obtained based on the feature fusion algorithm.

[0068] Using the recognition probabilities corresponding to the m classification results, the formula is calculated according to the summation decision rule:

[0069]

[0070] The recognition probabilities corresponding to each of the m classification results are obtained; where, This represents the recognition probability corresponding to the x-th classification result. This represents the recognition probability corresponding to the j-th classification result obtained after passing through the i-th classifier;

[0071] From the recognition probabilities corresponding to the m classification results, select the emotion recognition type with the highest recognition probability value as the final emotion recognition type result.

[0072] Preferably, in the above-mentioned digital human expression generation method, the step of generating digital human expression animation based on the final emotion recognition type result includes:

[0073] Using the final emotion recognition type result, retrieve the corresponding facial animation from the facial image library;

[0074] Facial expression animation technology is used to generate animated expressions onto the corresponding facial areas of the digital human.

[0075] According to a second aspect of this application, this application also provides a digital human expression generation system based on voice features and facial features, comprising:

[0076] A memory, a processor, and a digital human expression generation program based on voice features and facial features stored in the memory and running on the processor, wherein when the digital human expression generation program based on voice features and facial features is executed by the processor, the steps of the digital human expression generation method based on voice features and facial features provided by any of the above technical solutions are implemented.

[0077] In summary, the digital human expression generation scheme based on speech and facial features provided in this application locates the facial expression region in multiple consecutive frames of images to be processed using the YOLO algorithm, extracts the LBP feature vector associated with the expression in the facial expression region using the LBP operator, and then uses a random forest classifier to identify the LBP feature vector, thereby obtaining the facial emotion recognition type. This allows the corresponding facial emotion recognition type to be obtained from facial features. Next, the continuous speech data corresponding to the multiple consecutive frames of images to be processed undergoes pre-emphasis and frame-by-frame windowing processing to obtain a preprocessed speech signal. The MFCC feature extraction algorithm and nonlinear feature extraction algorithm are used to extract the linear and nonlinear speech features corresponding to the preprocessed speech signal, respectively. Finally, the linear and nonlinear speech features are input into an SVM classifier for speech emotion recognition, thus obtaining the corresponding speech emotion recognition type. Finally, a feature fusion algorithm based on a decision layer is used to fuse the above facial emotion recognition type and speech emotion recognition type, and then a decision is made to obtain the final emotion recognition type result. The above method can improve the accuracy of emotion type recognition through bimodal emotion analysis of voice and facial features, and then generate more accurate digital human facial animation based on the final emotion recognition result. At the same time, it combines linear and nonlinear voice features for speech recognition, resulting in higher recognition accuracy and more ideal recognition effect, thereby solving the problems of poor recognition accuracy and poor recognition effect of existing multimodal emotion analysis technologies. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0079] Figure 1 This is a flowchart illustrating a method for generating facial expressions in a digital human based on voice and facial features, as provided in an embodiment of this application.

[0080] Figure 2 yes Figure 1The illustrated embodiment provides a flowchart of a method for locating facial expression regions based on the YOLO algorithm;

[0081] Figure 3 yes Figure 1 The illustrated embodiment provides a flowchart of a method for extracting LBP feature vectors using the LBP operator;

[0082] Figure 4 yes Figure 1 The illustrated embodiment provides a flowchart of a method for pre-emphasis and frame-segmented windowing processing of continuous speech data;

[0083] Figure 5 yes Figure 1 The illustrated embodiment provides a flowchart of a method for extracting linear speech features;

[0084] Figure 6 yes Figure 1 The illustrated embodiment provides a flowchart of a method for extracting nonlinear speech features;

[0085] Figure 7 yes Figure 1 The illustrated embodiment provides a flowchart of a speech emotion recognition method using an SVM classifier;

[0086] Figure 8 yes Figure 1 The illustrated embodiment provides a flowchart of a method for making decision judgments on emotion recognition types based on a feature fusion algorithm based on a decision layer;

[0087] Figure 9 yes Figure 1 The illustrated embodiment provides a flowchart of a method for generating digital human facial expression animations based on the final emotion recognition type result;

[0088] Figure 10 This is a flowchart illustrating a digital human facial expression generation method provided in an embodiment of this application;

[0089] Figure 11 This is a flowchart illustrating an MFCC feature extraction method provided in an embodiment of this application;

[0090] Figure 12 This is a schematic diagram of the structure of a digital human expression generation system based on voice features and facial features, provided in an embodiment of this application.

[0091] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0092] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0093] The existing technical solutions have the following technical problems:

[0094] Multimodal sentiment analysis (MMA) refers to combining information from multiple perceptual modalities, such as facial expressions, voice, and physiological signals, to improve the accuracy of sentiment recognition. MMA technology can use deep learning methods to integrate information from different modalities. However, due to the complexity of various signal data, data preprocessing under different modalities and feature fusion are quite complex, resulting in poor fusion capabilities, low recognition accuracy, and ultimately, poor recognition performance.

[0095] To address the problems in existing technologies, the following embodiments of this application provide a digital human expression generation scheme based on speech and facial features. This scheme utilizes a feature layer fusion strategy to perform bimodal emotion recognition by combining speech and facial features through different feature fusion algorithms. This improves the accuracy of expression recognition and generates the most suitable expression. Furthermore, the technical solution of this application combines linear and nonlinear features of speech for speech recognition, resulting in more ideal speech emotion recognition performance. The nonlinear features can compensate for the shortcomings of traditional speech features in emotion recognition. This solution solves the problems in existing technologies where data preprocessing and feature fusion across different modalities are complex, fusion capabilities are poor, recognition accuracy is low, and the final recognition effect is poor.

[0096] To achieve the above objectives, see [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating a method for generating facial expressions in a digital human based on voice and facial features, provided as an embodiment of this application. Figure 1 As shown, this method for generating digital human expressions based on voice and facial features includes:

[0097] S110: Based on the YOLO algorithm, locate the emotion-related facial expression regions in multiple consecutive frames of images to be processed, and use the LBP operator to extract the LBP feature vectors of the facial expression regions. The technical solution provided in this application, before using the LBP operator to extract the LBP feature vectors of facial expressions, first uses the YOLO algorithm to locate emotion-related facial expression regions in multiple consecutive frames of images to be processed, such as the two eyes and the mouth—three regions that can well express emotions; then, the LBP operator is used to process the aforementioned facial expression regions; after extracting the LBP histograms of the facial expression regions, the histograms of these regions (e.g., including the aforementioned three histograms of the two eyes and the mouth) are connected sequentially to obtain the LBP feature vector of the facial expression image.

[0098] Specifically, as a preferred embodiment, such as Figure 2 As shown, the method for locating emotion-related facial expression regions in multiple consecutive frames of images to be processed using the YOLO algorithm includes:

[0099] S111: For multiple consecutive frames of images to be processed, each frame of the image to be processed is divided into multiple square grids. In this embodiment of the application, after acquiring the image to be processed, the image to be processed is first divided into multiple square grids of size S*S. This allows for the prediction of the corresponding bounding box for each square grid, and then the selection of the corresponding target window for the image to be processed, thereby identifying the facial expression regions related to emotion expression that need to be detected.

[0100] S112: For multiple square grids, the YOLO algorithm is used to predict multiple bounding boxes for each square grid, resulting in multiple target windows. For each square grid, this application can use the YOLO algorithm to predict B bounding boxes for each network. These bounding boxes include the confidence score of the target facial expression region and the probability of each bounding box in the corresponding expression category.

[0101] S113: Using a non-maximum suppression algorithm, redundant windows in multiple target windows are removed based on multiple borders to obtain the facial expression region.

[0102] In the technical solution provided in this application embodiment, the YOLO detection process is as follows: acquire multiple consecutive frames of images to be processed, first divide the images to be processed into an S*S grid; predict B bounding boxes for each grid (including the confidence that each bounding box is a target facial expression region and the probability of each bounding box region in multiple categories); based on the previous step, S*S*B target windows can be predicted, then target windows with low category probability are removed according to the confidence threshold, and finally, the non-maximum suppression algorithm (NMS) is used to remove redundant windows.

[0103] In addition, as a preferred embodiment, such as Figure 3 As shown, the steps for extracting the LBP feature vector of a facial expression region using the LBP operator include:

[0104] S114: Define the LBP radius and neighborhood in the facial expression region. The algorithm for extracting LBP feature vectors from the facial expression region using the LBP operator first needs to define the LBP radius and neighborhood in the facial expression region obtained by the YOLO detection algorithm described above. Specifically, a radius R is selected in the facial expression region, which determines the size of the neighborhood for the LBP algorithm. Typically, the value of R is between 1 and 3. Then, the neighborhood for the LBP algorithm is defined, which is usually a circular neighborhood centered on the center pixel. The center pixel is any pixel in the region, and the pixel located at the center is called the center pixel. Additionally, the number of pixels within this neighborhood is usually determined by the radius R.

[0105] S115: Set the LBP value of each pixel in the neighborhood based on the gray values ​​of its neighboring pixels and the center pixel to obtain the LBP feature vector of the image to be processed. The LBP value, or Local Binary Pattern, is obtained by comparing the gray values ​​of the pixels surrounding each pixel with the aforementioned center pixel. Specifically, if the gray value is greater than or equal to the gray value of the center pixel, the gray value is set to 1; otherwise, it is set to 0. This yields a large number of binary values. Arranging these binary values ​​in clockwise or counterclockwise order creates a binary string, which is the LBP value of that pixel.

[0106] S116: Calculate the average Euclidean distance between the LBP feature vectors of each frame of the image to be processed and all other frames in a series of consecutive frames. After obtaining the LBP value of each pixel, combining the LBP values ​​of each frame of the image to be processed yields the LBP feature vector of each frame. By calculating the Euclidean clustering of the LBP feature vectors of all frames, the average Euclidean distance between the LBP feature vectors of each frame of the image to be processed and all other frames can be calculated.

[0107] S117: Select a predetermined number of frames with the smallest average Euclidean distance as keyframe images for expression recognition. This application calculates the average Euclidean distance between each frame of the image to be processed and the LBP feature vectors of all other frames of the image to be processed, and then selects the k frames with the smallest average distance as the recognition targets for subsequent expression recognition.

[0108] S118: The LBP feature vectors of the keyframe images are used as the LBP feature vectors of the facial expression region and input into the random forest recognizer. After extracting the LBP features of k frames as the image feature dimensions, the average value of the image feature dimensions of the k frames (k≥2) is taken as the average feature dimension. The average feature dimension is then input into the random forest recognizer to obtain the recognition result.

[0109] Figure 1 The digital human expression generation method based on speech and facial features provided in the illustrated embodiment further includes the following steps after extracting the LBP feature vector of the facial expression region:

[0110] S120: Use a random forest recognizer to identify LBP feature vectors and obtain facial emotion recognition type. The facial emotion recognition type is obtained by averaging the feature dimensions of k frames as the average feature dimension and then inputting this average feature dimension into the random forest recognizer.

[0111] In machine learning algorithms, random forest is a multi-class classification algorithm based on decision tree algorithms. Because a single decision tree algorithm is prone to various problems in multi-class classification, to improve the recognition performance, multiple decision trees are combined, and random subspace and sampling aggregation methods are used to improve and optimize the decision trees, resulting in the random forest recognition algorithm. The algorithm flow of random forest recognition is as follows: Assume the number of training samples in the database is ||D||, L is the dimension of each data sample, and m is the number of features to be randomly extracted from the sample features, where m is a given constant greater than the dimension L of the data samples. ||D|| times of sampling with replacement are performed on the existing data sample set D, and the sampled data subset is... Finally, the meta-foundation can be built on the sampled data subset, and for any node in the decision tree, the best data information it carries can be calculated after randomly selecting m attribute features, thus enabling the classification and recognition of the aggressive random forest.

[0112] S130: Pre-emphasis and frame-segmented windowing processing are performed on the continuous speech data corresponding to multiple consecutive frames of images to be processed to obtain a pre-processed speech signal. In the technical solution provided in this application embodiment, the continuous speech data corresponding to multiple consecutive frames of images to be processed contains noise and is non-stationary and non-periodic, thus requiring pre-emphasis and frame-segmented windowing processing. Specifically, the acquired continuous speech data is pre-emphasized by using a first-order high-pass filter to increase the energy of the high-frequency part of the signal. Then, the pre-emphasized speech signal is frame-segmented and windowed. Because the speech signal is non-stationary and time-varying, it can be considered stationary and time-invariant in a short time, thus enabling segmented processing of the speech signal.

[0113] Specifically, as a preferred embodiment, such as Figure 4 As shown, the steps of pre-emphasizing and frame-segmenting windowing processing of continuous speech data corresponding to multiple consecutive frames of images to be processed to obtain a preprocessed speech signal include:

[0114] S131: Using a first-order high-pass filter, the calculation formula based on the pre-emphasized signal is as follows:

[0115]

[0116] Continuous speech data is pre-emphasized to obtain a pre-emphasized speech signal, wherein... For the nth voice data, Let y[n] be the (n-1)th speech data, and y[n] be the nth preprocessed speech signal; a is the filter coefficient used for pre-emphasis, and the value of a is usually between 0.95 and 0.98.

[0117] When continuous speech data is acquired, a first-order high-pass filter is used to pre-emphasize the continuous speech data, thereby increasing the energy of the high-frequency part of the signal. Assuming that the given time-domain input signal is x[n], the pre-emphasized signal after the above pre-emphasized signal calculation formula is y[n].

[0118] S132: The pre-emphasized speech signal is framed to obtain multiple frames of speech signal. The pre-emphasized speech signal y[n] needs to be framed and windowed. Because the speech signal is a non-stationary and time-varying signal, but within a short time range, it can be considered stationary and time-invariant. This short time is generally 10-30ms. Therefore, in order to reduce the overall non-stationary and time-varying effects of the speech signal, the speech signal needs to be segmented during speech signal processing. Each segment of the speech signal is called a frame, and the frame length is generally 25ms. In order to ensure a smooth transition between frames and maintain its continuity, the framing generally adopts an overlapping segmentation method to ensure that adjacent frames overlap by a certain amount. The time difference between the starting positions of adjacent frames is called the frame shift. In this application, the frame shift is taken as 10ms.

[0119] S133: Using the Hamming window as the window function, the calculation formula for the window function is as follows:

[0120]

[0121] The calculated window function is used to window the multi-frame speech signal to obtain a periodic speech signal; where L represents the number of points in the speech signal and w[n] represents the window function.

[0122] Because a Fourier Transform (FFT) will be performed on the signal later, and the FFT requires that the signal either ranges from -∞ to +∞ or is a periodic signal. In the real world, it's impossible to acquire signals with a time range from -∞ to +∞; only signals of finite duration can be acquired. Since the framed speech signal is aperiodic, frequency leakage will occur after the FFT. To minimize leakage error, a weighting function, also called a window function, is used for windowing. Windowing primarily aims to make the time-domain signal appear to better meet the periodicity requirement of the FFT process, thus reducing leakage.

[0123] In this embodiment, the Hamming window is selected as the window function, which makes the value of the above multi-frame speech signal at the window boundary approximately 0, thereby making the signal approach a periodic signal. The window function is shown in the above window function calculation formula.

[0124] S140: Using the MFCC feature extraction algorithm and the nonlinear feature extraction algorithm, linear and nonlinear speech features corresponding to the preprocessed speech signal are extracted, respectively. For example... Figure 11As shown, the MFCC feature extraction algorithm includes steps such as Fast Fourier Transform (FFT), MEL filter bank filtering, logarithmic operation, DCT, and dynamic feature extraction. Finally, the windowed periodic speech signal is converted into 29-dimensional MFCC data, thereby extracting linear speech features.

[0125] Specifically, as a preferred embodiment, such as Figure 5 As shown, in the above digital human expression generation method, step S140, which involves using the MFCC feature extraction algorithm and the nonlinear feature extraction algorithm to extract the linear and nonlinear speech features corresponding to the preprocessed speech signal, respectively, includes:

[0126] S141: Perform a Fast Fourier Transform on the windowed periodic speech signal to obtain the Fourier Transformed speech signal.

[0127] Since it's often difficult to discern the characteristics of a speech signal from its time-domain transformations, it's usually necessary to convert the speech signal to its frequency-domain energy distribution for observation. Because different energy distributions represent different speech characteristics, after multiplying by a Hamming window, each frame must undergo a Fast Fourier Transform (FFT) to obtain the energy distribution in the frequency spectrum. The FFT of each frame after windowing is performed to obtain the spectrum of each frame. The power spectrum of the speech signal is then obtained by taking the square of the modulus of the spectrum. Let the DFT of the speech signal be:

[0128]

[0129] in, Let N be the input speech signal, and let N represent the number of points in the Fourier transform.

[0130] Humans have varying perceptual abilities for speech signals of different frequencies: below 1 kHz, the relationship is linear with frequency; above 1 kHz, the relationship is logarithmic with frequency. The higher the frequency, the weaker the perceptual ability. To mitigate this poor perceptual ability, a Mel filter is needed to filter the periodic speech signal after the Fast Fourier Transform. The conversion formula between the Mel frequency m and the speech signal frequency f is as follows:

[0131]

[0132] S142: Using a Mel filter, the calculation formula for the center frequency of the Mel filter is as follows:

[0133]

[0134] Calculate the center frequency of the Mel filter, where, and These represent the highest and lowest frequencies within the application range of the Mel filter, respectively, and N represents the number of points in the periodic speech signal when performing a Fast Fourier Transform. This represents the sampling frequency, and M is the number of triangular filters in the Mel filter. This represents the center frequencies of m Mel filters. represent The inverse function of , where the constant 1125 is derived from 2595 / ln(10) ≈1125, is used directly in this application without modification.

[0135] The Mel filter is composed of multiple triangular filters. It allows for feature extraction from preprocessed speech signals. The center frequency corresponding to the Mel filter is... The formula for calculating the center frequency of the Mel filter is as above, and will not be repeated here.

[0136] In addition, after obtaining the center frequency of the Mel filter, it is necessary to use this center frequency to calculate the frequency response. Specifically:

[0137] S143: The center frequency using the Mel filter, calculated according to the frequency response formula:

[0138]

[0139] Calculate the frequency response of the Mel filter; where, This represents the center frequency of the m-th Mel filter. This represents the center frequency of the (m-1)th Mel filter. This represents the center frequency of the (m+1)th Mel filter (or the triangular filter if it is the same Mel filter). m is the number of filters, and k is the center frequency of a specific Mel filter.

[0140] Using the frequency response of this Mel filter, the logarithmic energy, i.e., the cepstral coefficients, can be calculated. Specifically, in the frequency domain, assuming the signal after the vocal cords vibrate is... Then it passes through the cavity, at which point the cavity can be considered a filter, using... To describe, the generated sound signal is X(k) = · Next, we take the logarithm, and finally perform an inverse Fourier transform to obtain the cepstral coefficients. Specifically,

[0141] S144: The frequency response and Fourier transform of the speech signal using the Mel filter, calculated according to the logarithmic energy formula:

[0142]

[0143] Calculate the logarithmic energy of the output of the m-th filter; where, This represents the logarithmic energy of the output of the m-th filter. This represents the speech signal after Fourier transform. This represents the frequency response of the Mel filter.

[0144] Logarithmic operations include taking the absolute value and the log operation. Taking the absolute value uses only the amplitude value, ignoring the phase effect, because phase information has little effect in speech recognition. The log operation is used to separate the envelope and details; the envelope represents timbre, and the details represent the envelope pitch. Obviously, speech recognition aims to identify timbre. Furthermore, human perception is proportional to the logarithm of frequency, which can be simulated using the logarithm. After the FFT transform, convolution becomes multiplication; after taking the logarithm, multiplication becomes addition, thus converting the convolutional signal into an additive signal.

[0145] S145: Using the logarithmic energy output by the m-th filter, the MFCC coefficients of the speech signal are obtained by transforming it according to the discrete cosine transform formula.

[0146] Because the pitch information and tract information became additive in the previous step, it is necessary to separate the above information. The above information has the following properties: in the spectrogram (note that it is within one frame of FFT transformation), (1) the pitch information changes rapidly in the frequency domain; (2) the tract information changes slowly in the frequency domain; therefore, performing another DCT can separate them, and this process is called "cep spectral domain". Therefore, the low-frequency part of the cepstral domain describes the tract information, and the high-frequency part describes the pitch information.

[0147] The specific DCT formula is as follows:

[0148]

[0149] Substituting the logarithmic energy s(m) into the discrete cosine transform, the L-order Mel-scale Cepstrum parameters are obtained, where L refers to the order of the MFCC coefficients, typically taken as 12-16. Here, M is the triangular filter scheme, and C(n) represents the MFCC coefficients obtained from the discrete cosine transform.

[0150] Furthermore, the volume (i.e., energy) of a single frame is also an important feature of speech and is very easy to calculate. Therefore, as... Figure 11 As shown, the logarithmic energy of a frame is usually added (defined as the sum of the squares of the signal within a frame, then taking the logarithmic value to the base 10, and then multiplying by 10), which adds one dimension to the basic speech features of each frame, including a logarithmic energy and the remaining cepstral parameters.

[0151] Standard cepstral parameters (MFCC) only reflect the static characteristics of speech parameters; the dynamic characteristics of speech can be described by the difference spectrum of these static features. Experiments have shown that combining dynamic and static features is the only way to effectively improve the system's recognition performance. Specifically, as shown in step S146 below.

[0152] S146: Using the MFCC coefficients of the speech signal, calculated according to the differential parameter formula:

[0153]

[0154] Calculate the dynamic difference parameters of the preprocessed speech signal as the linear speech features corresponding to the preprocessed speech signal; where dt represents the t-th first-order difference; Ct represents the t-th cepstral coefficient; Q represents the order of the cepstral coefficient; and K represents the time difference of the first derivative.

[0155] Through the calculation of the above MFCC feature extraction algorithm, a 39-dimensional MFCC, i.e., dynamic difference parameter, can be extracted as the linear speech feature corresponding to the above preprocessed speech signal.

[0156] In addition, this application embodiment also needs to extract the nonlinear speech features corresponding to the preprocessed speech signal.

[0157] The extraction of nonlinear speech features from speech signals mainly employs the approximate entropy method. Approximate entropy (ApEn) has advantages such as not requiring a large amount of data and strong noise and interference resistance. The ApEn algorithm mainly distinguishes the complexity of the speech signal time series occurrence process from a statistical perspective, reflecting the self-similarity of the speech signal time series in terms of patterns.

[0158] Specifically, as a preferred embodiment, such as Figure 6 As shown, step S140 above, which involves using the MFCC feature extraction algorithm and the nonlinear feature extraction algorithm to extract linear and nonlinear speech features corresponding to the preprocessed speech signal, respectively, includes:

[0159] S147: Using the approximate entropy algorithm, for the N-point pre-emphasized speech signal sequence {y(n)} of the pre-emphasized speech signal combination, the formula for calculating the w-dimensional vector is as follows:

[0160]

[0161] The combination yields a w-dimensional vector, where X(n) represents the w-dimensional vector, w represents the dimension, and n represents the nth pre-emphasized speech signal.

[0162] ApEn's algorithm first assumes a time series {y(n)} of length N, where n = 1, 2, 3, ..., N. Then, it reconstructs a new w-dimensional vector X(n), whose vector form is shown in the w-dimensional vector calculation formula above. After calculating any w-dimensional vector X(n) and all remaining vectors X(k), the following steps are also included:

[0163] S148: Calculate the relative Euclidean distance between any w-dimensional vector X(n) corresponding to the N-point pre-emphasized speech signal sequence and all remaining vectors X(k).

[0164] The formula for calculating the relative Euclidean clustering is as follows:

[0165]

[0166] S149: Count the number of w-dimensional vectors whose relative Euclidean distance is less than or equal to a predetermined distance threshold.

[0167] Statistical relative Euclidean distance The number of w-dimensional vectors X(n) nd (where r is the set threshold and SD is the standard deviation of the time series) is denoted as:

[0168]

[0169] Where nd represents the number of w-dimensional vectors X(n), and N represents the total number of dimensions; Let represent the proportion of the total number of w-dimensional vectors whose relative Euclidean distance between the nth w-dimensional vector and all w-dimensional vectors (including itself) is ≤ γ under the similarity tolerance γ.

[0170] S1410: The number of w-dimensional vectors used, calculated based on the proximity formula:

[0171]

[0172] Calculate the proximity between multiple w-dimensional vectors, where W represents the dimension of the vectors and N represents the total dimension of the speech sequence. This represents the degree of proximity between w-dimensional vectors.

[0173] S1411: Using the proximity between w-dimensional vectors and the proximity between w+1-dimensional vectors, the approximate entropy calculation formula is obtained:

[0174]

[0175] The approximate entropy is calculated and used as the nonlinear speech feature corresponding to the preprocessed speech signal, where, Represents the degree of proximity between w-dimensional vectors. It represents the degree of proximity between w+1 dimensional vectors.

[0176] The approximate entropy A obtained by calculating it in the above manner can be used as the nonlinear speech feature corresponding to the preprocessed speech signal. In this embodiment, speech recognition is performed by combining linear and nonlinear speech features, resulting in a more ideal recognition effect for speech emotion recognition. The nonlinear features can compensate for the shortcomings of traditional speech features in emotion recognition.

[0177] Figure 1 The technical solution provided in the illustrated embodiment, after extracting the linear and nonlinear speech features corresponding to the preprocessed speech signal, further includes the following steps:

[0178] S150: Input linear and nonlinear speech features into an SVM classifier to perform speech emotion recognition and obtain the speech emotion recognition type.

[0179] Specifically, as a preferred embodiment, such as Figure 7 As shown, in the above digital human expression generation method, step S150, which involves inputting linear and nonlinear speech features into an SVM classifier for speech emotion recognition to obtain the speech emotion recognition type, includes:

[0180] S151: Divide the data samples corresponding to linear speech features and the data samples corresponding to nonlinear speech features into training data samples and test data samples respectively.

[0181] S152: Using a ten-fold cross-validation method, the SVM classifier is trained using training data samples and tested using test data samples to obtain the recognition accuracy of various speech emotion recognition types.

[0182] S153: The recognition accuracy of multiple speech emotion recognition types is averaged to obtain the average recognition accuracy as the final speech emotion recognition type.

[0183] In the technical solution provided in this application embodiment, the cepstral coefficients MFCC obtained in the above steps and the approximate entropy (ApEn) of the nonlinear speech feature parameters obtained in S2.2.2 are input into the SVM recognition classifier to perform emotion recognition on the linear and nonlinear speech features of the speech data to be processed, respectively. In this application embodiment, the SVM recognition classifier method is used for speech emotion recognition, and the classifier adopts a ten-fold cross-validation method. That is, the data sample is randomly divided into ten equal subsets, and each subset is used as the test data sample for emotion recognition in turn. The other nine subsets are classified as the training data samples for speech emotion recognition. The training model is obtained by training the SVM recognition classifier, and the recognition rate of various emotions (e.g., 6 types) is obtained by testing with the test data subsets. The recognition accuracy obtained after ten sets of cross-validation is obtained, and the average recognition rate obtained by averaging the ten sets of recognition accuracy is used as the final result of speech emotion recognition type. The ten-fold cross-validation method can avoid the overfitting phenomenon that may be caused by training on a single data sample in the database through multiple training and testing.

[0184] Figure 1 The technical solution provided in the illustrated embodiment, after obtaining the voice emotion recognition type, further includes:

[0185] S160: Based on the feature fusion algorithm based on the decision layer, a decision judgment is made on the facial emotion recognition type and the voice emotion recognition type to obtain the final emotion recognition type result. This embodiment of the application utilizes a feature layer fusion strategy to perform bimodal emotion recognition by applying different feature fusion algorithms to the voice emotion recognition type and the facial expression recognition type. This improves the accuracy of expression recognition and generates the most suitable expression. This multimodal approach to emotion recognition improves the accuracy of emotion recognition, resulting in more appropriate expressions and enhancing realism and user experience.

[0186] Specifically, as a preferred embodiment, such as Figure 8 As shown, the steps described above for making decisions on facial emotion recognition type and voice emotion recognition type based on the feature fusion algorithm of the decision layer to obtain the final emotion recognition type result include:

[0187] S161: Obtain the recognition probability corresponding to the facial emotion recognition type and the recognition probability corresponding to the voice emotion recognition type.

[0188] S162: Using the recognition probabilities corresponding to facial emotion recognition types and voice emotion recognition types, a feature fusion algorithm is used to obtain the recognition probabilities corresponding to m classification results. Based on the feature fusion algorithm at the decision layer, assuming that the test object can obtain the recognition probabilities corresponding to m classification results when its features of n modalities are recognized and classified by n classifiers, then based on these recognition probabilities, the recognition probabilities corresponding to the m classification results can be obtained using the summation decision rule calculation formula.

[0189] S163: Using the recognition probabilities corresponding to the m classification results, calculate the formula according to the summation decision rule:

[0190]

[0191] The recognition probabilities corresponding to each of the m classification results are obtained; where, This represents the recognition probability corresponding to the x-th classification result. This represents the recognition probability corresponding to the j-th classification result obtained after passing through the i-th classifier.

[0192] S164: From the recognition probabilities corresponding to the m classification results, select the emotion recognition type with the highest recognition probability value as the final emotion recognition type result.

[0193] In the technical solution provided in this application embodiment, for multi-class recognition, the feature fusion algorithm based on the decision layer assumes that when the test object's features of n modalities are recognized and classified by n classifiers, the recognition probabilities corresponding to m classification results are respectively { Then, based on the specific rules of decision fusion, the recognition probability is calculated to obtain the set of recognition probabilities for m classification results. Finally, select the one with the highest classification probability from the obtained set. The corresponding label category is taken as the final result of the test object in the recognition process. In this embodiment, a summation decision rule is used to solve the problem. The summation decision rule compares the summed probabilities of the same category generated by each different classifier for the same experimental object, and the emotion recognition type corresponding to the largest value is the final classification result. The final emotion recognition type result is then output.

[0194] The technical solution provided in this application, after selecting the emotion recognition type with the highest recognition probability value as the final emotion recognition type result, further includes the following steps:

[0195] S170: Generate digital human facial animation based on the final emotion recognition type result.

[0196] Specifically, as a preferred embodiment, such as Figure 9 As shown, in the above method for generating digital human expressions, step S170, which involves generating digital human expression animations based on the final emotion recognition type result, includes:

[0197] S171: Using the final emotion recognition type result, retrieve the facial animation corresponding to the final emotion recognition type result from the facial image library.

[0198] S172: Use facial expression animation technology to generate facial expression animations onto the corresponding facial areas of the digital human. Based on the final emotion recognition type result, call the corresponding expression images from various pre-created expression image libraries, and then use facial expression animation technology to generate expression images from the expression image library onto the digital human.

[0199] In addition, such as Figure 10 As shown in the embodiment of this application, the digital human expression generation method includes:

[0200] S201: Obtain emotional audio from video.

[0201] S202: Speech feature extraction.

[0202] S203: Recognition Classifier: The extracted speech features are input into this recognition classifier for classification to obtain the speech emotion recognition type.

[0203] S204: Obtain facial expression images from video.

[0204] S205: Facial expression feature extraction.

[0205] S206: Recognition Classifier: Input the extracted facial expression features into the recognition classifier for classification to obtain the facial emotion recognition type.

[0206] S207: Decision-level feature fusion: The above-mentioned speech emotion recognition type and face emotion recognition type are used to make a decision using a feature fusion algorithm based on the decision layer, so as to obtain the final emotion recognition type result.

[0207] S208: Recognition Result Output: Output the final emotion recognition type result, and then call the facial animation corresponding to the final emotion recognition type result to generate it onto the digital human's facial area.

[0208] In summary, the digital human expression generation method based on speech and facial features provided in this application locates the facial expression region in multiple consecutive frames of images to be processed using the YOLO algorithm, extracts the LBP feature vector associated with the expression in the facial expression region using the LBP operator, and then uses a random forest classifier to identify the LBP feature vector, thereby obtaining the facial emotion recognition type. This allows the corresponding facial emotion recognition type to be obtained from facial features. Next, the continuous speech data corresponding to the multiple consecutive frames of images to be processed is pre-emphasized and windowed to obtain a pre-processed speech signal. The MFCC feature extraction algorithm and the nonlinear feature extraction algorithm are used to extract the linear and nonlinear speech features corresponding to the pre-processed speech signal, respectively. Finally, the linear and nonlinear speech features are input into an SVM classifier for speech emotion recognition, thus obtaining the corresponding speech emotion recognition type. Finally, the facial emotion recognition type and the speech emotion recognition type are fused using a feature fusion algorithm based on a decision layer, and a decision is made to obtain the final emotion recognition type result. The above method can improve the accuracy of emotion type recognition through bimodal emotion analysis of voice and facial features, and then generate more accurate digital human facial animation based on the final emotion recognition result. At the same time, it combines linear and nonlinear voice features for speech recognition, resulting in higher recognition accuracy and more ideal recognition effect, thereby solving the problems of poor recognition accuracy and poor recognition effect of existing multimodal emotion analysis technologies.

[0209] In addition, based on the same concept of the above method embodiments, this application also provides a digital human expression generation system based on voice features and facial features to implement the above method of this application. Since the principle and method of solving the problem in this system embodiment are similar, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, and will not be described in detail here.

[0210] See Figure 12 , Figure 12 This is a schematic diagram illustrating the structure of a digital human expression generation system based on voice and facial features, provided as an embodiment of this application. Figure 12 As shown, the facial expression generation system includes:

[0211] The processor 1001, communication bus 1002, communication module 1003, memory 1004, and a digital human expression generation program based on voice features and facial features stored in memory 1004 and running on processor 1001, wherein when the digital human expression generation program based on voice features and facial features is executed by the processor, the steps of the digital human expression generation method based on voice features and facial features provided in any of the above embodiments are implemented.

[0212] This application utilizes a feature layer fusion strategy to perform dual-modal emotion recognition by combining speech emotion features and facial expression features through different feature fusion algorithms. This can improve the accuracy of expression recognition and generate the most suitable expression.

[0213] This application combines linear and nonlinear features of speech for speech recognition, resulting in a more ideal speech emotion recognition effect. The nonlinear features can compensate for the shortcomings of traditional speech features in emotion recognition.

[0214] In summary, the digital human expression generation scheme based on voice and facial features provided in this application embodiment improves the accuracy of emotion recognition and generates more appropriate expressions through multimodal emotion recognition, thereby enhancing the user experience. Furthermore, the selected emotional voice nonlinear features exhibit superior recognition performance compared to traditionally used emotional voice recognition features in some emotional aspects. Integrating traditional emotional voice features with the nonlinear features presented in this paper for voice emotion recognition yields even more significant recognition results. Additionally, this application embodiment uses YOLO to detect the most emotionally expressive parts of the human image, such as the eyes and mouth, for emotion recognition, thereby improving the efficiency and accuracy of recognition.

[0215] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, and optical storage) containing computer-usable program code.

[0216] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0217] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0218] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0219] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, third, etc., does not indicate any order. These words can be interpreted as names.

[0220] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0221] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating facial expressions in digital humans based on voice and facial features, characterized in that, include: Based on the YOLO algorithm, the regions of facial expressions related to emotion are located in multiple consecutive frames of images to be processed, and the LBP operator is used to extract the LBP feature vectors of the facial expression regions. The LBP feature vector is identified using a random forest recognizer to obtain the facial emotion recognition type. The continuous speech data corresponding to the continuous multi-frame images to be processed are pre-emphasized and framed windowed to obtain a pre-processed speech signal. The linear speech features and nonlinear speech features corresponding to the preprocessed speech signal are extracted using the MFCC feature extraction algorithm and the nonlinear feature extraction algorithm, respectively. The linear and nonlinear speech features are input into an SVM classifier to perform speech emotion recognition, and the speech emotion recognition type is obtained. Based on the feature fusion algorithm based on the decision layer, the facial emotion recognition type and the voice emotion recognition type are judged to obtain the final emotion recognition type result; Based on the final emotion recognition type result, generate digital human facial expression animation.

2. The digital human expression generation method according to claim 1, characterized in that, The step of locating emotion-related facial expression regions in multiple consecutive frames of images to be processed using the YOLO algorithm includes: For the continuous multiple frames of images to be processed, each frame of the image to be processed is divided into multiple square grids; For the multiple square grids, the YOLO algorithm is used to predict multiple borders for each square grid, resulting in multiple target windows; The non-maximum suppression algorithm is used to remove redundant windows from the multiple target windows based on the multiple borders to obtain the facial expression region.

3. The digital human expression generation method according to claim 1 or 2, characterized in that, The step of extracting the LBP feature vector of the facial expression region using the LBP operator includes: Define the LBP radius and neighborhood in the facial expression region; The LBP value of each pixel in the neighborhood is set according to the gray values ​​of the neighboring pixels and the center pixel of each pixel in the neighborhood, so as to obtain the LBP feature vector of the image to be processed. Calculate the average Euclidean distance between the LBP feature vectors of each frame of the image to be processed and all other frames in the continuous multi-frame image to be processed. A predetermined number of frames with the smallest average Euclidean distance are selected as the recognition keyframes for facial expression recognition. The LBP feature vector of the recognition keyframe image is used as the LBP feature vector of the facial expression region and input into the random forest recognizer.

4. The digital human expression generation method according to claim 1, characterized in that, The step of performing pre-emphasis and frame-segmented windowing processing on the continuous speech data corresponding to the continuous multiple frames of images to be processed to obtain a preprocessed speech signal includes: Using a first-order high-pass filter, the calculation formula based on the pre-emphasized signal is as follows: ; The continuous speech data is pre-emphasized to obtain a pre-emphasized speech signal, wherein, For the nth voice data, Let y[n] be the (n-1)th speech data, y[n] be the nth preprocessed speech signal, and a be the filter coefficients used for pre-emphasis. The pre-emphasized speech signal is processed into frames to obtain multiple frames of speech signal; Using a Hamming window as the window function, the calculation formula for the window function is as follows: ; The calculated window function is used to window the multi-frame speech signal to obtain a periodic speech signal; where L represents the number of points in the speech signal and w[n] represents the window function.

5. The digital human expression generation method according to claim 4, characterized in that, The step of using the MFCC feature extraction algorithm and the nonlinear feature extraction algorithm to extract the linear speech features and nonlinear speech features corresponding to the preprocessed speech signal, respectively, includes: The windowed periodic speech signal is subjected to a fast Fourier transform to obtain the Fourier transformed speech signal. Using the Mel filter, the center frequency of the Mel filter is calculated according to the formula: ; Calculate the center frequency of the Mel filter, where, and These represent the highest and lowest frequencies within the application range of the Mel filter, respectively, and N represents the number of points in the periodic speech signal when performing a Fast Fourier Transform. This represents the sampling frequency, and M is the number of triangular filters in the Mel filter. This represents the center frequencies of m Mel filters. represent The inverse function; Using the center frequency of the Mel filter, calculate the frequency response using the formula: ; Calculate the frequency response of the Mel filter; where, This represents the center frequency of the m-th Mel filter. This represents the center frequency of the (m-1)th Mel filter. This represents the center frequency of the (m+1)th Mel filter; Using the frequency response of the Mel filter and the Fourier-transformed speech signal, the logarithmic energy calculation formula is applied: ; Calculate the logarithmic energy of the output of the m-th filter; where, This represents the logarithmic energy of the output of the m-th filter. This represents the speech signal after the Fourier transform. This represents the frequency response of the Mel filter; Using the logarithmic energy output by the m-th filter, the MFCC coefficients of the speech signal are obtained by transforming it according to the discrete cosine transform formula; Using the MFCC coefficients of the speech signal, the differential parameter calculation formula is applied: ; Calculate the dynamic difference parameters of the preprocessed speech signal as the linear speech features corresponding to the preprocessed speech signal; where Ct represents the t-th cepstral coefficient; Q represents the order of the cepstral coefficient; and K represents the time difference of the first derivative.

6. The digital human expression generation method according to claim 4, characterized in that, The step of using the MFCC feature extraction algorithm and the nonlinear feature extraction algorithm to extract the linear speech features and nonlinear speech features corresponding to the preprocessed speech signal, respectively, includes: Using the approximate entropy algorithm, for the N-point pre-emphasized speech signal sequence {y(n)} of the pre-emphasized speech signal combination, the following formula is used for calculation: ; The combination yields a w-dimensional vector, where X(n) represents the w-dimensional vector, w represents the dimension, and n represents the nth point of the pre-emphasized speech signal; Calculate the relative Euclidean distance between any w-dimensional vector X(n) corresponding to the N-point pre-emphasized speech signal sequence and all remaining vectors X(k); Count the number of w-dimensional vectors whose relative Euclidean distance is less than or equal to a predetermined distance threshold; Using the number of the w-dimensional vectors, the following formula is used to calculate the proximity: ; Calculate the proximity between multiple w-dimensional vectors, where, This represents the number of w-dimensional vectors, where W represents the dimension of the vectors and N represents the total number of points in the speech sequence. This represents the degree of proximity between w-dimensional vectors; Using the proximity between the w-dimensional vectors and the proximity between the w+1-dimensional vectors, the approximate entropy calculation formula is obtained: ; The approximate entropy is calculated and used as the nonlinear speech feature corresponding to the preprocessed speech signal, wherein, Represents the degree of proximity between w-dimensional vectors. It represents the degree of proximity between w+1 dimensional vectors.

7. The digital human expression generation method according to claim 1, characterized in that, The step of inputting the linear and nonlinear speech features into an SVM classifier for speech emotion recognition to obtain the speech emotion recognition type includes: The data samples corresponding to the linear speech features and the data samples corresponding to the nonlinear speech features are divided into training data samples and test data samples respectively. The SVM classifier is trained using the training data samples and tested using the test data samples, using a 10-fold cross-validation method to obtain various speech emotion recognition types and their recognition accuracy. The recognition accuracy of the various voice emotion recognition types is averaged to obtain the average recognition accuracy as the final voice emotion recognition type.

8. The digital human expression generation method according to claim 1, characterized in that, The step of making a decision on the facial emotion recognition type and the voice emotion recognition type based on the feature fusion algorithm based on the decision layer to obtain the final emotion recognition type result includes: Obtain the recognition probability corresponding to the facial emotion recognition type and the recognition probability corresponding to the voice emotion recognition type; Using the recognition probability corresponding to the facial emotion recognition type and the recognition probability corresponding to the voice emotion recognition type, the recognition probability corresponding to m classification results is obtained based on the feature fusion algorithm; Using the recognition probabilities corresponding to the m classification results, the formula is calculated according to the summation decision rule: ; The recognition probabilities corresponding to each of the m classification results are obtained; where, This represents the recognition probability corresponding to the x-th classification result. This represents the recognition probability corresponding to the j-th classification result obtained after passing through the i-th classifier; From the recognition probabilities corresponding to the m classification results, the emotion recognition type with the highest recognition probability value is selected as the final emotion recognition type result.

9. The digital human expression generation method according to claim 1, characterized in that, The step of generating digital human facial expression animation based on the final emotion recognition type result includes: Using the final emotion recognition type result, retrieve the corresponding facial animation from the facial image library; The facial expression animation is generated onto the facial area corresponding to the digital human using facial expression animation technology.

10. A digital human expression generation system based on voice features and facial features, characterized in that, include: The present invention includes a memory, a processor, and a digital human expression generation program based on voice features and facial features stored in the memory and running on the processor, wherein the digital human expression generation program based on voice features and facial features, when executed by the processor, implements the steps of the digital human expression generation method based on voice features and facial features as described in any one of claims 1 to 9.

Images