A dual-modal astronaut emotion recognition method based on facial expressions and speech in videos

Abstract

The present invention discloses a dual-modal astronaut emotion recognition method based on facial expressions and voices in videos: for the facial expression modality in the video, video frames are intercepted from RAVDESS to determine whether they contain faces, and the original images containing faces are cropped to the facial area; the original facial images are edited into facial images in a simulated space environment; the pre-processed facial images are input into a convolutional neural network ATT_CNN containing an attention mechanism to obtain astronaut facial expression modality emotion recognition results; for the voice modality in the video, random noise is superimposed on audio data in RAVDESS to obtain a voice signal in a simulated space environment; the signal is input into a voice enhancement network to obtain a clean voice signal, and its MFCC features are extracted and input into a CNN_LSTM network to obtain astronaut voice modality emotion recognition results; finally, the facial expression emotion recognition results and the voice emotion recognition results are decision-fused to obtain the final dual-modal astronaut emotion recognition results.

Description

Technical Field

[0001] The present invention relates to the technical fields of emotion recognition, image processing and speech processing for astronauts in the field of manned space flight, and in particular to a dual-modal astronaut emotion recognition method based on facial expressions and speech in videos. Background Art

[0002] As early as the mid-1980s, Russian space biomedical experts pointed out that "it's not medicine, but psychology that limits human space exploration." With the continuous advancement of human space exploration, the length and complexity of manned space missions are also increasing. Long-term manned flight has become the mainstream mission in the current space field. During space missions, astronauts are often exposed to extreme environments such as ionizing radiation, weightlessness, noise, isolation, hypoxia, and circadian rhythm (biological clock) disruptions. From a psychological perspective, in such extreme environments, astronauts may experience maladaptive problems in their mental state, emotional reactions, and interpersonal communication, which may affect their work and even endanger the completion of the space mission and the safety of the astronauts. During long-term space missions, it is extremely important to pay attention to astronauts' emotional changes and mental health.

[0003] In 2018, CIMON, the world's first AI-powered astronaut assistant, jointly developed by IBM, Airbus, and the German Aerospace Center (DLR), was sent to the International Space Station for 14 months of space testing. In 2019, the upgraded CIMON-2 returned to the ISS. Unlike the first generation, CIMON-2 utilizes IBM Cloud's Watson Tone Analyzer, capable of assessing and responding to astronauts' emotions through voice recognition. In 2020, Adrian et al. proposed a new emotion detection system based on anthropological facial landmarks (AFL) for isolated and enclosed environments. This system outperformed existing emotion detection models based on action units (AUs). In 2010, Gao Hui et al. established a stress corpus in a simulated spaceflight environment and, using nonlinear features based on the Teager energy operator and a hidden Markov model, achieved a 98.6% recognition rate for calm-to-annoyance recognition. In 2012, software developed by Beijing Normal University, which uses brainwaves to monitor astronauts' basic cognitive functions and emotional changes in space, was launched into space aboard the Shenzhou 9 spacecraft. In 2018, Liu Yang and others developed a speech emotion recognition model and software platform using feature extraction and Gaussian mixture model methods, achieving an 86% recognition rate for stressful emotions in a Chinese corpus. Currently, emotion recognition for astronauts, both domestically and internationally, is primarily single-modal, focusing on facial image-based, speech-based, and EEG-based emotion recognition. However, single-modal emotion recognition suffers from inevitable drawbacks during its development, such as low recognition accuracy and uncertain feature quality. Therefore, to improve the robustness of emotion recognition algorithms, it is possible to fuse information from multiple single modalities, leading to the development of multimodal emotion recognition. Summary of the Invention

[0004] This paper provides a dual-modal astronaut emotion recognition method based on facial expressions and speech in video, addressing the low accuracy and lack of guaranteed feature quality of existing single-modal emotion recognition techniques. This method employs a decision-layer fusion approach, extracting and recognizing features from astronauts' facial expressions and speech separately. Fusion is then performed at the decision-making end to generate dual-modal emotion recognition results, enabling astronaut emotion recognition for long-term space missions.

[0005] To achieve the above object, the technical solution of the present invention is:

[0006] A dual-modal astronaut emotion recognition method based on facial expressions and speech in videos includes the following steps:

[0007] S1. Extract video frames from the existing public dataset RAVDESS, use the Viola-Jones face detection algorithm to determine whether they contain faces, and crop the original images containing faces to the facial region.

[0008] S2. Edit the original facial image into a facial image in a simulated space environment using a bilateral filtering algorithm and a liquefaction algorithm;

[0009] S3. Input the preprocessed facial image into the convolutional neural network ATT_CNN with an attention mechanism to obtain the astronaut's facial expression modality emotion recognition results;

[0010] S4. Superimpose random noise on the audio data from the public dataset RAVDESS to generate speech signals in a simulated space environment.

[0011] S5. The signal is input into a speech enhancement network to obtain a clean speech signal and extract its MFCC features. Subsequently, the signal is input into a CNN-LSTM network, which is a fusion of a convolutional neural network (CNN) and a long short-term memory (LSTM) network, to obtain the astronaut's speech modality emotion recognition results.

[0012] S6. Make a decision fusion based on the facial expression emotion recognition results and the speech emotion recognition results to obtain the final dual-modal astronaut emotion recognition results.

[0013] Furthermore, the step S1 is specifically as follows:

[0014] S11. Use Haar-like features to describe facial features. The feature template contains two types of rectangles, white and black. Haar-like features are defined as the difference between the sum of the pixels in the white and black rectangles. This reflects the local grayscale changes of the image to a certain extent, quantifies facial features, and distinguishes between faces and non-faces.

[0015] S12. Create an integral image and use it to quickly obtain several different rectangular features. For a point (x, y) with x as the horizontal coordinate and y as the vertical coordinate in the integral image coordinate system, the integral image value of the point is equal to the sum of all pixels located in the upper left corner of the point. The expression is: S(x, y) = ∑ x′≤x ∑ y′≤y f(x′,y′), where S(x,y) is the integral image value of the point (x,y), f(x′,y′) is the pixel value of the point with abscissa x′ and ordinate y′ in the coordinate system, and the integral image satisfies the following relationship: S(x,y)=f(x,y)+S(x-1,y)+S(x,y-1)-S(x-1,y-1), where f(x,y) is the pixel value of the point A(x,y), S(x-1,y) is the integral image value of the point (x-1,y), S(x,y-1) is the integral image value of the point (x,y-1), and S(x-1,y-1) is the integral image value of the point (x-1,y-1);

[0016] S13. Using the AdaBoost algorithm for training, an image usually generates a feature set that is much larger than the image dimension. For example, a 24*24 image will generate 162336 rectangular features. However, not all features are useful. Therefore, an AdaBoost feature classifier is needed to extract useful features. AdaBoost is a strong classifier formed by linearly combining a series of weak classifiers. Assuming that an image generates N features, for the i-th feature f i , i=1,2,3…N, the strong classifier expression is: Among them, a is a detection sub-window, α i is the coefficient, h(a) is a strong classifier, h i (a) is a weak classifier, which is represented by a simple threshold function: Among them, θ i is the threshold;

[0017] S14. Build a cascade classifier. Since the face region occupies only a small portion of a normal image, training all features would be computationally intensive. Therefore, a cascade of multiple strong classifiers is employed. The first classifier uses a small number of features to remove most non-face regions, while the subsequent classifier uses more complex features to remove even more complex non-face regions.

[0018] S15. After face detection obtains the face bounding box, the original image can be cropped to the face area.

[0019] Furthermore, the step S2 is specifically as follows:

[0020] S21. Use the bilateral filtering algorithm to achieve facial skin smoothing of the original image. In the facial contour area, the grayscale value difference between adjacent pixels is large, forming a more obvious boundary, visually forming the facial contour and facial features. Based on this characteristic, the weight of the pixels with a large grayscale difference from the center pixel in the area can be increased to strengthen the impact of the boundary on the image, thereby achieving the preservation of the facial boundary. The original Gaussian filter only uses the distance difference as a variable. On this basis, the pixel value difference can be added. That is, the closer the distance to the center point, the higher the weight, but the larger the difference in pixel value from the center point, the smaller the weight. In addition, the influence of the pixel value difference is greater than the distance value. The formula for bilateral filtering is: in, Where G(p) is the bilateral filter function, p is the center pixel, x and y are the horizontal and vertical coordinates of point p, q is another pixel calculated with the center pixel, m and n are the horizontal and vertical coordinates of point q, F(p) is the pixel value of point p, F(q) is the pixel value of point q, Ω represents the image, σs is the standard deviation of the spatial domain Gaussian function, σ r is the standard deviation of the Gaussian function in pixel domain, is the spatial domain kernel, is the pixel domain kernel, and e is a natural constant. The essence of bilateral filtering is the superposition of two Gaussian functions, one with distance as the variable and the other with pixel difference as the variable, and the convolution sum is performed, η(p) -1 It is normalized. is a Gaussian function with pixel value as variable, It is a Gaussian function with pixel difference as variable;

[0021] S22. Use the dlib model and liquefaction algorithm to extract 68 facial feature points and realize facial swelling of the original image. The liquefaction algorithm is essentially a pixel migration or image distortion algorithm. It can be regarded as moving a certain pixel to a target position. For example, a swollen face is to stretch the pixels on the periphery of the face outward. That is, the position of the target point is calculated and the current point is replaced with the pixel of the target point. In order to make the changed image smoother, resampling technology can be used for interpolation processing. Assume that the current point is X(x,y) and the center point of the deformation area is designated as C(C x ,C y ), C x and C y are the horizontal and vertical coordinates of point C, the radius of the deformation area is r, and the deformation end point (from the center point to a certain position M) is adjusted to M (M x ,M y ), M x and M y are the horizontal and vertical coordinates of point M, respectively. The degree of deformation is strength. The target position of the current point after deformation is U, which is defined as: The above formula is improved by adding the deformation degree control variable strength. The improved facial swelling formula is: the horizontal coordinate of the point U after deformation is The ordinate of the deformed point U Among them, the variable variable coefficient coefficient

[0022] S23. After the above operations, the original facial image is edited into a facial image in a simulated space environment as the input data required for the experiment.

[0023] Furthermore, the step S3 is specifically as follows:

[0024] S31. Perform face alignment, data augmentation, and normalization on the edited facial images to reduce the impact of image quality on recognition results and improve the robustness of the model.

[0025] S32. Use the VGG-16 network to encode and decompose the image by region. Decompose the feature map of the last convolutional layer (Conv42 in the VGG-16 network) into 24 local blocks. The region decomposition operation is performed on the feature map of the convolutional layer rather than on the original image. This is because sharing some convolution operations can reduce the model size and expand the receptive field of subsequent neurons. Based on the 512×28×28 feature map and the 24 points covering the facial information area, 24 local blocks can be obtained, each of size 512×6×6.

[0026] S33. Each local block is encoded by a patch gated unit (PG-Unit) using vector shape features. An attention network is then used to estimate the information content of the local block. Finally, the parameters of the entire network are learned by minimizing the softmax loss, constructing a convolutional neural network (ATT_CNN) model with an attention mechanism.

[0027] S34. Input the facial image obtained in step S31 into the ATT_CNN model to obtain emotion recognition results based on the astronaut's facial expression.

[0028] Furthermore, the step S4 is specifically as follows:

[0029] S41. Using the RAVDESS dataset, we superimposed clean speech signals with noise at different signal-to-noise ratios to generate noisy speech signals, which served as the experimental input data. Specifically, we mixed white noise with the clean speech signals at six signal-to-noise ratios: -5dB, -3dB, 0dB, 3dB, 5dB, and 10dB.

[0030] Furthermore, the step S5 is specifically as follows:

[0031] S51. The noisy speech is input into the speech enhancement network to obtain the logarithmic STFT amplitude of the processed clean speech, which is superimposed with the logarithmic STFT phase of the noisy speech signal and then subjected to the ISTFT transform to obtain the processed clean speech signal;

[0032] S52 pre-emphasizes, frames, and windows the clean speech signal to form a short-time analysis window information; obtains the corresponding spectrum data for each short-time analysis window information by the FFT algorithm; obtains the Mel spectrum data by the Mel filter bank; finally, performs cepstrum analysis on the Mel spectrum data to obtain speech MFCC feature data;

[0033] S53. A convolutional module consisting of three 3×3 convolutional layers is connected to a 1×1 convolutional layer to adjust the number of channels. This is then connected to a max pooling layer to construct a convolutional module. This convolutional module is then connected to a dropout layer, which is then connected to an LSTM network model consisting of three LSTM modules. Finally, the extracted multidimensional features are converted to one dimension using a Flatten layer and then fed into a fully connected layer, resulting in a CNN-LSTM model that fuses a convolutional neural network (CNN) and a LSTM.

[0034] S54. Input the MFCC feature data obtained in step S52 into the CNN_LSTM model to obtain emotion recognition results based on the astronaut's speech.

[0035] Furthermore, the step S6 is specifically as follows:

[0036] S61. Define the recognition framework θ containing four mutually exclusive emotion categories: angry, normal, sad, and happy;

[0037] S62. After single-modal recognition, the recognition results based on facial expression and voice can be obtained respectively, and then the decision-level fusion is performed using a weighted summation method. The specific formula is: Among them, E represents the final emotion recognition result, represents the emotion recognition result based on facial expressions, represents the emotion recognition result based on speech, α and β represent the weights of the two respectively, and α+β=1, and the value of j is 1, 2, 3, 4, representing anger, normal, sadness and happiness respectively.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention fuses facial images and speech signals and applies them to the field of emotion recognition in manned spaceflight, realizing astronaut emotions for long-term space missions. After preprocessing the facial image in a simulated space environment, the image is input into a convolutional neural network ATT_CNN with an attention mechanism to obtain the astronaut facial expression modal emotion recognition result; the speech signal in the simulated space environment is input into a speech enhancement network to obtain a clean speech signal, and its MFCC features are extracted and input into a network CNN_LSTM composed of a fusion of a convolutional neural network CNN and a long short-term memory network LSTM to obtain the astronaut speech modal emotion recognition result; the facial expression emotion recognition result and the speech emotion recognition result are decision-fused to obtain the final dual-modal astronaut emotion recognition result. The advantage of the present invention is that the dual-modal emotion recognition based on astronaut facial images and speech, compared with a single modality, not only solves the problems that single-modal information is easily affected by various noises, it is difficult to fully reflect the emotional state, and the feature quality cannot be guaranteed, but also effectively improves the emotion recognition accuracy and algorithm robustness, and has strong promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Flowchart of the present invention.

[0041] Figure 2 ATT_CNN model diagram of a convolutional neural network with an attention mechanism.

[0042] Figure 3 The CNN_LSTM model diagram is composed of the fusion of convolutional neural network CNN and long short-term memory network LSTM. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the embodiments.

[0044] like Figure 1-3 As shown, a dual-modal astronaut emotion recognition method based on facial expressions and speech in videos includes the following steps:

[0045] S1. Extract video frames from the existing public dataset RAVDESS, use the Viola-Jones face detection algorithm to determine whether they contain faces, and crop the original images containing faces to the facial region.

[0046] S11. Use Haar-like features to describe facial features. The feature template contains two types of rectangles, white and black. Haar-like features are defined as the difference between the sum of the pixels in the white and black rectangles. This reflects the local grayscale changes of the image to a certain extent, quantifies facial features, and distinguishes between faces and non-faces.

[0047] S12. Create an integral image and use it to quickly obtain several different rectangular features. For a point (x, y) with x as the horizontal coordinate and y as the vertical coordinate in the integral image coordinate system, the integral image value of the point is equal to the sum of all pixels located in the upper left corner of the point. The expression is: S(x, y) = ∑ x′≤x ∑ y′≤yf (x′, y′), where S(x, y) is the integral image value of the point (x, y), f(x′, y′) is the pixel value of the point with abscissa x′ and ordinate y′ in the coordinate system, and the integral image satisfies the following relationship: S(x, y) = f(x, y) + S(x-1, y) + S(x, y-1) - S(x-1, y-1), where f(x, y) is the pixel value of the point A(x, y), S(x-1, y) is the integral image value of the point (x-1, y), S(x, y-1) is the integral image value of the point (x, y-1), and S(x-1, y-1) is the integral image value of the point (x-1, y-1);

[0048] S13. Using the AdaBoost algorithm for training, an image usually generates a feature set that is much larger than the image dimension. For example, a 24*24 image will generate 162336 rectangular features. However, not all features are useful. Therefore, an AdaBoost feature classifier is needed to extract useful features. AdaBoost is a strong classifier formed by linearly combining a series of weak classifiers. Assuming that an image generates N features, for the i-th feature f i , i=1,2,3…N, the strong classifier expression is: Among them, a is a detection sub-window, α i is the coefficient, h(a) is a strong classifier, h i (a) is a weak classifier, which is represented by a simple threshold function: Among them, θ i is the threshold;

[0049] S14. Build a cascade classifier. Since the face region occupies only a small portion of a normal image, training all features would be computationally intensive. Therefore, a cascade of multiple strong classifiers is employed. The first classifier uses a small number of features to remove most non-face regions, while the subsequent classifier uses more complex features to remove even more complex non-face regions.

[0050] S15. After face detection obtains the face bounding box, the original image can be cropped to the face area.

[0051] S2. Edit the original facial image into a facial image in a simulated space environment using a bilateral filtering algorithm and a liquefaction algorithm;

[0052] S21. Use the bilateral filtering algorithm to achieve facial skin smoothing of the original image. In the facial contour area, the grayscale value difference between adjacent pixels is large, forming a more obvious boundary, visually forming the facial contour and facial features. Based on this characteristic, the weight of the pixels with a large grayscale difference from the center pixel in the area can be increased to strengthen the impact of the boundary on the image, thereby achieving the preservation of the facial boundary. The original Gaussian filter only uses the distance difference as a variable. On this basis, the pixel value difference can be added. That is, the closer the distance to the center point, the higher the weight, but the larger the difference in pixel value from the center point, the smaller the weight. In addition, the influence of the pixel value difference is greater than the distance value. The formula for bilateral filtering is: in, Where G(p) is the bilateral filter function, p is the center pixel, x and y are the horizontal and vertical coordinates of point p, q is another pixel calculated with the center pixel, m and n are the horizontal and vertical coordinates of point q, F(p) is the pixel value of point p, F(q) is the pixel value of point q, Ω represents the image, σ s is the standard deviation of the spatial domain Gaussian function, σ r is the standard deviation of the Gaussian function in pixel domain, is the spatial domain kernel, is the pixel domain kernel, and e is a natural constant. The essence of bilateral filtering is the superposition of two Gaussian functions, one with distance as the variable and the other with pixel difference as the variable, and the convolution sum is performed, η(p) -1 It is normalized. is a Gaussian function with pixel value as variable, It is a Gaussian function with pixel difference as variable;

[0053] S22. Use the dlib model and liquefaction algorithm to extract 68 facial feature points and realize facial swelling of the original image. The liquefaction algorithm is essentially a pixel migration or image distortion algorithm. It can be regarded as moving a certain pixel to a target position. For example, a swollen face is to stretch the pixels on the periphery of the face outward. That is, the position of the target point is calculated and the current point is replaced with the pixel of the target point. In order to make the changed image smoother, resampling technology can be used for interpolation processing. Assume that the current point is X(x,y) and the center point of the deformation area is designated as C(C x ,C y ), C x and C y are the horizontal and vertical coordinates of point C, the radius of the deformation area is r, and the deformation end point (from the center point to a certain position M) is adjusted to M (M x ,M y ), M x and M yare the horizontal and vertical coordinates of point M, respectively. The degree of deformation is strength. The target position of the current point after deformation is U, which is defined as: The above formula is improved by adding the deformation degree control variable strength. The improved facial swelling formula is: the horizontal coordinate of the point U after deformation is The ordinate of the deformed point U Among them, the variable variable coefficient coefficient

[0054] S23. After the above operations, the original facial image is edited into a facial image in a simulated space environment as the input data required for the experiment.

[0055] S3. Input the preprocessed facial image into the convolutional neural network ATT_CNN with an attention mechanism to obtain the astronaut's facial expression modality emotion recognition results;

[0056] S31. Perform face alignment, data augmentation, and normalization on the edited facial images to reduce the impact of image quality on recognition results and improve the robustness of the model.

[0057] S32. Use the VGG-16 network to encode and decompose the image by region, decomposing the feature map of the last convolutional layer (Conv42 in the VGG-16 network) into 24 local blocks. The region decomposition operation is performed on the feature map of the convolutional layer rather than on the original image. This is because sharing some convolution operations can reduce model size and expand the receptive field of subsequent neurons. Based on the 512×28×28 feature map and the 24 points covering the facial information area, 24 local blocks are obtained, each of size 512×6×6;

[0058] S33. Each local block is encoded by the patch gated unit (PG-Unit) through the vector shape feature, and the attention network is used to estimate the amount of information of the local block. Finally, the parameters in the entire network are learned by minimizing the softmax loss to construct a convolutional neural network (ATT_CNN) model with attention mechanism. Specifically, in each region-specific PG-Unit, the cropped local feature map is fed into two convolutional layers without reducing the spatial resolution in order to retain more information. The final feature map is then processed in two branches. The first branch encodes the input feature map into local features of vector shape; the second branch consists of an attention network to estimate scalar weights to represent the importance of the local block, and then the calculated weights are used to weight the local features. Assume pk The feature map representing the k-th local block input has a size of 512×6×6. Represents the last feature map before the two branches, whose size is 512×6×6, where Represents the operations in the convolutional layer. Indicates that the kth PG-Unit will feature map As input, we learn local specific facial features ψ k , where ψ k is an unweighted eigenvector. α k is a scalar that indicates the degree to which the kth local block is not occluded. Refers to the operations in the attention network, including a pooling operation, a convolution operation, two inner productions and a sigmoid activation function. k =α k ·ψ k , indicating that the kth PG-Unit uses α k For local specific facial features ψ k Weighted and output its weighted feature φ k , where φ k Represents a feature map with weights for each local block;

[0059] S34. Input the facial image obtained in step S31 into the ATT_CNN model to obtain emotion recognition results based on the astronaut's facial expression.

[0060] S4. Superimpose random noise on the audio data from the public dataset RAVDESS to generate speech signals in a simulated space environment.

[0061] S41. Using the RAVDESS dataset, we superimposed clean speech signals with noise at different signal-to-noise ratios to generate noisy speech signals, which served as the experimental input data. Specifically, we mixed white noise with the clean speech signals at six signal-to-noise ratios: -5dB, -3dB, 0dB, 3dB, 5dB, and 10dB.

[0062] S5. The signal is input into a speech enhancement network to obtain a clean speech signal and extract its MFCC features. Subsequently, the signal is input into a CNN-LSTM network, which is a fusion of a convolutional neural network (CNN) and a long short-term memory (LSTM) network, to obtain the astronaut's speech modality emotion recognition results.

[0063] S51. The noisy speech is input into the speech enhancement network to obtain the logarithmic STFT amplitude of the processed clean speech, which is superimposed with the logarithmic STFT phase of the noisy speech signal and then subjected to the ISTFT transform to obtain the processed clean speech signal;

[0064] S52 pre-emphasizes, frames, and windows the clean speech signal to form a short-time analysis window information; obtains the corresponding spectrum data for each short-time analysis window information by the FFT algorithm; obtains the Mel spectrum data by the Mel filter bank; finally, performs cepstrum analysis on the Mel spectrum data to obtain speech MFCC feature data;

[0065] S53. A convolutional module consisting of three 3×3 convolutional layers is connected to a 1×1 convolutional layer to adjust the number of channels. This is then connected to a max pooling layer to construct a convolutional module. This convolutional module is then connected to a dropout layer, which is then connected to an LSTM network model consisting of three LSTM modules. Finally, the extracted multidimensional features are converted to one dimension using a Flatten layer and then fed into a fully connected layer, resulting in a CNN-LSTM model that fuses a convolutional neural network (CNN) and a LSTM.

[0066] S54. Input the MFCC feature data obtained in step S52 into the CNN_LSTM model to obtain emotion recognition results based on the astronaut's speech.

[0067] S6. Decision-making fusion of the facial expression emotion recognition results and the speech emotion recognition results to obtain the final dual-modal astronaut emotion recognition results;

[0068] S61. Define the recognition framework θ containing four mutually exclusive emotion categories: angry, normal, sad, and happy;

[0069] S62. After single-modal recognition, the recognition results based on facial expression and voice can be obtained respectively, and then the decision-level fusion is performed using a weighted summation method. The specific formula is: Among them, E represents the final emotion recognition result, represents the emotion recognition result based on facial expressions, represents the emotion recognition result based on speech, α and β represent the weights of the two respectively, and α+β=1, and the value of j is 1, 2, 3, 4, representing anger, normal, sadness and happiness respectively.

[0070] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Without departing from the spirit and principles of the present invention, any modifications, equivalent substitutions, improvements, etc. made to the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A dual-modal astronaut emotion recognition method based on facial expressions and speech in videos, characterized by: The following steps are involved: S1. Extract video frames from the existing public dataset RAVDESS, use the Viola-Jones face detection algorithm to determine whether they contain faces, and crop the original images containing faces to the facial region. S2. Edit the original facial image into a facial image in a simulated space environment using a bilateral filtering algorithm and a liquefaction algorithm; S3. Input the preprocessed facial image into the convolutional neural network ATT_CNN with an attention mechanism to obtain the astronaut's facial expression modality emotion recognition results; S4. Superimpose random noise on the audio data from the public dataset RAVDESS to generate speech signals in a simulated space environment. S5. The signal is input into a speech enhancement network to obtain a clean speech signal and extract its MFCC features. Subsequently, the signal is input into a CNN-LSTM network, which is a fusion of a convolutional neural network (CNN) and a long short-term memory (LSTM) network, to obtain the astronaut's speech modality emotion recognition results. S6. Make a decision fusion based on the facial expression emotion recognition results and the speech emotion recognition results to obtain the final dual-modal astronaut emotion recognition results.

2. The dual-modal astronaut emotion recognition method based on facial expression and voice in video according to claim 1 is characterized in that: The step S1 is specifically as follows: S11. Use Haar-like features to describe facial features. The feature template contains two types of rectangles, white and black. Haar-like features are defined as the difference between the sum of the pixels in the white and black rectangles. They reflect local changes in image grayscale and quantify facial features to distinguish between faces and non-faces. S12. Create an integral image and use it to quickly obtain several different rectangular features. For a point (x, y) with abscissa x and ordinate y in the integral image coordinate system, the integral image value of that point is equal to the sum of all pixels located in the upper left corner of that point. The expression is: S(x, y) = ∑ x′≤x ∑ y′≤y f(x′, y′), where S(x, y) is the integral image value of the point (x, y), f(x′, y′) is the pixel value of the point with abscissa x′ and ordinate y′ in the coordinate system, and the integral image satisfies the following relationship: S(x, y)=f(x, y)+S(x-1, y)+S(x, y-1)-S(x-1, y-1), where f(x, y) is the pixel value of the point A(x, y), S(x-1, y) is the integral image value of the point (x-1, y), S(x, y-1) is the integral image value of the point (x, y-1), and S(x-1, y-1) is the integral image value of the point (x-1, y-1); S13. Using AdaBoost algorithm for training, usually an image will generate a feature set higher than the image dimension. Assuming that an image will generate N features, for the i-th feature f i , i=1,2,3...N, the strong classifier expression is: Among them, a is a detection sub-window, α i is the coefficient, h(a) is a strong classifier, h i (a) is a weak classifier, h i (a) is expressed as a simple threshold function: Among them, θ i is the threshold; S14. Establish a cascade classifier, using a cascade method of multiple strong classifiers. The first classifier is responsible for removing most non-face areas using a small number of features, and the subsequent classifier uses more complex features to remove more complex non-face areas. S15. After face detection obtains the face bounding box, the original image is cropped to the face area.

3. The dual-modal astronaut emotion recognition method based on facial expression and voice in video according to claim 1 is characterized in that: The step S2 is specifically as follows: S21. Use bilateral filtering algorithm to realize facial skin smoothing of original image. The formula for bilateral filtering is: in, Where G(p) is the bilateral filter function, p is the center pixel, x and y are the horizontal and vertical coordinates of point p, q is another pixel calculated with the center pixel, m and n are the horizontal and vertical coordinates of point q, F(p) is the pixel value of point p, F(q) is the pixel value of point q, Ω represents the image, σ s is the standard deviation of the spatial domain Gaussian function, σ r is the standard deviation of the Gaussian function in pixel domain, is the spatial domain kernel, is the pixel domain kernel, e is a natural constant; The essence of bilateral filtering is the superposition of two Gaussian functions, one with distance as the variable and the other with pixel difference as the variable, and the convolution sum is done, η(p) -1 It is normalized. is a Gaussian function with pixel value as variable, It is a Gaussian function with pixel difference as variable; S22. The dlib model and liquefaction algorithm are used to extract 68 facial feature points and reduce facial swelling in the original image. The liquefaction algorithm is essentially a pixel migration or image distortion algorithm, which can be regarded as moving a pixel to a target position. The swelling of the face is caused by stretching the pixels around the face outward. That is, the position of the target point is calculated and the current point is replaced with the pixel of the target point. To make the transformed image smoother, resampling technology is used for interpolation. Assume that the current point is X(x, y) and the center point of the deformed area is designated as C(C x , C y ), C x and C y are the horizontal and vertical coordinates of point C, the radius of the deformation area is r, and the deformation end point is adjusted to M (M x , M y ), M x and M y are the horizontal and vertical coordinates of point M, respectively. The degree of deformation is strength. The target position of the current point after deformation is U, which is defined as: The above formula is improved by adding the deformation degree control variable strength. The improved facial swelling formula is: the horizontal coordinate of the point U after deformation is The ordinate of the deformed point U Among them, the variable variable coefficient coefficient S23. Edit the original facial image into a facial image in a simulated space environment as input data required for the experiment.

4. The dual-modal astronaut emotion recognition method based on facial expression and speech in video according to claim 1 is characterized in that: The step S3 is specifically as follows: S31. Perform face alignment, data augmentation, and normalization on the edited facial images to reduce the impact of image quality on recognition results and improve the robustness of the model. S32. Use the VGG-16 network to encode and decompose the image. Decompose the feature map of the last convolutional layer into 24 local blocks. The regional decomposition operation is performed on the feature map of the convolutional layer, rather than on the original image. This is because sharing some convolution operations can reduce model size and expand the receptive field of subsequent neurons. Based on the 512×28×28 feature map and 24 points covering the facial information area, 24 local blocks are obtained, each with a size of 512×6×6. S33. Each local block is encoded by a patch gating unit (PG-Unit) using vector shape features. An attention network is then used to estimate the information content of the local block. Finally, the parameters of the entire network are learned by minimizing the softmax loss, constructing a convolutional neural network (ATT_CNN) model with an attention mechanism. S34. Input the facial image obtained in step S31 into the ATT_CNN model to obtain emotion recognition results based on the astronaut's facial expression.

5. The dual-modal astronaut emotion recognition method based on facial expression and voice in video according to claim 1 is characterized in that: The step S4 is specifically as follows: S41. Using the RAVDESS dataset, clean speech signals are superimposed with noise at different signal-to-noise ratios to obtain noisy speech signals as the input data required for the experiment.

6. The dual-modal astronaut emotion recognition method based on facial expression and speech in video according to claim 1 is characterized in that: The step S5 is specifically as follows: S51. The noisy speech is input into the speech enhancement network to obtain the logarithmic STFT amplitude of the processed clean speech, which is superimposed with the logarithmic STFT phase of the noisy speech signal and then subjected to the ISTFT transform to obtain the processed clean speech signal; S52 pre-emphasizes, frames, and windows the clean speech signal to form a short-time analysis window information; obtains the corresponding spectrum data for each short-time analysis window information by the FFT algorithm; obtains the Mel spectrum data by the Mel filter bank; finally, performs cepstrum analysis on the Mel spectrum data to obtain speech MFCC feature data; S53. A convolutional module consisting of three 3×3 convolutional layers is connected to a 1×1 convolutional layer to adjust the number of channels. This is then connected to a max pooling layer to construct a convolutional module. This convolutional module is then connected to a dropout layer, which is then connected to an LSTM network model consisting of three LSTM modules. Finally, the extracted multidimensional features are converted to one dimension using a Flatten layer and then fed into a fully connected layer, resulting in a CNN-LSTM model that fuses a convolutional neural network (CNN) and a LSTM. S54. Input the MFCC feature data obtained in step S52 into the CNN_LSTM model to obtain emotion recognition results based on the astronaut's speech.

7. The dual-modal astronaut emotion recognition method based on facial expression and speech in video according to claim 1 is characterized in that: The step S6 is specifically as follows: S61. Define the recognition framework θ containing four mutually exclusive emotion categories: angry, normal, sad, and happy; S62. After single-modal recognition, the recognition results based on facial expression and voice are obtained respectively, and then weighted summation is used for decision-level fusion. The specific formula is: Among them, E represents the final emotion recognition result, represents the emotion recognition result based on facial expressions, represents the emotion recognition result based on speech, α and β represent the weights of the two respectively, and α+β=1, and the value of j is 1, 2, 3, 4, representing anger, normal, sadness and happiness respectively.

Images