Postoperative emotional state recognition method and device fusing face and voice features

Abstract

The invention discloses a postoperative emotional state recognition method and device fusing facial and voice features, and the method is implemented through the following steps: firstly, collecting the audio data and facial expression data of 132 patients at the age of 65 years or above in a data collection stage, so as to build a diversified data set; then, in a data preprocessing stage, the audio data is subjected to wavelet transformation and image processing, so that the audio data is suitable for training of a deep learning model. Next, in network model design, VGG16 is adopted as a basic model, transfer learning is carried out, and meanwhile, an RNN architecture is introduced to further improve the recognition accuracy; the hardware acceleration training is beneficial to improving the training efficiency. And finally, in a desktop application development stage, a friendly interface is provided for the user, and the user is allowed to upload audio data and perform emotional state recognition. According to the method, audio and facial expression data are fused, and the emotional state of the user is recognized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a postoperative emotional state recognition method and device integrating facial and voice features. Background Art

[0002] With the continuous development of medicine and technology, the understanding and attention to the emotional state of postoperative patients is increasing. The assessment of postoperative emotional state is of great significance to medical institutions and medical staff. On the one hand, the emotional state of postoperative patients can reflect their rehabilitation process and physical and mental health, thereby helping medical staff to promptly identify possible problems and provide timely support. On the other hand, timely and accurate identification of the emotional state of postoperative patients, especially whether they are excited or depressed, can help optimize the rehabilitation plan, improve the treatment effect, and ultimately improve the quality of life of patients.

[0003] However, the assessment of postoperative emotional state usually takes a lot of time and manpower, and may be affected by subjective factors. In this case, the development of an automated emotion analysis system becomes an urgent need. The audio emotion analysis system based on machine learning can identify the emotional state (normal, excited, depressed) of people by analyzing the audio data read aloud by people after surgery. This system can greatly improve the efficiency and accuracy of emotional state assessment, allowing medical staff to focus more on individualized rehabilitation plans and treatment processes, and provide more precise care and support for postoperative patients.

[0004] There are some obvious shortcomings and limitations in the existing technology:

[0005] 1. Subjectivity and inaccuracy: Traditional emotion recognition methods mainly rely on subjective observation and self-reporting by patients, which may lead to subjectivity and inaccuracy in the assessment of emotional states. Patients may choose to conceal or misreport their emotional states for various reasons, and the subjective judgment of medical staff may be affected by personal bias and misunderstanding.

[0006] 2. Time-consuming and labor-intensive: Current methods of assessing emotional states usually require medical staff to observe and communicate with patients for a long time, which consumes a lot of time and human resources. This may lead to inefficiency in medical institutions and limit the attention of medical staff to other important tasks.

[0007] 3. Reliance on text and voice data: Existing technologies mainly focus on collecting text and voice audio data read by people after surgery and marking their emotional states. However, this method may not fully capture the diversity and complexity of patients' emotional states, because emotions are not only expressed through voice, but also through facial expressions, body language, etc.

[0008] 4. Lack of real-time monitoring and intervention: Traditional methods are often periodic assessments, but cannot provide continuous, real-time emotional state monitoring and intervention. This may lead to missed opportunities to identify and deal with patients' emotional problems in a timely manner, thus affecting rehabilitation effects and the improvement of quality of life.

[0009] 5. Dependence on technology: Existing technologies are mainly based on machine learning and audio processing technology, which may make some medical staff feel strange and unfamiliar. They may need additional training and support to be able to make full use of these technologies for emotional state assessment and processing.

[0010] In summary, existing technologies have many challenges and shortcomings in the field of postoperative emotional state assessment, and more innovative and effective methods are needed to solve these problems in order to achieve more accurate, real-time and objective emotional state identification and intervention. Summary of the invention

[0011] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and device for identifying postoperative emotional state by integrating facial and voice features, so as to solve the problems of subjectivity and low accuracy in the prior art.

[0012] A postoperative emotional state recognition method integrating facial and voice features comprises the following steps:

[0013] S1. Data collection: collect audio data and facial expression data of several patients;

[0014] S1.1. Identify a number of postoperative patients, all aged 65 years or above, with a male to female ratio of 1:1;

[0015] S1.2. Each patient was asked to read a standardized text in a specific emotional state; these emotional states covered a variety of possible emotions, including normal, excited, and depressed emotions;

[0016] S1.3. Use professional audio recording equipment to record the patient's reading to ensure the quality and accuracy of the audio data;

[0017] S1.4. Each patient's reading audio is saved as a .wav file and categorized and stored according to emotional state; subfolders are created corresponding to normal, excited, and depressed emotions;

[0018] S1.5. Count the audio files in each subfolder to determine how many samples are obtained in total and the number of samples in each emotional state;

[0019] S1.6. Facial data collection uses high-resolution cameras to record the facial expressions of each patient to ensure the quality and clarity of the data;

[0020] S1.7. The facial expression data of each patient was saved as image files in .jpg or .png format and categorized and stored according to emotional state. Subfolders were also created to represent normal, excited, and depressed emotions respectively.

[0021] S1.8. Determine how many facial expression samples are obtained in total and the number of samples in each emotional state by counting the image files in each subfolder and filtering the valid data;

[0022] S2. Data preprocessing: The audio data obtained in the data collection stage needs to go through a preprocessing step to facilitate subsequent feature extraction and model training. The audio data preprocessing part converts the patient's audio data into a format that can be used for neural network training. First, a wavelet transform is performed to generate a wavelet coefficient matrix. Then, these coefficient matrices are converted into RGB image format. The size of the RGB image is set to 224*224*3 so that it can be compatible with the preloaded model. The specific implementation steps are as follows:

[0023] S2.1. First, the audio data in each emotional state is loaded into the memory and prepared for processing;

[0024] S2.2. For each audio file, use the `audioread` function of the audio processing library MATLAB to read the .wav format file into an audio signal. These audio signals are time series data with a sampling rate of Fs.

[0025] S2.3. Since the lengths of different audio files may be different, the lengths of all audio signals are unified into a fixed length;

[0026] S2.4, classify the processed audio signal according to the emotional state and construct the corresponding data matrix; each row represents a sample, and each column represents the time series data of the audio signal;

[0027] S2.5, associate the emotion state labels with the data matrix to facilitate supervised learning during the model training phase;

[0028] S2.6. Finally, save the preprocessed data as a .mat file to facilitate subsequent feature extraction and model training;

[0029] S2.7, video data preprocessing; In the video data preprocessing part, the collected patient facial expression data is processed as the input of the neural network; first, face detection and key point location are performed to ensure the accurate location of the facial expression area; then, the facial image is cropped and resized to match the input specifications of the neural network; finally, these processed facial images are converted into 224*224 pixel RGB images;

[0030] S3. Network model design: After data preprocessing, a neural network model is designed to effectively extract the features of emotional state from audio data. A fusion CNN and RNN method is used to diagnose emotional state. Complex features are extracted from samples of size 224*224*3 using VGG16. The extracted features are fed into the RNN classifier to distinguish normal, excited, and depressed emotional states. The VGG16 model is used as a basis for transfer learning to adapt it to the task of emotional state recognition. The specific implementation steps are as follows:

[0031] S3.1. Extract key features from audio images or facial images using different VGG16 models;

[0032] For original VGG16:

[0033] X image_features =VGG16 base (I image )

[0034] X audio_features =VGG16 base (I audio )

[0035] Among them, I image is the input facial image, I audio is the input audio image;

[0036] S3.2, load the pre-trained VGG16 model as the basic model, and perform emotional state recognition based on the convolutional neural network structure in image recognition;

[0037] S3.3, first, retain the bottom convolutional layers of the VGG16 model, which can extract the basic features of the audio data and extract the facial expression information;

[0038] S3.4. Next, we need to replace the last fully connected layer of the VGG16 model to adapt to the emotional state recognition task. We replace the fully connected layer with a new fully connected layer with output nodes that match the number of emotional categories. This new fully connected layer can map the features extracted by the underlying convolutional layer to the probability distribution of different emotional states.

[0039] Replace the last fully connected layer of the VGG16 model with a new fully connected layer:

[0040] X fc =W fc ·X image_features +b fc

[0041] S3.5, add activation function and Dropout layer;

[0042] S3.6. Use the feature map as the input of the multi-layer RNN;

[0043] Use the convolutional feature map as the input of the multi-layer RNN:

[0044] X rnn_input =Convolution(X audio_features )

[0045] S3.7, apply Softmax classifier to classify the image;

[0046] Apply the Softmax classifier, where C represents the number of sentiment categories:

[0047] P emotion =Softmax(W softmax ·X rnn_input +b softmax )

[0048] Among them, P emotion is the probability distribution of the emotional state;

[0049] S3.8. After replacing the fully connected layer, you need to adjust the parameters related to the learning rate;

[0050] S4, hardware accelerated training;

[0051] S4.1. When setting model training options, use hardware acceleration to speed up the training process. Set `ExecutionEnvironment` to 'gpu' in the training options to perform calculations on the GPU.

[0052] S4.2. Set the training options, `MiniBatchSize` the number of samples per iteration, `MaxEpochs` the maximum number of iterations. These parameters will affect the training effect and speed. Use NVIDIA GeForce GTX 1650 4G video memory, and set MiniBatchSize to 16;

[0053] S4.3. Start model training. During the training process, the deep learning toolbox will automatically perform forward propagation and back propagation to update the weights of the model so that it can gradually adapt to the task of emotion state recognition.

[0054] S5. Design of emotional state fusion model: Design a model to fuse the feature extraction of audio data and facial expression data with the emotional state recognition model results to obtain the final emotional state recognition result; specifically implement it according to the following steps:

[0055] S5.1. Feature fusion: First, obtain the probability distribution results of the emotional state from the emotional state recognition model of the audio data, and the probability distribution results of the emotional state from the emotional state recognition model of the facial expression data; these probability distributions can be obtained through the Softmax classifier, and the probability of each category represents the possibility of the corresponding emotional state;

[0056] S5.2. Weight allocation: In order to merge the results of the two data sources, a weight allocation mechanism is introduced;

[0057] Introduce weight distribution as W = [w1, w2, w3], these weights can be determined through training or domain knowledge;

[0058] S5.3, Fusion method: Use a fusion method to fuse the emotion state probability distributions of the two data sources; adopt a weighted average, in which the probability distribution of each data source is multiplied by its weight and then added, which will produce a fused emotion state probability distribution;

[0059] Emotional state fusion can be expressed as the following formula:

[0060] Merged_Prediction=W[0]*A+W[1]*F

[0061] Merged_Prediction is the probability distribution of the fused emotional state, and W[0] and W[1] are the weights of the audio data and facial expression data;

[0062] S5.4, Final decision: According to the fused probability distribution of emotional states, the emotional state with the highest probability is selected as the final emotional state recognition result; this is achieved by finding the emotional state category corresponding to the maximum value in the probability distribution;

[0063] The final decision can be expressed as:

[0064] Final_Emotion=argmax(Merged_Prediction)

[0065] Here, Final_Emotion is the final emotional state, which is the emotional state category with the highest probability in Merged_Prediction;

[0066] S5.5, Threshold setting: To further improve the credibility of the results, a threshold is set to determine whether the result should be marked as uncertain when the highest probability is not obvious; if the highest probability is lower than the threshold, the result is marked as uncertain, otherwise the emotional state corresponding to the highest probability is selected;

[0067] S6. Desktop application development: After completing the training and optimization of the model, it is applied to the actual emotion state recognition scenario to provide users with a friendly desktop application interface. The specific implementation includes the following steps:

[0068] S6.1. Choose to use the App Designer tool in MATLAB.

[0069] S6.2. First, create a new App Designer project and add corresponding UI components on the interface, including buttons, text boxes, and charts, so that users can interact with the system;

[0070] S6.3. Add an audio input function to the interface so that users can record or upload audio files for emotion state recognition;

[0071] S6.4, by calling the trained model, inputting the audio data into the model, and identifying the emotional state; then, displaying the recognition result on the interface, displaying the recognized emotional state label;

[0072] S6.5, add functions to display audio waveform in real time, visualize results, and export recognition results;

[0073] S6.6. Finally, the application is tested to ensure that it works properly under different circumstances.

[0074] Preferably, in step S2.3, for an audio signal that is insufficient in length, zeros are added to the end, and for an audio signal that is too long, it is truncated.

[0075] Preferably, in step S3.3, the facial expression information includes facial edge and texture information.

[0076] Preferably, in step S3.5, an activation function and a Dropout layer are added. After step S3.3 of the neural network model, the following steps may be continued to enhance the performance and robustness of the model;

[0077] S3.5.1. Add activation function. Introduce activation function between fully connected layer 1 and fully connected layer 2. Use Rectified Linear Unit, i.e. ReLU activation function.

[0078] S3.5.2. Add a Dropout layer. A Dropout layer can be added between fully connected layer 1 and fully connected layer 2. Set the dropout rate to 0.5 to determine the probability of shutting down neurons.

[0079] Add the ReLU activation function:

[0080] X fc_activated=ReLU(X fc )

[0081] Add Dropout layer:

[0082] X fc_dropped = Dropout(X fc_activated ).

[0083] A postoperative emotional state recognition device integrating facial and voice features includes a system of the above-mentioned postoperative emotional state recognition method integrating facial and voice features, and also includes a computer or embedded device, a screen or display, an audio input device, a camera device and a speaker.

[0084] The beneficial effects of the present invention are:

[0085] (1) Diverse datasets: By collecting audio and facial expression data from different patients, we ensure that the system can effectively handle recognition tasks of various emotional states.

[0086] (2) Transfer learning and improved model architecture: Transfer learning is used to effectively improve the performance of the model. The introduction of the RNN architecture helps to process sequential data and improves the recognition accuracy of emotional states.

[0087] (3) User-friendly desktop application interface: It provides users with a convenient interface that enables them to easily identify and analyze emotional states and transform research results into practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0089] Figure 1 It is an overall framework diagram of a postoperative emotional state recognition method and device integrating facial and voice features of the present invention;

[0090] Figure 2 It is a network structure diagram of a postoperative emotional state recognition method and device integrating facial and voice features of the present invention;

[0091] Figure 3 This is a device diagram of a method and device for identifying postoperative emotional states by integrating facial and voice features according to the present invention.

[0092] Figure 4It is the original VGG16 model training diagram of a postoperative emotional state recognition method and device integrating facial and voice features of the present invention;

[0093] Figure 5 It is a training diagram of a VGG16 model based on transfer learning of a postoperative emotional state recognition method and device integrating facial and voice features of the present invention;

[0094] Figure 6 This is an improved VGG16+RNN model training diagram of a postoperative emotional state recognition method and device that integrates facial and voice features in the present invention. DETAILED DESCRIPTION

[0095] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0096] A postoperative emotional state recognition method integrating facial and voice features comprises the following steps:

[0097] S1. Data collection. In the data collection part, audio data and facial expression data of 132 patients aged 65 and above are collected. These audio data and facial expression data are the key inputs of the emotional state recognition system. By collecting audio data and facial expression data samples from different patients, a diversified data set covering different emotional states and sound characteristics can be obtained to improve the accuracy and effect of emotional state recognition. The specific implementation is as follows:

[0098] S1.1. We obtained the cooperation of 132 postoperative patients from medical institutions. All of these patients were over 65 years old, with a male to female ratio of 1:1, to ensure that the data obtained were representative.

[0099] S1.2 Each patient was asked to read a standardized text in a specific emotional state, such as normal, excited, and depressed mood. These emotional states were chosen to cover a variety of possible emotional states and make the dataset richer.

[0100] S1.3. Use professional audio recording equipment (Huawei mobile phone with third-party sound recorder software installed) to record the patient's reading to ensure the quality and accuracy of the audio data. The recording process is carried out in a quiet environment to avoid external interference from affecting the data.

[0101] Step 1.4: Each patient's reading audio is saved as a .wav file and categorized and stored according to emotional state. Three subfolders are created, corresponding to normal, excited, and depressed emotions.

[0102] S1.5. Count the audio files in each subfolder to determine how many samples are obtained in total and the number of samples in each emotional state.

[0103] S1.6 Facial data collection: High-resolution video equipment was used to record the facial expressions of each patient to ensure the quality and clarity of the data. The recording process was carried out under sufficient natural light to capture the details of facial expressions to the greatest extent possible.

[0104] S1.7. The facial expression data of each patient was saved as an image file (.jpg or .png) and classified and stored according to the emotional state. Three subfolders were also established to represent normal, excited, and depressed emotions, respectively.

[0105] S1.8. Determine how many facial expression samples are obtained in total and the number of samples in each emotional state by counting the image files in each subfolder and filtering the valid data.

[0106] Through the above steps, audio data and facial expression data of 132 postoperative patients aged 65 and above were collected, which will be used to train and evaluate the emotional state recognition system. The diversity and representativeness of this dataset will help ensure that the system can effectively cope with various emotional state recognition tasks.

[0107] S2, data preprocessing: The audio data obtained in the data acquisition phase needs to go through a preprocessing step to facilitate subsequent feature extraction and model training. The audio data preprocessing part converts the audio data of 132 patients into a format that can be used for neural network training. First, a wavelet transform is performed to generate a wavelet coefficient matrix. Then, these coefficient matrices are converted into RGB image format, and the size of the RGB image is set to 224*224*3 so that it can be compatible with the preloaded model. Specifically, follow the following steps:

[0108] S2.1. First, load the audio data in each emotional state into the memory and prepare for processing.

[0109] S2.2. For each audio file, use the audio processing library (MATLAB's `audioread` function) to read the .wav format file into an audio signal. These audio signals are usually time series data with a sampling rate of Fs.

[0110] S2.3. Since the lengths of different audio files may be different, the lengths of all audio signals are unified into a fixed length (eg, newLength). For audio signals that are insufficient in length, zeros are added to the end, and for audio signals that are too long, they are truncated.

[0111] S2.4. Classify the processed audio signal according to the emotional state and construct the corresponding data matrix. Each row represents a sample and each column represents the time series data of the audio signal.

[0112] S2.5. Associate the emotion state labels with the data matrix to facilitate supervised learning during the model training phase.

[0113] S2.6. Finally, save the preprocessed data as a .mat file to facilitate subsequent feature extraction and model training.

[0114] S2.7, Video data preprocessing: In the video data preprocessing section, the facial expression data of 132 patients collected were subjected to a series of operations to prepare them as input for the neural network. First, face detection and key point location were performed to ensure accurate location of the facial expression area. Subsequently, the facial images were cropped and resized to match the input specifications of the neural network. Finally, these processed facial images were converted into 224*224 pixel RGB images.

[0115] Through these preprocessing steps, the original audio data can be converted into a data matrix in a unified format, laying the foundation for subsequent feature extraction and model training. The goal of data preprocessing is to ensure the consistency and availability of the data and to provide high-quality input data for subsequent steps.

[0116] S3. Network model design: After data preprocessing is completed, a suitable neural network model needs to be designed to effectively extract the features of emotional state from audio data. The present invention proposes a fusion CNN and RNN method for diagnosing emotional state by using VGG16 to extract complex features from samples of size 224*224*3. The extracted features are fed into the RNN classifier to distinguish between normal, excited and depressed emotional states. The CNN-RNN network for emotional state diagnosis is shown in the attached figure. Figure 2 As shown in the figure, the VGG16 model is used as the basis and the transfer learning method is performed on it to adapt to the emotional state recognition task. The specific implementation steps are as follows:

[0117] S3.1. Extract key features from audio images or facial images using different VGG16 models.

[0118] For original VGG16:

[0119] Ximage_features =VGG16 base (I image )

[0120] X audio_features =VGG16 base (I audio )

[0121] Among them, I image is the input facial image, I audio is the input audio image.

[0122] S3.2. Load the pre-trained VGG16 model as the basic model. This model has achieved good results in the field of image recognition. Since audio data also has similar time series characteristics, emotional state recognition can be performed based on the convolutional neural network structure in image recognition.

[0123] S3.3. First, retain the bottom convolutional layers of the VGG16 model, which can extract the basic features of the audio data and extract information such as facial edges and textures when extracting facial expressions.

[0124] S3.4. Next, we need to replace the last fully connected layer of the VGG16 model to adapt to the emotional state recognition task. Replace the fully connected layer with a new fully connected layer with output nodes that match the number of emotional categories. This new fully connected layer can map the features extracted by the underlying convolutional layer to the probability distribution of different emotional states.

[0125] Replace the last fully connected layer of the VGG16 model with a new fully connected layer:

[0126] X fc =W fc ·X image_features +b fc

[0127] S3.5. Add activation function and Dropout layer. After step 3.3 of the neural network model, you can continue with the following steps to enhance the performance and robustness of the model:

[0128] S3.5.1. Add activation function. Introduce activation function between fully connected layer 1 and fully connected layer 2. Use Rectified Linear Unit (ReLU) activation function. This will help introduce nonlinear features and enable the model to better learn the complexity of the data.

[0129] S3.5.2. Add a Dropout layer. A Dropout layer can be added between fully connected layer 1 and fully connected layer 2. Dropout is a regularization technique that randomly turns off a portion of neurons during training, which helps prevent overfitting and improves the generalization ability of the model. Set the dropout rate to 0.5 to determine the probability of turning off a neuron.

[0130] Add the ReLU activation function:

[0131] X fc_activated =ReLU(X fc )

[0132] Add Dropout layer:

[0133] X fc_dropped = Dropout(X fc_activated )

[0134] S3.6. Use the feature map as the input of the multi-layer RNN.

[0135] Use the convolutional feature map as the input of the multi-layer RNN:

[0136] X rnn_input =Convolution(X audio_features )

[0137] S3.7. Apply Softmax classifier to classify the image.

[0138] Apply the Softmax classifier, where C represents the number of sentiment categories:

[0139] P emotion =Softmax(W softmax ·X rnn_input +b softmax )

[0140] Among them, P emotion is the probability distribution of the emotional state.

[0141] Step 3.8: After replacing the fully connected layer, it is necessary to adjust the parameters related to the learning rate. The present invention uses a dynamic learning rate to adapt to new tasks more quickly.

[0142] Through this network model design, we can make full use of the successful experience of the pre-loaded model in the image field and apply it to the task of emotional state recognition of audio data. Such a design can extract important features from audio data and provide strong support for subsequent training and classification.

[0143] The main difference between the improved VGG16 and the original VGG16 is that the fully connected layer is replaced, and the activation function and Dropout layer are added to adapt to the task of emotional state recognition. In addition, the RNN layer is introduced to process time series data. These improvements enable the network to better adapt to the task of emotional state classification, while the original VGG16 is mainly used for image classification.

[0144] S4. Hardware accelerated training: After designing a network model suitable for emotional state recognition, the model needs to be trained so that it can accurately recognize different emotional states. In order to improve the efficiency of training, hardware acceleration technology can be used. Specifically, follow the steps below:

[0145] S4.1. When setting model training options, use hardware acceleration to speed up the training process. Set `ExecutionEnvironment` to 'gpu' in the training options to perform calculations on the GPU. This can significantly increase the training speed.

[0146] S4.2, set the training options, `MiniBatchSize` (number of samples per iteration), `MaxEpochs` (maximum number of iterations). These parameters will affect the effect and speed of training. The present invention uses NVIDIA GeForce GTX 1650 4G video memory, so MiniBatchSize is set to 16.

[0147] S4.3. Start model training. During the training process, the deep learning toolbox will automatically perform forward propagation and backpropagation to update the weights of the model so that it can gradually adapt to the emotional state recognition task.

[0148] S5. Design of emotional state fusion model: In the previous steps, the feature extraction and emotional state recognition models of audio data and facial expression data have been prepared. Now, a model needs to be designed to fuse the results of these two models to obtain the final emotional state recognition result. The specific implementation steps are as follows:

[0149] S5.1, Feature Fusion: First, obtain the probability distribution results of the emotional state from the emotional state recognition model of the audio data, and the probability distribution results of the emotional state from the emotional state recognition model of the facial expression data. These probability distributions can be obtained through the Softmax classifier, and the probability of each category represents the possibility of the corresponding emotional state.

[0150] Assume that the probability distribution of the emotional state of the audio data is A=[a1,a2,a3], where a1, a2, and a3 represent the probabilities of normal, excited, and depressed emotional states, respectively.

[0151] Assume that the probability distribution of the emotional state of facial expression data is F = [f1, f2, f3], where f1, f2, and f3 represent the probabilities of normal, excited, and depressed emotional states, respectively.

[0152] S5.2. Weight assignment: In order to fuse the results of the two data sources, a weight assignment mechanism is introduced. These weights can be determined through training or domain knowledge. For example, the attention mechanism in deep learning can be used to let the model automatically learn how to assign weights. Another method is to manually assign weights based on the relative importance of audio data and facial expression data.

[0153] The weight distribution is introduced as W = [w1, w2, w3], and these weights can be determined through training or domain knowledge.

[0154] S5.3 Fusion methods: Once the weights are available, a fusion method can be used to fuse the emotion state probability distributions of the two data sources. A common method is weighted averaging, where the probability distribution of each data source is multiplied by its weight and then added. This will produce a fused emotion state probability distribution.

[0155] Emotional state fusion can be expressed as the following formula:

[0156] Merged_Prediction=W[0]*A+W[1]*F

[0157] Merged_Prediction is the probability distribution of the fused emotional state, and W[0] and W[1] are the weights of the audio data and facial expression data.

[0158] S5.4, Final decision: According to the fused emotional state probability distribution, the emotional state with the highest probability can be selected as the final emotional state recognition result. This is achieved by finding the emotional state category corresponding to the maximum value in the probability distribution.

[0159] The final decision can be expressed as:

[0160] Final_Emotion=argmax(Merged_Prediction)

[0161] Here, Final_Emotion is the final emotional state, which is the emotional state category with the highest probability in Merged_Prediction.

[0162] S5.5, Threshold setting: To further improve the credibility of the results, a threshold is set to determine whether the result should be marked as uncertain when the highest probability is not obvious. If the highest probability is lower than the threshold, mark the result as uncertain, otherwise select the emotional state corresponding to the highest probability.

[0163] Through the above steps, an emotion state fusion model is designed to merge the results of audio data and facial expression data to obtain the final emotion state recognition results. This model can be optimized based on experiments and performance evaluation to ensure accuracy and robustness.

[0164] S6. Desktop application development: After completing the training and optimization of the model, it needs to be applied to the actual emotional state recognition scenario to provide users with a friendly desktop application interface. The specific implementation steps are as follows:

[0165] S6.1. Choose to use the App Designer tool in MATLAB, which is a powerful visual application development tool that can help quickly build user interfaces.

[0166] S6.2. First, you need to create a new App Designer project and add corresponding UI components such as buttons, text boxes, charts, etc. to the interface so that users can interact with the system.

[0167] S6.3. Add an audio input function to the interface so that users can record or upload audio files for emotional state recognition.

[0168] S6.4, by calling the trained model, inputting the audio data into the model, and performing the recognition of the emotional state. Then, the recognition result is displayed on the interface, and the recognized emotional state label is displayed.

[0169] S6.5, add some additional functions, such as real-time display of audio waveform, result visualization, export of recognition results, etc., to enhance the user experience.

[0170] S6.6. During application development, the aesthetics, user-friendliness and ease of use of the interface need to be considered to ensure that users can use the system conveniently.

[0171] S6.7. Finally, the application is tested to ensure that it can operate normally under different conditions, and necessary repairs and optimizations are made.

[0172] Through desktop application development, the trained model can be applied to actual emotional state recognition scenarios, providing users with an intuitive and convenient interface that enables them to easily identify and analyze emotional states.

[0173] A postoperative emotional state recognition device integrating facial and voice features includes a system of the above-mentioned postoperative emotional state recognition method integrating facial and voice features, and also includes a computer or embedded device, a screen or display, an audio input device, a camera device and a speaker.

[0174] In a postoperative emotional state recognition method and device that integrates facial and voice features of the present invention: the role of step S1 is to collect the patient's audio data and facial expression data during the data collection stage, providing key input data for the subsequent emotional state recognition system.

[0175] Data collection is carried out using professional audio recording equipment and high-resolution video equipment. The principle is: by collecting diverse audio and facial expression data, different emotional states and sound characteristics can be covered, thereby improving the accuracy and effectiveness of emotional state recognition.

[0176] Its advantages are: the diversity and representativeness of the data help the system cope with the recognition tasks of various emotional states and improve the performance of the system.

[0177] The function of step S2 is to convert the audio data into a format that can be used by the model in the data preprocessing stage to facilitate subsequent feature extraction and model training.

[0178] The wavelet transform and data dimension conversion method is used. The principle is that the wavelet transform is used to generate the wavelet coefficient matrix, which is then converted into the RGB image format. This processing allows the audio data to be processed in the same way as the image data to be compatible with the pre-loaded model.

[0179] The advantages are: data preprocessing ensures the consistency and availability of data, provides high-quality input data, and makes it consistent with the processing method of image data to be compatible with the preloaded model.

[0180] The function of step S3 is to design a neural network model for extracting features of emotional states from audio data and facial expression data.

[0181] VGG16 is used as the basic model for transfer learning. The principle is that the model processes audio data and image data based on convolutional neural network (CNN) and recurrent neural network (RNN) to extract key features. Transfer learning uses the successful experience of pre-trained models in the image field and applies them to the task of emotional state recognition of audio data.

[0182] The purpose of step S3.8 is to improve the convergence speed of the model and use a dynamic learning rate to improve the efficiency and performance of training the neural network.

[0183] The principle of using dynamic learning rate is based on the following considerations: 1) Convergence speed: During the training process, the neural network may go through different stages, from the initial random weights to the weights close to convergence. At these different stages, the appropriate learning rate may be different. A larger learning rate helps converge faster, but may skip the solution before approaching the optimal solution. A smaller learning rate helps more fine-tuned weight adjustment, but may cause training to be too slow. The dynamic learning rate can automatically adjust the learning rate at different stages of training to balance these factors. 2) Avoid falling into the local minimum: When using a fixed learning rate, it may fall into the local minimum during training and it is difficult to escape. The dynamic learning rate can help the model jump out of the local minimum in these cases because it allows the learning rate to increase near the local minimum, giving the model a chance to continue exploring a larger weight space. 3) Robustness: The dynamic learning rate can make the model more robust and better adaptable to different types of data or problems. It can dynamically adjust the learning rate according to the size and direction of the gradient to adapt to different situations.

[0184] The advantages of using a dynamic learning rate are: 1) Faster convergence: Using a larger learning rate in the early stages of training can accelerate convergence, while reducing the learning rate when approaching the optimal solution can refine the weight adjustment. 2) Avoid gradient explosion or gradient disappearance: Dynamic learning rates can help prevent gradient values ​​from becoming too large or too small, leading to an unstable training process. 3) Better generalization performance: By adaptively adjusting the learning rate, dynamic learning rates can usually produce better model generalization performance and avoid overfitting.

[0185] The advantage is that using transfer learning can effectively utilize the knowledge of pre-loaded models, extract important features of audio and facial expression data, and improve model performance.

[0186] The function of step S4 is to perform training on the designed model to improve the training efficiency.

[0187] The principle of using hardware accelerated GPU is that by using GPU for calculation, the training speed can be significantly improved and the convergence of the model can be accelerated.

[0188] The advantage is that hardware accelerated training can speed up the training of the model, thereby obtaining the emotion state recognition model more quickly.

[0189] The function of step S5 is to fuse the results of the audio data and the facial expression data to obtain the final emotion state recognition result.

[0190] The weight distribution mechanism is introduced and the weighted average fusion method is used. The principle is: through weight distribution, the contribution of each data source to the final result can be determined, and then they are fused together by weighted average.

[0191] Its advantage is that fusing the results of multiple data sources can improve the accuracy of emotion state recognition and make the model more robust.

[0192] The purpose of step S6 is to apply the trained and optimized emotion state recognition model to the actual emotion state recognition scenario, and provide users with a friendly desktop application interface so that they can interact with the system, record or upload audio data, and obtain emotion state recognition results.

[0193] The App Designer tool in MATLAB is used for application development. The principle is as follows: App Designer is a visual application development tool that can be used to build user interfaces. It allows developers to add UI components by dragging and dropping, and then write the interaction logic with these components. In application development, the user's audio data can be passed to the trained emotional state recognition model. The model will analyze the audio data and return the results of the emotional state, and then display the results on the application interface.

[0194] Its advantages are: 1) It provides a user-friendly interface, allowing users to easily interact with the system, record audio data, and view recognition results. 2) It increases the usability of the system, enabling non-technical users to use the emotion state recognition function. 3) It can provide additional functions such as audio waveform display and result visualization to enhance the user experience. 4) Through application development, the research results are transformed into practical applications to provide users with useful emotion state analysis tools.

[0195] The following are the attached Figure 4 , 5 , 6 statements and analysis, and the conclusions drawn:

[0196] in Figure 4 The original VGG16 model was trained from scratch with an accuracy of 0.7522; Overfitting: Overfitting occurred, and there was a gap between the training and validation accuracy.

[0197] from Figure 4 As can be seen in the figure, the results of training the original VGG16 model from scratch show signs of overfitting, as the training accuracy is much higher than the validation accuracy. This means that the model performs well on the training data, but performs poorly on unseen data and is not suitable for generalizing to new data.

[0198] Figure 5 The VGG16 model is trained based on transfer learning. The accuracy is 0.9456. Overfitting: no overfitting occurs, and the training and verification accuracy are close.

[0199] Figure 5It shows that the VGG16 model training based on transfer learning improves the accuracy while reducing overfitting, and the gap between training and verification accuracy is small. This shows that transfer learning effectively utilizes the knowledge of the pre-loaded model and improves the performance and generalization ability of the model.

[0200] Figure 6 To improve VGG16+RNN model training: Accuracy: 0.9585; Overfitting: No overfitting occurred, and the training and verification accuracies were close.

[0201] Figure 6 The improved VGG16+RNN model training further improved the accuracy and maintained a low risk of overfitting. The introduction of the RNN architecture helps to process the sequential nature of audio data and improves the recognition accuracy of emotional states.

[0202] In summary, by comparing these three figures, we can conclude that transfer learning and improved model architecture have significantly contributed to the performance improvement and generalization ability of the emotional state recognition system, which can make the system more reliable and accurate in practical applications. The improved VGG16+RNN model performs best in terms of performance and generalization.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A postoperative emotional state recognition method integrating facial and voice features, characterized in that: The following steps are involved: S1. Data collection: collect audio data and facial expression data of several patients; S1.

1. Identify a number of postoperative patients, all aged 65 years or above, with a male to female ratio of 1:1; S1.

2. Each patient was asked to read a standardized text in a specific emotional state; these emotional states covered a variety of possible emotions, including normal, excited, and depressed emotions; S1.

3. Use professional audio recording equipment to record the patient's reading to ensure the quality and accuracy of the audio data; S1.

4. Each patient's reading audio is saved as a .wav file and categorized and stored according to emotional state; subfolders are created corresponding to normal, excited, and depressed emotions; S1.

5. Count the audio files in each subfolder to determine how many samples are obtained in total and the number of samples in each emotional state; S1.

6. Facial data collection uses high-resolution cameras to record the facial expressions of each patient to ensure the quality and clarity of the data; S1.

7. The facial expression data of each patient was saved as image files in .jpg or .png format and categorized and stored according to emotional state. Subfolders were also created to represent normal, excited, and depressed emotions respectively. S1.

8. Determine how many facial expression samples are obtained in total and the number of samples in each emotional state by counting the image files in each subfolder and filtering the valid data; S2, data preprocessing; The audio data obtained in the data collection phase needs to go through a preprocessing step to facilitate subsequent feature extraction and model training. The audio data preprocessing part converts the patient's audio data into a format that can be used for neural network training. First, a wavelet transform is performed to generate a wavelet coefficient matrix. Then, these coefficient matrices are converted into RGB image format. The size of the RGB image is set to 224*224*3 so that it can be compatible with the preloaded model. The specific implementation steps are as follows: S2.

1. First, the audio data in each emotional state is loaded into the memory and prepared for processing; S2.

2. For each audio file, use the `audioread` function of the audio processing library MATLAB to read the .wav format file into an audio signal. These audio signals are time series data with a sampling rate of Fs. S2.

3. Since the lengths of different audio files may be different, the lengths of all audio signals are unified into a fixed length; S2.4, classify the processed audio signal according to the emotional state and construct the corresponding data matrix; each row represents a sample, and each column represents the time series data of the audio signal; S2.5, associate the emotion state labels with the data matrix to facilitate supervised learning during the model training phase; S2.

6. Finally, save the preprocessed data as a .mat file to facilitate subsequent feature extraction and model training; S2.7, video data preprocessing; In the video data preprocessing part, the collected patient facial expression data is processed as the input of the neural network; first, face detection and key point location are performed to ensure the accurate location of the facial expression area; then, the facial image is cropped and resized to match the input specifications of the neural network; finally, these processed facial images are converted into 224*224 pixel RGB images; S3. Network model design: After data preprocessing, a neural network model is designed to effectively extract the features of emotional state from audio data. A fusion CNN and RNN method is used to diagnose emotional state. Complex features are extracted from samples of size 224*224*3 using VGG16. The extracted features are fed into the RNN classifier to distinguish normal, excited, and depressed emotional states. The VGG16 model is used as a basis for transfer learning to adapt it to the task of emotional state recognition. The specific implementation steps are as follows: S3.

1. Extract key features from audio images or facial images using different VGG16 models; For original VGG16: X image_features =VGG16 base (I image ) X audio_features =VGG16 base (I audio ) Among them, I image is the input facial image, I audio is the input audio image; S3.2, load the pre-trained VGG16 model as the basic model, and perform emotional state recognition based on the convolutional neural network structure in image recognition; S3.3,First, retain the bottom convolutional layers of the VGG16 model, which can extract the basic features of the audio data and extract the facial expression information; S3.

4. Next, we need to replace the last fully connected layer of the VGG16 model to adapt to the emotional state recognition task. We replace the fully connected layer with a new fully connected layer with output nodes that match the number of emotional categories. This new fully connected layer can map the features extracted by the underlying convolutional layer to the probability distribution of different emotional states. Replace the last fully connected layer of the VGG16 model with a new fully connected layer: X fc =W fc ·X image_features +b fc S3.5, add activation function and Dropout layer; S3.

6. Use the feature map as the input of the multi-layer RNN; Use the convolutional feature map as the input of the multi-layer RNN: X rnn_input =Convolution(X audio_features ) S3.7, apply Softmax classifier to classify the image; Apply the Softmax classifier, where C represents the number of sentiment categories: P emotion =Softmax(W softmax ·X rnn_input +b softmax ) Among them, P emotion is the probability distribution of the emotional state; S3.

8. After replacing the fully connected layer, you need to adjust the parameters related to the learning rate; S4, hardware accelerated training; S4.

1. When setting model training options, use hardware acceleration to speed up the training process. Set `ExecutionEnvironment` to 'gpu' in the training options to perform calculations on the GPU. S4.

2. Set the training options, `MiniBatchSize` the number of samples per iteration, `MaxEpochs` the maximum number of iterations. These parameters will affect the training effect and speed. Use NVIDIA GeForce GTX 1650 4G video memory, and set MiniBatchSize to 16; S4.

3. Start model training. During the training process, the deep learning toolbox will automatically perform forward propagation and back propagation to update the weights of the model so that it can gradually adapt to the task of emotion state recognition. S5. Design of emotional state fusion model: Design a model to fuse the feature extraction of audio data and facial expression data with the emotional state recognition model results to obtain the final emotional state recognition result; specifically implement it according to the following steps: S5.

1. Feature fusion: First, obtain the probability distribution results of the emotional state from the emotional state recognition model of the audio data, and the probability distribution results of the emotional state from the emotional state recognition model of the facial expression data; these probability distributions can be obtained through the Softmax classifier, and the probability of each category represents the possibility of the corresponding emotional state; S5.

2. Weight allocation: In order to merge the results of the two data sources, a weight allocation mechanism is introduced; Introduce weight distribution as W = [w1, w2, w3], these weights can be determined through training or domain knowledge; S5.3, Fusion method: Use a fusion method to fuse the emotion state probability distributions of the two data sources; adopt a weighted average, in which the probability distribution of each data source is multiplied by its weight and then added, which will produce a fused emotion state probability distribution; Emotional state fusion can be expressed as the following formula: Merged_Prediction=W[0]*A+W[1]*F Merged_Prediction is the probability distribution of the fused emotional state, and W[0] and W[1] are the weights of the audio data and facial expression data; S5.4, Final decision: According to the fused probability distribution of emotional states, the emotional state with the highest probability is selected as the final emotional state recognition result; this is achieved by finding the emotional state category corresponding to the maximum value in the probability distribution; The final decision can be expressed as: Final_Emotion=argmax(Merged_Prediction) Here, Final_Emotion is the final emotional state, which is the emotional state category with the highest probability in Merged_Prediction; S5.5, Threshold setting: To further improve the credibility of the results, a threshold is set to determine whether the result should be marked as uncertain when the highest probability is not obvious; if the highest probability is lower than the threshold, the result is marked as uncertain, otherwise the emotional state corresponding to the highest probability is selected; S6. Desktop application development: After completing the training and optimization of the model, it is applied to the actual emotion state recognition scenario to provide users with a friendly desktop application interface. The specific implementation includes the following steps: S6.

1. Choose to use the App Designer tool in MATLAB. S6.

2. First, create a new App Designer project and add corresponding UI components on the interface, including buttons, text boxes, and charts, so that users can interact with the system; S6.

3. Add an audio input function to the interface so that users can record or upload audio files for emotion state recognition; S6.4, by calling the trained model, inputting the audio data into the model, and identifying the emotional state; then, displaying the recognition result on the interface, displaying the recognized emotional state label; S6.5, add functions to display audio waveform in real time, visualize results, and export recognition results; S6.

6. Finally, the application is tested to ensure that it works properly under different circumstances.

2. The postoperative emotional state recognition method integrating facial and voice features according to claim 1, characterized in that: In step S2.3, if the audio signal is not long enough, zeros are added to the end, and if the audio signal is too long, it is truncated.

3. The postoperative emotional state recognition method integrating facial and voice features according to claim 1, characterized in that: In step S3.3, the facial expression information includes facial edge and texture information.

4. The postoperative emotional state recognition method integrating facial and voice features according to claim 1, characterized in that: In step S3.5, activation function and Dropout layer are added. After step S3.3 of the neural network model, the following steps can be continued to enhance the performance and robustness of the model; S3.5.

1. Add activation function. Introduce activation function between fully connected layer 1 and fully connected layer 2. Use RectifiedLinear Unit, i.e. ReLU activation function. S3.5.

2. Add a Dropout layer. A Dropout layer can be added between fully connected layer 1 and fully connected layer 2. Set the dropout rate to 0.5 to determine the probability of shutting down neurons. Add the ReLU activation function: X fc_activated =ReLU(X fc ) Add Dropout layer: X fc_dropped =Dropout(X fc_activated )。 5. A postoperative emotional state recognition device integrating facial and voice features, comprising a system of a postoperative emotional state recognition method integrating facial and voice features as described in any one of claims 1 to 4, characterized in that: Also includes computers or embedded devices, screens or displays, audio input devices, cameras, and speakers.

Images