Real-time emotion recognition system and method based on video stream and voice

Abstract

The invention discloses a real-time emotion recognition system and method based on video streams and voices, and the system comprises a data collection module which is used for collecting video frames of facial expressions of a user in real time through a camera, and collecting voice signals of the user in real time through a microphone; the feature extraction module is used for inputting the video frames into a convolutional neural network to extract visual features, and inputting the voice signals into a long short-term memory network to extract time sequence features; the weighted fusion module is used for carrying out weighted fusion on the visual features and the time sequence features to generate a multi-modal feature vector; and the emotion recognition module is used for calling an API (Application Program Interface) of the external large model, further recognizing the emotion in the video frame, obtaining an emotion recognition result fed back by the external large model, and further combining the emotion recognition result fed back by the external large model with the multi-modal feature vector so as to generate and output a final emotion recognition result. According to the invention, the accuracy, efficiency and real-time performance of emotion recognition can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to a real-time emotion recognition system and method based on video stream and voice. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and deep learning technologies, emotion recognition, as a core task in human-computer interaction, has received extensive attention and research. Emotion recognition can not only help intelligent systems understand the user's emotional state, but can also be used in application scenarios such as health monitoring, intelligent customer service, and emotional feedback in the education field.

[0003] Traditional emotion recognition systems are mostly based on a single modality, such as using only facial images or only voice signals to classify and recognize emotions. However, emotional expression is multimodal, including both non-verbal signals (such as facial expressions, gestures, body postures) and language and voice signals. Therefore, it is difficult for a single-modality emotion recognition system to fully and accurately capture the user's true emotional state.

[0004] In order to solve the above problems, multimodal emotion recognition has gradually become a hot research direction. Multimodal emotion recognition can capture emotional clues more comprehensively by integrating visual and auditory information. However, how to efficiently integrate data from different modalities and achieve real-time emotional feedback during system operation is a difficulty in current technology. Summary of the invention

[0005] In view of the shortcomings of existing emotion recognition technology, the present invention proposes a large language model question-answering system and method based on knowledge graph enhancement, which further analyzes facial expressions in video frames by combining external large models, such as ChatGPT, Kimi, and other online emotion recognition models that support uploaded image files. The external large model interacts with the system provided by the present invention through an API interface, performs in-depth emotion analysis in combination with the features in the video frame, and combines the final feedback emotion results with the system's own multimodal analysis results to generate the final weighted results. This fusion mechanism enables the present invention to not only make accurate judgments based on real-time user emotion input, but also use the powerful emotion analysis capabilities of the large model to improve emotion recognition accuracy.

[0006] The present invention provides a real-time emotion recognition system based on video stream and speech, comprising: Data acquisition module: used to collect video frames of user's facial expressions through the camera in real time, and collect user's voice signals through the microphone in real time; Feature extraction module: used for inputting the video frame into a convolutional neural network to extract visual features, and inputting the speech signal into a long short-term memory network to extract temporal features; Weighted fusion module: used for weighted fusion of the visual features and the temporal features to generate a multimodal feature vector; Emotion recognition module: used to call the API of the external large model, further identify the emotions in the video frame, obtain the emotion recognition results fed back by the external large model, and further combine the emotion recognition results fed back by the external large model with the multimodal feature vector to generate and output the final emotion recognition results.

[0007] Furthermore, the feature extraction module includes: extracting a multi-layer feature map from the user's facial expression through a convolutional neural network.

[0008] Furthermore, the feature extraction module includes: extracting hidden states and unit states from the speech signal through a long short-term memory network.

[0009] Furthermore, the feature extraction module includes: extracting visual features from the video frames through a pre-trained model including OpenFace and FaceNet.

[0010] Furthermore, the feature extraction module includes: extracting time series features from the speech signal through a spectrogram generation model and an emotion feature extraction model including MFCC and RNN.

[0011] Furthermore, the system also includes an incremental learning module: used for recording the user's emotional data in real time, and using an incremental learning algorithm to update the emotion recognition model.

[0012] Furthermore, the incremental learning algorithm includes an iCaRL algorithm and an EWC algorithm.

[0013] The present invention also discloses a real-time emotion recognition method based on video stream and voice, comprising the following steps: Step S1: The video frame of the user's facial expression is collected in real time through the camera, and the voice signal of the user is collected in real time through the microphone; Step S2: inputting the video frame into a convolutional neural network to extract visual features, and inputting the speech signal into a long short-term memory network to extract temporal features; Step S3: weighted fusion of the visual features and the temporal features to generate a multimodal feature vector; Step S4: calling the API of the external large model to further identify the emotions in the video frame, obtaining the emotion recognition result fed back by the external large model, and further combining the emotion recognition result fed back by the external large model with the multimodal feature vector to generate and output the final emotion recognition result.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. It improves the accuracy of emotion recognition and avoids the one-sidedness of single modality recognition.

[0015] 2. The emotion recognition system can learn and adapt to the user's emotional changes in real time during long-term use, avoiding the problem of "catastrophic forgetting".

[0016] 3. It solves the problem of large computing resources occupied in real-time calling of large emotion recognition models, ensuring the efficiency and real-time performance of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A functional module diagram of a real-time emotion recognition system based on video stream and speech provided by the present invention; Figure 2 A flowchart of a real-time emotion recognition method based on video stream and speech provided by the present invention; Figure 3 A schematic diagram of a process flow when the system or method provided by the present invention processes an embodiment; Figure 4 is a schematic diagram of a processing flow of a multimodal model in an embodiment of the present invention; Figure 5 It is a processing logic block diagram of the incremental learning module of the present invention; Figure 6 It is a processing logic block diagram when the present invention calls the large model strategy. DETAILED DESCRIPTION

[0018] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0019] like Figure 1 As shown, the present invention provides a real-time emotion recognition system based on video stream and voice, the system comprising: Data acquisition module 100: used to collect video frames of user's facial expressions in real time through a camera, and collect user's voice signals in real time through a microphone; Feature extraction module 200: used to input the video frame into the convolutional neural network to extract visual features, and input the speech signal into the long short-term memory network to extract temporal features; Weighted fusion module 300: used to perform weighted fusion of visual features and temporal features to generate a multimodal feature vector; Emotion recognition module 400: used to call the API of the external large model, further identify the emotions in the video frame, obtain the emotion recognition results fed back by the external large model, and further combine the emotion recognition results fed back by the external large model with the multimodal feature vector to generate and output the final emotion recognition results.

[0020] like Figure 1 As shown in the overall system architecture diagram, the emotion recognition system provided by the present invention includes five modules in total: data acquisition module, feature extraction module, weighted fusion module, emotion recognition module and incremental learning module. When the user interacts with the system through the camera and microphone in the data acquisition module, the system first extracts and weights emotion features through the feature extraction module and the weighted fusion module to achieve dual-modal emotion recognition, and then calls the API of the external large model to optimize the emotion recognition results, and then the system can be continuously updated and optimized through the incremental learning module.

[0021] Specifically, in the feature extraction module, the emotional features, i.e., visual features, in the video frame can be extracted through a convolutional neural network (CNN) or a pre-trained model such as OpenFace or FaceNet. In the specific extraction process, a multi-layer feature map is extracted from the facial expressions of the face in the video frame through a convolutional neural network.

[0022] Among them, the OpenFace model is a face recognition model based on deep learning. It uses a deep convolutional neural network (CNN) and a face recognition algorithm with three main components, including face detection, face alignment and face feature extraction. It can be used to recognize and understand the emotions expressed by the face, provide support for fields such as sentiment analysis and psychological research, and has the advantages of high accuracy and robustness, fast processing speed and strong scalability. The FaceNet model is a deep learning model proposed to solve problems such as face recognition, verification and clustering. The FaceNet model has a wide range of applications in the field of face recognition. It has high recognition accuracy, good real-time and robustness, and is an efficient and accurate face recognition model with broad application prospects.

[0023] In the feature extraction module, the emotional features in the speech signal are extracted through the Long Short-Term Memory (LSTM) network, which extracts the temporal features of the speech signal including the hidden state and the unit state. The LSTM network is a special recurrent neural network structure that can capture long-term dependencies in time series data.

[0024] In other embodiments, the time series features may be extracted from the speech signal by using a spectrogram generation model and an emotion feature extraction model including MFCC and RNN.

[0025] Among them, MFCC (Mel Frequency Cepstral Coefficients) is a feature widely used in speech processing and automatic recognition. Based on the auditory characteristics of the human ear, it converts the spectrum of the speech signal into a representation form that is more suitable for the auditory characteristics of the human ear. MFCC features are mainly used in tasks such as speech recognition, speech enhancement, and speaker recognition. MFCC features are an important tool in the field of speech processing, and they play a key role in tasks such as speech recognition and speaker recognition.

[0026] RNN, or Recurrent Neural Network, is a neural network architecture specifically designed to process sequence data. Compared with traditional neural networks (such as fully connected neural networks or convolutional neural networks), the core advantage of RNN is that it can capture time dependencies and dynamic behaviors in sequences. In speech recognition, RNN can process the timing characteristics in audio signals and convert them into text. By extracting and classifying speech signals through the recurrent neural network (RNN) model, the corresponding spectrogram can be generated.

[0027] Specifically, a spectrogram is a speech spectrum graph, which is obtained by processing the received time domain signal. It is widely used in the fields of phonetics, signal processing and audio. It is a view of spectrum analysis, especially for speech data. The horizontal axis of the spectrogram represents time, the vertical axis represents frequency, and the value of the coordinate point represents the energy of the speech data. The principle is to decompose the speech signal into multiple frequency components and plot the intensity of each frequency component over time.

[0028] The spectrogram generation model is a process of converting speech signals into visual spectrograms. The process of generating spectrograms specifically includes: Acquisition: Use a microphone or other recording device to collect voice signals and convert them into digital signals; Preprocessing: perform denoising, filtering and other preprocessing; Framing: Split the speech signal into multiple short time frames for spectrum analysis. The frame length and overlap rate can be set as needed.

[0029] Windowing: To avoid spectrum leakage during the framing process, each frame of the signal is usually windowed. Common window functions include Hanning window and Hamming window.

[0030] Fourier transform: Perform Fourier transform on each frame of windowed signal, converting it from time domain to frequency domain to obtain spectrum information.

[0031] Draw a spectrogram: Arrange the spectrum information of each frame in time order and draw it into a two-dimensional image, namely a spectrogram. In a spectrogram, the depth of color represents the size of energy, and a color bar is usually used to represent the strength of energy.

[0032] In the weighted fusion module, initial weights are set for visual features and temporal features, and the comprehensive score is calculated by weighting. According to the recognition effects of different modes during actual operation, the system can use dynamic adjustment strategies to adaptively optimize the weights to ensure accurate recognition under different emotional expressions.

[0033] In the emotion recognition module, the system calls large models (such as GPT, GPT-Emotion, etc.) to perform further emotion recognition on video frames, and can readjust and calculate the weights of visual features and temporal features based on the emotion recognition results fed back by the large model to optimize emotion recognition.

[0034] GPT, or Generative Pre-trained Transformer, is an advanced dialogue generation model. With the continuous advancement of artificial intelligence technology, GPT is gradually showing its potential in understanding and responding to emotional needs, which is mainly reflected in emotion recognition and understanding. The GPT-Emotion model is a model of GPT in the application of emotion recognition. By combining the GPT-Emotion big data model to identify emotions in video frames, it is helpful to improve the accuracy of emotion recognition.

[0035] In order to reduce the computational overhead caused by real-time calling of the large model, the system can adopt a frame extraction strategy to call the large model only for some key frames, or locally process the feature information of some layers and then transmit it to the large model for advanced recognition.

[0036] In an optional implementation, the system may further include an incremental learning module, which is used to record the user's emotion data in real time and use an incremental learning algorithm to update the emotion recognition model.

[0037] Specifically, the system records the user's emotional data in real time and uses an incremental learning algorithm to update the emotion recognition model, ensuring that the model does not forget old data because of new data.

[0038] Among them, the incremental learning algorithm may include the iCaRL algorithm and the EWC algorithm. By introducing knowledge distillation or memory-based algorithms such as iCaRL and EWC, the system can maintain the grasp of previously learned content when receiving new data.

[0039] The iCaRL (Incremental Classifier and Representation Learning) algorithm is a strategy for incremental class learning. It allows the model to perform incremental learning when only a small number of classes of training data exist at the same time, and can gradually add new categories. By using a memory buffer and knowledge distillation, iCaRL can retain previously learned knowledge and avoid forgetting. At the same time, iCaRL uses a dynamic threshold strategy to control the forgetting and retention of the model to adapt to the distribution of data of different categories. The iCaRL algorithm can effectively control computing requirements and memory usage while maintaining high classification accuracy.

[0040] The EWC (Elastic Weight Consolidation) algorithm is an algorithm designed to solve the catastrophic forgetting problem of neural networks in sequential learning tasks. The EWC algorithm is an effective strategy for helping neural networks maintain their memory of old tasks in sequential learning tasks, thereby avoiding catastrophic forgetting and continuing to perform well on new tasks.

[0041] Therefore, based on the above embodiments, the present invention provides a real-time emotion recognition system based on video stream and voice, which performs recognition by combining the bimodal data of video and voice, and sets weights to perform weighted fusion on the recognition results of the two modalities to obtain the final emotion judgment. At the same time, the present invention also introduces an incremental learning mechanism, allowing the system to update the model according to the user's real-time data during operation, ensuring that the system can continue to learn and adapt to the dynamic changes of the user's emotions. In addition, the present invention also ensures the high efficiency of large model recognition by optimizing the large model API call strategy.

[0042] like Figure 4 As shown, the system adopts a multimodal emotion recognition model, which may include an input layer, a hidden layer, a multimodal fusion layer, and an output layer, which respectively correspond to the data acquisition module, feature extraction module, weighted fusion module, and emotion recognition module in the system.

[0043] Specifically, the input layer includes video stream input (Video) and voice input (Audio).

[0044] The video stream input is a continuous video frame, and the size of each frame is , where W is the width of the frame, H is the height of the frame, and C is the number of channels (usually RGB three channels). The input shape is [N,T,W,H,C], where N is the batch size and T is the number of time frames.

[0045] The speech input is a speech signal, which is preprocessed to generate a spectrogram or LSTM feature. The input shape is [N, F, T], where N is the batch size, F is the feature dimension, and T is the number of time frames.

[0046] In the hidden layer, there are video processing network and speech processing network.

[0047] In the video processing network, a convolutional neural network (CNN) is used to extract visual features including emotional features from video frames. The formula of the feature extraction network is as follows: In the formula, is the video feature, is the input video frame. The convolutional layer uses multiple layers of convolution and pooling operations to gradually reduce the spatial dimensions of the video frame and increase the abstract representation of emotional features.

[0048] In the speech processing network, the time series features are extracted through recurrent neural network (RNN) or long short-term memory network (LSTM). LSTM can effectively capture the emotional change characteristics in speech. The formula used is as follows: In the formula, is the voice feature, is the input speech feature.

[0049] In the multimodal fusion layer, visual features and temporal features are fused through weighted summation or attention mechanism to obtain comprehensive emotional features, where a simple weighted formula as shown below can be used: In the formula, and are the weights of video and voice respectively, is the final fused feature vector. By dynamically adjusting the weights, the influence of the two modes can be optimized.

[0050] In the output layer, the emotion recognition result is output through the fully connected layer and the softmax activation function. The formula of the output layer is as follows: In the formula, is the weight matrix, is the bias matrix, is the probability of the output emotion recognition result, is the normalized exponential function.

[0051] When training the model, the cross entropy loss function is used to measure the gap between the model prediction and the true label. The loss function definition formula is as follows: In the formula, is the loss function, is the true label of the i-th sample, is the predicted probability of the model, N is the batch size, C is the number of categories, and N is the total number of samples.

[0052] At the same time, during the training process, the Adam optimizer is used to update the parameters. The update formula of the optimizer is as follows: in, is the learning rate, are model parameters, is the loss function, t and t+1 are the number of iterations, is the gradient of the loss function with respect to the current function.

[0053] like Figure 5 As shown in Figure 1, in order to avoid catastrophic forgetting during training, we introduced regularized incremental learning algorithms, such as the elastic weight preservation (EWC) algorithm. By regularizing the important parameters of the old tasks during training, we prevent these parameters from being over-modified in the new tasks.

[0054] Among them, the calculation formula of the EWC loss function is as follows: In the formula, is the loss function of the current task, is the regularization strength, is the Fisher information matrix, The parameters for the old task.

[0055] like Figure 6 As shown in the figure, in order to reduce the computational cost in real-time emotion recognition, the system adopts a frame extraction strategy in the call and optimization of the big model API. That is, the big model API is only called for key frames with significant emotion changes. The selection of frames can be done by calculating the amplitude of emotion changes between frames (such as changes in facial features). When the amplitude of the change is greater than a threshold, the system will call the big model for recognition.

[0056] The specific formula is as follows: In the formula, is the magnitude of the change in emotional characteristics, is the emotional feature of the t+1th frame, is the emotional feature of the tth frame. When the set threshold is exceeded (assuming the threshold is m), the call of the large model API is triggered.

[0057] like Figure 2 and Figure 3 As shown, the present invention also provides a real-time emotion recognition method based on video stream and voice, comprising the following steps: Step S1: The video frame of the user's facial expression is collected in real time through the camera, and the user's voice signal is obtained through the microphone; Step S2: Input the video frame into the convolutional neural network to extract visual features, and input the speech signal into the long short-term memory network to extract temporal features; Step S3: weighted fusion of visual features and temporal features to generate a multimodal feature vector; Step S4: Call the API of the external large model to further identify the emotions in the video frame, obtain the emotion recognition results fed back by the external large model, and further combine the emotion recognition results fed back by the external large model with the multimodal feature vector to generate and output the final emotion recognition results.

[0058] In summary, the real-time emotion recognition system and method based on video stream and voice provided by the present invention has the following beneficial effects compared with the prior art: 1. Dual-modal combination for more comprehensive recognition: The dual-modal combination of video and voice solves the problem that a single modality cannot fully reflect the emotional state.

[0059] 2. Incremental learning, more adaptable: The system can dynamically learn the user's emotional changes, achieve personalized emotion recognition, and prevent "catastrophic forgetting".

[0060] 3. Optimized calling of large models with better real-time performance: While introducing large models for deep recognition, the calling strategy is optimized to avoid real-time problems and improve efficiency.

[0061] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A real-time emotion recognition system based on video stream and speech, characterized in that: include: Data acquisition module: used to collect video frames of user's facial expressions through the camera in real time, and collect user's voice signals through the microphone in real time; Feature extraction module: used for inputting the video frame into a convolutional neural network to extract visual features, and inputting the speech signal into a long short-term memory network to extract temporal features; Weighted fusion module: used for weighted fusion of the visual features and the temporal features to generate a multimodal feature vector; Emotion recognition module: used to call the API of the external large model, further identify the emotions in the video frame, obtain the emotion recognition results fed back by the external large model, and further combine the emotion recognition results fed back by the external large model with the multimodal feature vector to generate and output the final emotion recognition results.

2. A real-time emotion recognition system based on video stream and voice according to claim 1, characterized in that: The feature extraction module includes: extracting a multi-layer feature map from the user's facial expression through a convolutional neural network.

3. A real-time emotion recognition system based on video stream and voice according to claim 1, characterized in that: The feature extraction module includes: extracting hidden states and unit states from speech signals through a long short-term memory network.

4. A real-time emotion recognition system based on video stream and voice according to claim 1, characterized in that: The feature extraction module includes: extracting visual features from the video frame by using a pre-trained model including OpenFace and FaceNet.

5. A real-time emotion recognition system based on video stream and voice according to claim 1, characterized in that: The feature extraction module includes: extracting time series features from the speech signal through a spectrogram generation model and an emotion feature extraction model including MFCC and RNN.

6. A real-time emotion recognition system based on video stream and voice according to claim 1, characterized in that: The system also includes an incremental learning module: used for recording the user's emotion data in real time, and using an incremental learning algorithm to update the emotion recognition model.

7. A real-time emotion recognition system based on video stream and voice according to claim 6, characterized in that: The incremental learning algorithm includes the iCaRL algorithm and the EWC algorithm.

8. A real-time emotion recognition method based on video stream and speech, characterized in that: The following steps are involved: Step S1: The video frame of the user's facial expression is collected in real time through the camera, and the voice signal of the user is collected in real time through the microphone; Step S2: inputting the video frame into a convolutional neural network to extract visual features, and inputting the speech signal into a long short-term memory network to extract temporal features; Step S3: weighted fusion of the visual features and the temporal features to generate a multimodal feature vector; Step S4: calling the API of the external large model to further identify the emotions in the video frame, obtaining the emotion recognition result fed back by the external large model, and further combining the emotion recognition result fed back by the external large model with the multimodal feature vector to generate and output the final emotion recognition result.

Images