Subjective mode and objective mode fused emotion recognition model method and system
By integrating multimodal data from facial, speech, text, and EEG signals into a maritime crew emotion recognition model, and utilizing deep learning networks and multi-head attention mechanisms for feature interaction and decision-level fusion, the accuracy problem of maritime crew emotion recognition was solved, achieving accurate identification and real-time support for crew emotional states.
Patent Information
- Application Number
- CN202511431820.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-09
- Publication Date
- 2025-11-21
AI Technical Summary
Existing technologies lack multimodal fusion methods that can simultaneously combine subjective modalities (facial, voice, text) and objective modalities (electroencephalogram signals) in the special working environment of seafarers at sea, making it difficult to accurately identify the emotional state of seafarers.
This paper proposes an emotion recognition model that integrates subjective and objective modalities. By acquiring and preprocessing facial images, speech audio, and EEG signal data, it utilizes deep learning networks and multi-head attention mechanisms for feature interaction and fusion, and combines decision-level fusion strategies to achieve emotion recognition.
It enables accurate identification of the emotional state of seafarers at sea, improves the comprehensiveness and robustness of emotion recognition, and provides real-time support in harsh environments.
Smart Images
Figure CN120983040A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a human emotion recognition technology, and more particularly to an emotion recognition model method and system that integrates subjective and objective modalities. Background Technology
[0002] In recent years, multimodal emotion recognition methods have been gradually proposed to more reliably capture the complex physiological characteristics of emotions. Some studies focus on the fusion of external behavioral signals; for example, a novel multimodal fusion attention network based on adaptive and factorization-based bilinear pooling has been proposed for audiovisual emotion recognition, achieving leading performance in this field. Other studies have found that fusing physiological signals with external signals can achieve better results. Researchers have proposed a decision-level fusion method combining facial expressions and EEG signals for real-time emotion recognition, which has also yielded good results. However, past research has primarily relied on facial expressions and EEG, neglecting speech as an important source of emotional information. To address this issue, recent research has proposed more comprehensive mode selection, utilizing facial expressions, speech, and EEG to determine a person's emotions from multiple perspectives. Furthermore, multimodal emotion recognition models are beginning to combine EEG data with facial video features, improving emotion recognition performance through multimodal feature fusion. These studies indicate that multimodal emotion recognition generally outperforms single-modal models, and the influence of multimodality has a positive impact on the performance of emotion recognition models.
[0003] In the maritime shipping industry, the mental health and emotional state of crew members are crucial to the smooth operation of ships. However, in the maritime environment, crew members may face challenges such as severe weather, feelings of isolation, and long voyages, all of which can negatively impact their mood. Therefore, accurately understanding and managing the emotional state of crew members is particularly important in this unique working environment.
[0004] In summary, the current problem is:
[0005] Current technologies lack emotion recognition models tailored to the unique working environment and psychological state of seafarers, and lack multimodal fusion methods that can simultaneously combine subjective modalities (facial, voice, text) and objective modalities (EEG signals), making it difficult to accurately monitor and identify seafarers' emotions. Summary of the Invention
[0006] The purpose of this invention is to provide an emotion recognition model method and system that integrates subjective and objective modalities, which can accurately identify the emotional state of seafarers.
[0007] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:
[0008] An emotion recognition model method that integrates subjective and objective modalities is proposed. In this method, subjective facial-voice-text modalities and objective electroencephalographic modalities are modeled separately, and then decision-level fusion is used to recognize the emotional state of seafarers.
[0009] A method for emotion recognition that integrates subjective and objective modalities, the method comprising:
[0010] S1, For the personnel being analyzed, obtain relevant subjective modal data and objective modal data;
[0011] S2 involves preprocessing the subjective and objective modal data of the analyzed personnel, including:
[0012] S21, Preprocess the face image to obtain a facial image sequence.
[0013] S22, preprocess the speech audio to obtain the speech audio sequence.
[0014] S23, the speech audio sequence is transcribed into a speech character stream and divided into word element sequences.
[0015] S24, preprocess physiological information to obtain frequency domain features of the EEG channel;
[0016] S3, for facial image sequence data, extract facial expression embedding features to obtain expression embedding feature sequence;
[0017] S4, for the speech audio sequence, extract the audio feature sequence;
[0018] S5 analyzes and processes the word element sequence to obtain the text feature sequence;
[0019] S6, capture the static visual features of the current frame;
[0020] S7. The facial expression embedding feature sequence, audio feature sequence and text feature sequence are encoded using 4GRU to obtain their respective dynamic single-modal feature representations.
[0021] S8 uses a multi-head attention mechanism to achieve feature interaction and fusion for static visual features, facial expression embedding feature sequences, audio feature sequences, and text feature sequences, resulting in a comprehensive feature representation.
[0022] S9 inputs the comprehensive features into the multilayer perceptron and outputs the emotion recognition result in the subjective modality.
[0023] S10: Input the frequency domain features of the extracted EEG channels into the stacked bidirectional LSTM model to capture time-series dependencies and output the emotion recognition results in the objective modality.
[0024] S11 employs a decision-level fusion strategy to weightedly fuse the "emotion recognition results under the subjective modality" and the "emotion recognition results under the objective modality" to obtain a comprehensive emotion recognition result.
[0025] Furthermore, step S1 also includes:
[0026] The process of acquiring subjective modality data includes:
[0027] Facial image sequence data of the subjects are collected and analyzed by high-definition cameras installed in the ship's bridge and rest areas.
[0028] Voice audio data of the personnel being analyzed is collected using microphones installed in the ship's work area.
[0029] Speech recognition technology is used to convert audio into text data to obtain the spoken expressions of the people being analyzed.
[0030] The process of acquiring objective modal data includes:
[0031] The physiological information data of the subjects is collected and analyzed through wearable physiological monitoring devices, including electroencephalogram (EEG), heart rate, and skin conductance response physiological signals.
[0032] Electroencephalogram (EEG) data, as the primary objective modal data, is used to reflect the true emotional state of the subjects being analyzed.
[0033] Furthermore,
[0034] In step S21, the face image is preprocessed to obtain a face image sequence. Specifically, the OpenFace detector is used to detect and locate the face in the video frame, the detected face image is cropped to a size of 224×224 pixels, and then aligned and normalized to finally obtain a standardized face image sequence.
[0035] In step S22, the speech audio is preprocessed to obtain a speech audio sequence. Specifically, the speech signal is divided into several segments of 20~30ms. A window function is applied to each segment to detect and retain the sound signal, remove silent and quiet frames, eliminate or reduce background noise, and obtain a speech audio sequence.
[0036] In step S23, the speech audio sequence is transcribed into a speech character stream and divided into word element sequences. The tool used is an automatic speech recognition tool.
[0037] In step S24, the physiological information is preprocessed to obtain the frequency domain features of the EEG channel. Specifically, the time domain signal of the EEG is first converted into a frequency domain signal by fast Fourier transform, and then the frequency domain features of the selected EEG channel are calculated based on the fast Fourier transform results.
[0038] Furthermore,
[0039] In step S3, for the facial image sequence data, the expression embedding features of the facial sequence are extracted to obtain the expression embedding feature sequence. Specifically, for the preprocessed facial image sequence data, a pre-trained deep learning network is used to extract the expression embedding features of the cropped facial sequence. M adjacent frames before and after the current frame are selected, and the expression embedding features of the M adjacent frames are calculated through the deep learning network. Then, the output is represented as the expression embedding feature sequence.
[0040] Further, in step S4, for the speech audio sequence, an audio feature sequence is extracted. Specifically, for the preprocessed speech audio sequence, the librosa tool is used to extract the audio feature sequence, including Mel frequency cepstral coefficients, pitch and short-time energy features, and then they are concatenated into an audio feature sequence.
[0041] Further, in step S5, the word element sequence is analyzed and processed to obtain the text feature sequence. Specifically, for the word elements, spoken words are obtained through the ASR tool, and the spoken words are input into the pre-trained BERT model to generate word embedding vectors, thus obtaining the text feature sequence.
[0042] Furthermore,
[0043] In step S6, the static visual features of the current frame are captured, specifically by using a static visual feature extractor based on the Inception-ResNet structure to capture the static visual features of the current frame.
[0044] Furthermore,
[0045] In step S8, for static visual features, facial expression embedding feature sequences, audio feature sequences, and text feature sequences, a multi-head attention mechanism is used to achieve feature interaction and fusion, resulting in a comprehensive feature representation, specifically as follows:
[0046] Static visual features, facial expression embedding feature sequences, audio feature sequences, and text feature sequences are input into a transformer-based fusion module. Feature interaction and fusion are achieved through a multi-head attention mechanism to obtain a comprehensive feature representation.
[0047] An emotion recognition model system that integrates subjective and objective modalities includes a memory and a processor, and is capable of running computer programs. The emotion recognition model system is equipped with computer programs that can implement the emotion recognition model method described above.
[0048] The main advantages of the emotion recognition model method and system that integrates subjective and objective modalities of the present invention compared to existing technologies are as follows:
[0049] Subjective modalities (facial-voice-text) and objective modalities (EEG signals) are processed independently through two parallel channels: "Transformer fusion module + bidirectional LSTM". The results are then integrated using a decision-level fusion strategy, which enables accurate identification of the emotional state of seafarers. Attached Figure Description
[0050] Figure 1 This is a schematic diagram of the overall architecture of the emotion recognition model method that integrates subjective and objective modalities according to the present invention;
[0051] Figure 2 This is a schematic diagram of the basic architecture of the Transformer model of the emotion recognition model method of the present invention;
[0052] Figure 3 This is a schematic diagram of the unidirectional LSTM structure of the emotion recognition model method of the present invention, used for capturing time series features;
[0053] Figure 4 This is a schematic diagram of the bidirectional LSTM structure of the emotion recognition model method of the present invention, which is used to model the temporal dependency of EEG frequency domain features. Detailed Implementation
[0054] The following provides further details on specific embodiments of the present invention:
[0055] See Figures 1 to 4 This embodiment provides an emotion recognition model method that integrates subjective and objective modalities. The target of this emotion recognition model method is seafarers who have been engaged in maritime shipping for a long time. This emotion recognition model method is specifically used to analyze the emotional state of seafarers to determine whether there are any abnormalities in their emotional state, thereby providing seafarers with more accurate emotion management and support.
[0056] For ease of description, the crew members who are the target will be defined as "analysis target personnel".
[0057] See Figure 1 The emotion recognition model method of this embodiment includes the following steps S1 to S11.
[0058] S1, Data Acquisition
[0059] For the individuals being analyzed, obtain their relevant subjective and objective modal data. Specifically, this includes:
[0060] Subjective modal data:
[0061] 1) Collect and analyze facial image sequence data of the subjects by installing high-definition cameras in areas such as the ship's bridge and rest area;
[0062] 2) Collect and analyze the voice and audio data of the personnel being analyzed using microphones installed in the ship's work area;
[0063] 3) Use speech recognition technology to convert audio into text data to obtain the spoken expressions of the people being analyzed;
[0064] Objective modal data:
[0065] 1) Collect and analyze physiological information data of the subjects through wearable physiological monitoring devices, including physiological signals such as electroencephalogram (EEG), heart rate, and skin conductance response;
[0066] 2) Among them, electroencephalogram (EEG) data serves as the primary objective modal data, used to reflect the true emotional state of the analyzed individuals.
[0067] S2, Data Preprocessing
[0068] The subjective and objective modal data of the personnel being analyzed, obtained in step S1, are preprocessed. This preprocessing mainly includes steps S21 to S24:
[0069] S21, Face Image Preprocessing
[0070] The facial image sequence data in the subjective modality data is preprocessed, including face detection, alignment and normalization. Specifically, the OpenFace detector is used to detect and locate faces in the video frames, the detected facial images are cropped to a size of 224×224 pixels, and then aligned and normalized to obtain a standardized facial image sequence.
[0071] S22, Speech audio preprocessing
[0072] The speech signal is divided into several segments of 20-30ms. A window function is applied to each segment to detect and retain the sound signal, remove silent and quiet frames, and eliminate or reduce background noise to obtain the speech audio sequence.
[0073] S23, Acquisition of speech character stream
[0074] Using an automatic speech recognition (ASR) tool, the preprocessed audio sequence is transcribed into a stream of speech characters and divided into sequences of word elements.
[0075] S24, Physiological Information Preprocessing
[0076] First, the EEG time-domain signal is converted into a frequency-domain signal using Fast Fourier Transform (FFT). Then, the frequency domain characteristics of the selected EEG channels are calculated based on the FFT results, including the power spectral density characteristics of delta waves (0.5-4Hz), theta waves (4-8Hz), alpha waves (8-13Hz), beta waves (13-30Hz), and gamma waves (30-100Hz).
[0077] S3, for the facial image sequence data preprocessed in step S21, a pre-trained deep learning network (DLN) is used to extract the expression embedding features of the cropped facial sequence. Specifically, the M adjacent frames before and after the current frame are selected, and their expression embedding features are calculated through the deep learning network (DLN). The output is then represented as an expression embedding feature sequence, denoted as r. e .
[0078] S4, for the preprocessed speech audio sequence in step S22, the librosa tool is used to extract audio feature sequences, including MFCC (Mel-frequency cepstral coefficients), pitch, and short-time energy features, and then these are concatenated into an audio feature sequence, denoted as r. a .
[0079] S5, for the word elements obtained in step S23, use the ASR tool to extract spoken words from these word elements, and input the spoken words into the pre-trained BERT model to generate word embedding vectors, thus obtaining the text feature sequence, represented as r. w .
[0080] S6 employs a static visual feature extractor based on the Inception-ResNet architecture to capture the static visual features of the current frame, denoted as h. s .
[0081] The static visual feature extractor is initialized from a pre-trained representation embedding model extracted by a deep learning network (DLN).
[0082] S7, the facial expression embedding feature sequence r obtained in step S3 e The audio feature sequence r obtained in step S4 a and the text feature sequence r obtained in step S5 w 4GRU encoding is performed to obtain their respective dynamic single-modal feature representations. The details are as follows:
[0083] The dynamic unimodal feature representation of the facial expression embedding feature sequence is h. e ;
[0084] The dynamic single-modal features of an audio feature sequence are represented as h a ;
[0085] The dynamic unimodal feature representation of a text feature sequence is h. w .
[0086] S8, multimodal feature fusion
[0087] The static visual features h obtained in step S6 s , and all dynamic single-modal features h obtained in step S7 e h a and h w The input is fed into the transformer-based fusion module, where feature interaction and fusion are achieved through a multi-head attention mechanism to obtain a comprehensive feature representation.
[0088] The output of the second multi-head attention can be as follows:
[0089]
[0090] S9, subjective modality recognition output.
[0091] The integrated features obtained in step S8 are input into a multilayer perceptron (MLP) to output the emotion recognition result in the subjective modality.
[0092] S10, Objective Modality Recognition Output
[0093] The frequency domain features of the EEG channel extracted in step S24 are input into the stacked bidirectional LSTM model to capture time series dependencies and output the emotion recognition results in the objective modality.
[0094] S11, Decision-level Fusion
[0095] A decision-level fusion strategy is adopted to weight and fuse the recognition results of step S9 and step S10 to obtain a more comprehensive and accurate emotion recognition result.
[0096] See Figure 1 The figure shows the overall model architecture of the emotion recognition model method of this embodiment. The model consists of subjective modal pathways and objective modal pathways, which process facial, speech, text features and EEG signals respectively, and perform weighted fusion at the decision layer to output a comprehensive emotion recognition result.
[0097] See Figure 2The diagram shows the basic architecture of the Transformer model. This structure is mainly used to illustrate the attention mechanism principle that the emotion recognition model method of this embodiment draws on when modeling subjective modal feature interaction.
[0098] See Figure 3 The diagram shows the architecture of the LSTM model. This structure can effectively capture long-term dependencies in time series data, providing a foundation for time series modeling of objective modalities.
[0099] See Figure 4 The diagram shows the architecture of a bidirectional LSTM model. Compared to a unidirectional structure, this model can simultaneously acquire forward and backward dependent features, improving the accuracy and completeness of EEG signal sequence analysis.
[0100] The main advantages of the emotion recognition model method in this embodiment are:
[0101] 1) In the emotion recognition model method of this embodiment, subjective modalities (facial-voice-text) and objective modalities (EEG signals) are processed independently through two parallel channels: "Transformer fusion module + bidirectional LSTM". The results are then integrated using a decision-level fusion strategy, thereby enabling accurate identification of the emotional state of seafarers.
[0102] In addition, the emotion recognition model method of this embodiment has other advantages, as follows:
[0103] 2) In the emotion recognition model method of this embodiment, the continuous facial, speech and text sequences are first processed by 4GRU to obtain "dynamic" features, and then fed together with the "static" visual features extracted by single frame Inception-ResNet into the Transformer decoder for cross-attention, so as to achieve the complementarity of temporal dynamic and spatial static information.
[0104] 3) In the decision-level fusion strategy, after the two paths output probability vectors respectively, instead of using simple averaging or weighting, learnable fusion weights are introduced (which can be adjusted in real time using a small MLP or attention network), so that the system can adaptively favor subjective or objective modalities based on the real-time signal-to-noise ratio, thereby improving the reliability of the final recognition.
[0105] 4) Targeted application scenarios: This invention is the first to be designed for the special working environment of seafarers at sea. Taking into account factors such as their isolation, long-term voyages, and harsh environments, it can provide real-time support for ship navigation safety and the mental health of seafarers.
[0106] 5) Comprehensive multimodal coverage: Unlike existing technologies that only focus on the fusion of "facial expression + EEG" or "voice + facial expression", this invention is the first to simultaneously incorporate four modalities: facial, voice, text and EEG, which improves the comprehensiveness and robustness of emotion recognition.
[0107] Furthermore, this embodiment also provides an emotion recognition model system, which includes a memory and a processor and is capable of running a computer program. The emotion recognition model system is equipped with a computer program that can implement the above-described emotion recognition model method.
[0108] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for emotion recognition that integrates subjective and objective modalities, characterized in that: In the aforementioned emotion recognition model method, subjective facial-voice-text modalities and objective electroencephalographic modalities are modeled separately, and then decision-level fusion is used to realize the emotion state recognition of seafarers.
2. A method for emotion recognition that integrates subjective and objective modalities, characterized in that: The emotion recognition model method includes: S1, For the personnel being analyzed, obtain relevant subjective modal data and objective modal data; S2 involves preprocessing the subjective and objective modal data of the analyzed personnel, including: S21, Preprocess the face image to obtain a facial image sequence. S22, preprocess the speech audio to obtain the speech audio sequence. S23, the speech audio sequence is transcribed into a speech character stream and divided into word element sequences. S24, preprocess physiological information to obtain frequency domain features of the EEG channel; S3, for facial image sequence data, extract facial expression embedding features to obtain expression embedding feature sequence; S4, for the speech audio sequence, extract the audio feature sequence; S5 analyzes and processes the word element sequence to obtain the text feature sequence; S6, capture the static visual features of the current frame; S7. The facial expression embedding feature sequence, audio feature sequence and text feature sequence are encoded using 4GRU to obtain their respective dynamic single-modal feature representations. S8 uses a multi-head attention mechanism to achieve feature interaction and fusion for static visual features, facial expression embedding feature sequences, audio feature sequences, and text feature sequences, resulting in a comprehensive feature representation. S9 inputs the comprehensive features into the multilayer perceptron and outputs the emotion recognition result in the subjective modality. S10: Input the frequency domain features of the extracted EEG channels into the stacked bidirectional LSTM model to capture time-series dependencies and output the emotion recognition results in the objective modality. S11 employs a decision-level fusion strategy to weightedly fuse the "emotion recognition results under subjective modality" and the "emotion recognition results under objective modality" to obtain a comprehensive emotion recognition result.
3. The emotion recognition model method that integrates subjective and objective modalities according to claim 2, characterized in that: Step S1 also includes: The process of acquiring subjective modality data includes: Facial image sequence data of the subjects are collected and analyzed by high-definition cameras installed in the ship's bridge and rest areas. Voice audio data of the personnel being analyzed is collected using microphones installed in the ship's work area. Speech recognition technology is used to convert audio into text data to obtain the spoken expressions of the people being analyzed. The process of acquiring objective modal data includes: The physiological information data of the subjects is collected and analyzed through wearable physiological monitoring devices, including electroencephalogram (EEG), heart rate, and skin conductance response physiological signals. Electroencephalogram (EEG) data, as the primary objective modal data, is used to reflect the true emotional state of the subjects being analyzed.
4. The emotion recognition model method that integrates subjective and objective modalities according to claim 2, characterized in that: In step S21, the face image is preprocessed to obtain a face image sequence. Specifically, the OpenFace detector is used to detect and locate the face in the video frame, the detected face image is cropped to a size of 224×224 pixels, and then aligned and normalized to finally obtain a standardized face image sequence. In step S22, the speech audio is preprocessed to obtain a speech audio sequence. Specifically, the speech signal is divided into several segments of 20~30ms. A window function is applied to each segment to detect and retain the sound signal, remove silent and quiet frames, eliminate or reduce background noise, and obtain a speech audio sequence. In step S23, the speech audio sequence is transcribed into a speech character stream and divided into word element sequences. The tool used is an automatic speech recognition tool. In step S24, the physiological information is preprocessed to obtain the frequency domain features of the EEG channel. Specifically, the time domain signal of the EEG is first converted into a frequency domain signal by fast Fourier transform, and then the frequency domain features of the selected EEG channel are calculated based on the fast Fourier transform results.
5. The emotion recognition model method that integrates subjective and objective modalities according to claim 2, characterized in that: In step S3, for the facial image sequence data, the expression embedding features of the facial sequence are extracted to obtain the expression embedding feature sequence. Specifically, for the preprocessed facial image sequence data, a pre-trained deep learning network is used to extract the expression embedding features of the cropped facial sequence. M adjacent frames before and after the current frame are selected, and the expression embedding features of the M adjacent frames are calculated through the deep learning network. Then, the output is represented as the expression embedding feature sequence.
6. The emotion recognition model method that integrates subjective and objective modalities according to claim 2, characterized in that: In step S4, for the speech audio sequence, an audio feature sequence is extracted. Specifically, for the preprocessed speech audio sequence, the librosa tool is used to extract the audio feature sequence, including Mel frequency cepstral coefficients, pitch and short-time energy features, and then they are concatenated into an audio feature sequence.
7. The emotion recognition model method that integrates subjective and objective modalities according to claim 2, characterized in that: In step S5, the word element sequence is analyzed and processed to obtain the text feature sequence. Specifically, for the word elements, spoken words are obtained through the ASR tool, and the spoken words are input into the pre-trained BERT model to generate word embedding vectors, thus obtaining the text feature sequence.
8. The emotion recognition model method that integrates subjective and objective modalities according to claim 2, characterized in that: In step S6, the static visual features of the current frame are captured, specifically by using a static visual feature extractor based on the Inception-ResNet structure to capture the static visual features of the current frame.
9. The emotion recognition model method that integrates subjective and objective modalities according to claim 2, characterized in that: In step S8, for static visual features, facial expression embedding feature sequences, audio feature sequences, and text feature sequences, a multi-head attention mechanism is used to achieve feature interaction and fusion, resulting in a comprehensive feature representation, specifically as follows: Static visual features, facial expression embedding feature sequences, audio feature sequences, and text feature sequences are input into a transformer-based fusion module. Feature interaction and fusion are achieved through a multi-head attention mechanism to obtain a comprehensive feature representation.
10. An emotion recognition model system that integrates subjective and objective modalities, the emotion recognition model system comprising a memory and a processor, and capable of running computer programs, characterized in that: The emotion recognition model system is equipped with a computer program capable of implementing the emotion recognition model method as described in any one of claims 1 to 9.