An emotion recognition method fusing multi-modal physiological signal features and a related device
By employing an end-to-end deep learning architecture for multimodal physiological signal feature recognition, this method addresses the shortcomings of traditional feature extraction and multimodal fusion, achieving highly efficient emotion recognition and improving recognition accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-27
- Publication Date
- 2026-07-03
AI Technical Summary
In emotion recognition, existing technologies struggle to capture the details of physiological waveforms and fine-grained temporal patterns using traditional time-frequency domain features. Furthermore, multimodal signal fusion neglects deep nonlinear correlations, resulting in insufficient recognition accuracy.
An end-to-end deep learning architecture is constructed, employing multi-scale perception, attention, and gating mechanisms to extract and fuse local features from ECG and pulse signals respectively, and to improve recognition accuracy through global temporal information fusion.
It significantly improves the accuracy and robustness of emotion recognition, simplifies application deployment complexity, and enhances the comprehensiveness and stability of emotion monitoring.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention belongs to the field of biomedical signal processing technology, specifically relating to an emotion recognition method and related device that integrates multimodal physiological signal features. Background Technology
[0002] Affective computing aims to endow computers with the ability to recognize, understand, and respond to human emotions, and has significant application value in fields such as human-computer interaction, mental health monitoring, assisted driving, and telemedicine. Compared to external behavioral signals such as facial expressions and voice tone, which are easily controlled by subjective faking, physiological signals are regulated by the human autonomic nervous system and can objectively, in real time, and are difficult to fake, reflect an individual's true emotional state. Among these, electrocardiogram (ECG) signals and pulse wave (BVP) signals have become mainstream research objects in the field of emotion recognition due to their portable acquisition devices and rich information content.
[0003] Although deep learning-based physiological signal emotion recognition technology has made some progress, existing technical solutions still face the following core technical bottlenecks in practical applications, limiting the improvement of recognition accuracy: On the one hand, existing solutions often rely on traditional time-frequency domain statistical features, while changes in emotion are reflected not only in instantaneous physiological waveform distortions but also in periodic rhythmic trend changes. Manual features are difficult to fully characterize the local waveform details and fine-grained temporal patterns related to emotions. On the other hand, most existing methods use simple feature splicing or decision voting strategies for feature fusion, ignoring the distribution differences of different modalities in the feature space and failing to effectively model deep nonlinear correlations, making it difficult for the model to accurately determine the complementary relationship and intrinsic correlation between the two signals.
[0004] Therefore, there is an urgent need for an emotion recognition scheme that can extract effective discriminative features from ECG and BVP signals end-to-end and achieve dynamic complementary enhancement between modalities to improve recognition performance. Summary of the Invention
[0005] The purpose of this invention is to provide an emotion recognition method and related apparatus that integrates multimodal physiological signal features, aiming to solve the problems of poor feature extraction and difficulty in effectively modeling multimodal fusion in existing technologies. By constructing an end-to-end deep learning architecture, and utilizing multi-scale perception, attention, and gating mechanisms, efficient feature extraction and fusion are achieved, significantly improving the accuracy and robustness of emotion recognition.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides an emotion recognition method that integrates multimodal physiological signal features, comprising the following steps: The original electrocardiogram (ECG) signal and the original pulse signal of the subject were segmented to obtain ECG signal segment sequences and pulse signal segment sequences, respectively. Feature extraction was performed on the electrocardiogram (ECG) signal segment sequence and the pulse signal segment sequence respectively to obtain the ECG local feature sequence and the pulse local feature sequence. The local feature sequences of electrocardiogram and pulse are fused to obtain a fused feature sequence; The obtained fused feature sequence is then fused with global temporal information to obtain a global feature sequence; The emotion recognition prediction result is obtained based on the obtained global feature sequence.
[0007] Preferably, a sliding window is used to segment the original electrocardiogram (ECG) signal and the original pulse signal of the subject to be tested, respectively, to obtain ECG signal segment sequences and pulse signal segment sequences.
[0008] Preferably, the local feature perception module is used to extract features from the electrocardiogram signal segment sequence and the pulse signal segment sequence respectively to obtain the local feature sequence of electrocardiogram and the local feature sequence of pulse.
[0009] Preferably, the local feature perception module includes a waveform perception submodule, a phase perception submodule, and a perception feature fusion submodule, wherein: The waveform sensing submodule is used to perform multi-scale one-dimensional convolution parallel extraction on electrocardiogram signal segment sequences or pulse signal segment sequences to obtain waveform features; The phase sensing submodule is used to perform parallel dilated convolution on the electrocardiogram signal segment sequence or pulse signal segment sequence to obtain phase features; The sensing feature fusion submodule is used to fuse the obtained waveform features and phase features to obtain an electrocardiogram local feature sequence or a pulse local feature sequence.
[0010] Preferably, the waveform sensing submodule includes multiple parallel convolutional layers, and the outputs of the multiple convolutional layers are connected to the channel attention layer; The phase-aware submodule includes multiple parallel dilated convolutional layers, and the outputs of the multiple parallel dilated convolutional layers are connected to the channel attention layer. The perceptual feature fusion submodule includes a serial channel fusion convolutional layer, multiple residual convolutional blocks, and a feature aggregation pooling layer.
[0011] Preferably, the local feature sequences of electrocardiogram and pulse are fused using a multimodal fusion module to obtain a fused feature sequence.
[0012] Preferably, a multimodal fusion module is used to fuse the local feature sequences of electrocardiogram and pulse to obtain a fused feature sequence. Specifically, the method is as follows: Linear projections were performed on the local feature sequences of electrocardiogram and pulse respectively to obtain the electrocardiogram projection features and pulse projection features. Generate ECG query vectors using ECG projection features and generate ECG key vectors and ECG value vectors using pulse projection features; A pulse query vector is generated using pulse projection features, and a pulse key vector and pulse value vector are generated using electrocardiogram projection features. Based on the ECG query vector, ECG key vector, pulse query vector, and pulse key vector, the attention weight matrix corresponding to ECG and pulse is calculated. The ECG value vector and pulse value vector are weighted using the attention weight matrix to obtain ECG-assisted fusion features and pulse-assisted fusion features respectively; The ECG-assisted fusion features and pulse-assisted fusion features are scaled and then added to the corresponding ECG projection features and pulse projection features respectively using the residual method to obtain the ECG enhancement feature sequence and the pulse enhancement feature sequence. The ECG enhancement feature sequence and the pulse enhancement feature sequence are concatenated in feature dimension and then linearly transformed to one-dimensional features. The gating coefficients are obtained by passing the one-dimensional features through an S-shaped activation function; The ECG enhancement feature sequence and the pulse enhancement feature sequence are fused based on the gating coefficient to obtain the fused feature sequence.
[0013] Preferably, the obtained fused feature sequence is fused with global temporal information to obtain a global feature sequence. The specific method is as follows: The fused feature sequence is convolved in one dimension in the sequence dimension to obtain the position embedding; The obtained position embedding and fused feature sequences are added together to obtain a sequence; A learnable classification token is appended to the beginning of the obtained sequence to obtain the token sequence; By modeling the token sequence using global temporal information, we can obtain emotion classification and discrimination features.
[0014] Secondly, the present invention provides an emotion recognition system that integrates multimodal physiological signal features, comprising: The signal segmentation unit is used to segment the acquired original electrocardiogram (ECG) signal and original pulse signal of the subject to be tested, respectively, to obtain ECG signal segment sequences and pulse signal segment sequences; The feature extraction unit is used to extract features from the electrocardiogram signal segment sequence and the pulse signal segment sequence respectively, to obtain the local feature sequence of electrocardiogram and the local feature sequence of pulse. The feature fusion unit is used to fuse the local feature sequences of electrocardiogram and pulse to obtain a fused feature sequence; The temporal information fusion unit is used to perform global temporal information fusion on the obtained fusion feature sequence to obtain a global feature sequence; The emotion recognition unit is used to obtain emotion recognition prediction results based on the obtained global feature sequence.
[0015] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the method described thereon.
[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention provides an emotion recognition method that integrates multimodal physiological signal features. By segmenting the original electrocardiogram (ECG) and pulse signals separately, long-duration sequences are converted into standardized signal fragment sequences, laying a data foundation for subsequent local feature extraction and temporal modeling. This segmentation method effectively preserves the local variation patterns of the signals along the time axis, resulting in more refined feature extraction and avoiding the computational burden and information dilution problems associated with directly processing long sequences. Subsequently, local feature extraction is performed on both types of signals separately, fully mining the fine-grained information related to emotion in each modality. Compared to traditional manual feature extraction methods, this scheme, through a learnable feature extraction structure, can adaptively capture local waveform changes in ECG and pulse signals, improving the discriminative ability of feature representation and providing high-quality feature input for subsequent multimodal fusion. Finally, the ECG local feature sequences and pulse local feature sequences are fused, overcoming the limitations of simple splicing or decision-level fusion. Therefore, by constructing the interaction relationship between multiple modalities, this application can effectively integrate complementary emotional information in two types of physiological signals, so that the fused feature sequence contains not only the unique features of each modality, but also reflects the synergistic effect between the two, thereby enhancing the comprehensiveness and robustness of emotional representation.
[0017] Furthermore, by performing global temporal information fusion on the fusion feature sequence, the evolution of emotional state in the time dimension can be captured. Emotion does not exist in isolation in a single signal segment, but rather exhibits a certain continuity and changing trend. This step, by modeling the temporal dependencies between segments, enables the model to understand the dynamic changes of emotion as a whole, thereby improving the stability and consistency of the recognition results in the time dimension.
[0018] In summary, this application achieves end-to-end mapping from raw physiological signals to emotion categories through emotion recognition and prediction based on global feature sequences. The entire process requires no manual feature design or intermediate intervention, simplifying the complexity of application deployment. Furthermore, through multi-level information extraction and fusion, it significantly improves the accuracy and generalization ability of emotion recognition, providing feasible technical support for emotion monitoring based on ECG and pulse signals from wearable devices in real-world scenarios. Attached Figure Description
[0019] Figure 1 This is a general framework diagram of the emotion recognition method that integrates multimodal physiological signal features as described in this invention; Figure 2 This is a schematic diagram of the structure of the local feature perception module described in this invention; Figure 3 This is a schematic diagram of the structure of the multimodal fusion module described in this invention; Figure 4 This is a schematic diagram of the global timing modeling module described in this invention; Figure 5 The figure shows the experimental results of training and verification performance of the emotion recognition system described in this invention. Detailed Implementation
[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0021] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0022] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0023] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0024] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0025] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0026] Example 1 See Figure 1 This embodiment provides a multimodal physiological signal feature fusion and emotion recognition method. It uses ECG and BVP (also known as PPG) signals synchronously collected from the subject within the same time period as input. The signals are sequentially processed through preprocessing and segmentation, local feature perception, a multimodal fusion module, a global temporal modeling module, and a classifier module to output the emotion recognition result. This overall process can be implemented by an end-to-end multimodal emotion recognition model. The method can be used for both model training and deployment inference. In this embodiment, a publicly available emotion recognition dataset is used for training and inference experiments. The dataset includes synchronously collected ECG and BVP signals, with a sample signal length of 10 seconds. Emotional arousal and valence labels are provided, allowing for binary (high / low arousal or positive / negative valence) or four-class labeling for supervised training and inference. During the training phase, emotion data samples containing synchronous ECG and BVP signals are input into the model for forward propagation to obtain the classifier output, which is then processed using cross-entropy loss. Perform backpropagation optimization and update of model parameters:
[0027] in, The total number of categories, The true label for the sample belonging to category c. To predict the probability that a sample belongs to class c for the model, you can use... The optimizer and weight decay optimization training yield a multimodal emotion recognition model. During the inference phase, the trained multimodal emotion recognition model is used to perform forward propagation and output an emotion category prediction. The specific feature extraction and emotion recognition method includes the following steps: Step 1: Preprocessing and sliding window segmentation.
[0028] The original ECG signal and original pulse signal BVP of the subject to be tested are preprocessed to obtain preprocessed ECG and BVP signals. The preprocessed ECG and BVP signals are then segmented into corresponding signal segment sequences using a sliding window method.
[0029] In a preferred implementation of this invention, the sizes of the original input tensors ECG and BVP received by the model can be expressed as follows: After being divided, it forms The sequence of fragments is input to subsequent modules. For batch size, The original signal length, The number of signal segments. This represents the length of the signal segment.
[0030] Step 2: The local feature perception module extracts local feature sequences.
[0031] The local feature perception module is used to extract features from each signal segment of the ECG signal segment sequence and the BVP signal segment sequence to extract and fuse the waveform and phase characteristics of different modes, thereby obtaining the ECG local feature sequence and the BVP local feature sequence.
[0032] The preferred embodiment of the local feature perception module in this invention includes a waveform perception submodule, a phase perception submodule, and a perception feature fusion submodule. The input is a sequence of ECG and BVP signal segments, with a tensor size of [missing information]. Output the local feature sequences of ECG and BVP, with a tensor size of [size missing]. D represents the local feature dimension. (Question:) Step 3: The multimodal fusion module generates a fused feature sequence.
[0033] The ECG local feature sequence and the BVP local feature sequence are input into the multimodal fusion module to model the complementary relationship between the ECG and BVP local feature sequences and perform adaptive feature fusion expression to obtain the fused feature sequence.
[0034] The preferred implementation scheme of the multimodal fusion module in this invention includes cross-attention enhancement and gating mechanism fusion. Its input consists of local feature sequences from ECG and BVP, with tensor size denoted as... Output the fused feature sequence, with tensor size denoted as . .
[0035] Step 4: The global temporal modeling module obtains the emotion classification and discrimination features.
[0036] After adding positional encoding information and classification tokens to the fused feature sequence, it is input into the global temporal modeling module to perform global temporal information fusion modeling to obtain a global feature sequence. The feature vector at the corresponding position of the classification token at the head of the sequence is used as the emotion classification discrimination feature.
[0037] The preferred implementation scheme of the global temporal modeling module in this invention combines feature sequences with convolutional positional encoding, and adds a classification token (CLS). The input is a Transformer encoder structure that fuses and aggregates temporal sequence feature information, outputting a global feature sequence with a tensor size of [size missing]. The size of the feature tensor at the corresponding position of the classification token is .
[0038] Step 5: The classifier outputs the emotion recognition prediction result.
[0039] Based on the aforementioned emotion classification features, the classifier maps to a low-dimensional emotion classification space through layer normalization, multiple linear layers, and activation functions, outputting unnormalized output values for each emotion category. Vector, tensor size is , For the number of categories, then through Normalization yields the emotion recognition prediction results.
[0040] Example 2 Based on Example 1, this example provides a method for multimodal physiological signal feature fusion and emotion recognition. The preprocessing and sliding window segmentation in step 1 are specifically as follows:
[0041] Preprocessing operations are performed on the raw ECG and raw BVP signals respectively. The preprocessing may include: DC component removal, bandpass filtering, outlier removal, baseline drift correction, resampling, and z-score normalization. DC component removal: DC component calculation is performed using the mean method; Bandpass filtering: Bandpass filter parameters are designed using a Butterworth filter, where the ECG passband can be selected from 0.5Hz to 50Hz, and the BVP passband can be selected from 0.5Hz to 20Hz; Outlier removal: Outlier detection is performed using threshold pruning, and outliers are replaced using interpolation; Baseline drift correction: Baseline correction is performed using wavelet decomposition to remove approximate components; Resampling: Used to unify the sampling rate, which is set to 256Hz; z-score normalization can be performed using the following formula:
[0042] in, The signal sample value within the window, , These represent the mean and standard deviation of the signal within the window, respectively. This is the standardized signal value.
[0043] Sliding window splitting.
[0044] The preprocessed ECG and BVP signals are synchronously segmented with the same window length and step size. Let the window length be... Step size is Then, synchronization fragment sequences are obtained from the preprocessed ECG and BVP. , ,in and Window segments corresponding to the same time position, The number of windows. In a preferred embodiment, the window length is used. 1 second, step size 0.5 seconds, number of windows It is 19.
[0045] Example 3 See Figure 2 Based on Example 1, this example provides a multimodal physiological signal feature fusion and emotion recognition method. Local feature sequences are extracted from ECG and BVP inputs using structurally identical but parameter-independent local feature perception modules. The input signal segment sequence is... Specifically, it includes the following steps: The waveform perception submodule extracts waveform features.
[0046] The waveform sensing submodule employs multi-scale one-dimensional convolution in parallel extraction to capture waveform morphology information at different scales. The convolution kernel size can utilize a parallel strategy of "small kernel + large kernel," where the small kernel is... The large core is The outputs of each branch are concatenated along the channel dimension and then channel attention, i.e., the SE module, is introduced to adaptively weight features at different scales in order to highlight waveform morphology information related to emotion.
[0047] The specific formula for the SE module is as follows:
[0048] in, The feature tensor after splicing the branches, This indicates element-wise multiplication. , For linear mapping, It is a non-linear activation. Adaptive global pooling is employed.
[0049] Phase sensing submodule The phase-aware submodule employs a parallel dilated convolution branch to capture rhythm / period-related phase information. This is achieved by injecting zero-value holes between the effective weights of the convolution kernel, allowing for the sensing of different frequencies and phase information. Multiple dilation rates are set for the convolution kernels. The kernel size can be selected as 9, and the SE channel attention is also introduced to enhance key phase features after the output of each branch is concatenated in the channel dimension.
[0050] Perceptual feature fusion submodule and fragment aggregation The perceptual feature fusion submodule employs multi-layer residual convolutional blocks for feature fusion, which can fuse waveform features and phase features to output a feature vector representation of the signal segment. Specifically, the waveform branch and phase branch outputs are concatenated in the channel dimension and then processed by a convolutional kernel. The convolutional layers perform channel fusion and compression, followed by feature fusion using five residual convolutional blocks. These residual convolutional blocks employ a bottleneck structure, containing convolutional layers, normalization layers, and a non-linear activation GELU. The second and fourth residual convolutional blocks expand the channel dimension and compress the feature dimension. Finally, global average pooling is performed on the fused fragment feature matrix in the time dimension to obtain fragment-level local feature vectors. This leads to the formation of local feature sequences. and .
[0051] In the convolution process, Instance Norm is used as the normalization layer.
[0052] Example 4 See Figure 3 This embodiment provides a method for multimodal physiological signal feature fusion and emotion recognition, wherein the input of the multimodal fusion module is an ECG local feature sequence. and BVP local feature sequences A multimodal information enhancement strategy employing cross-attention mechanism is adopted. This strategy extracts ECG-related information from the BVP (Browser Verification Portfolio) and injects it into the ECG representation for enhancement, and conversely, extracts BVP-related information from the ECG representation and injects it into the BVP representation for enhancement. Specifically, the strategy includes the following steps: Step 1, linear projection.
[0053] right and Linear projection is performed separately to enhance the representation, while the projection preserves the feature dimensions:
[0054] Step 2: Enhanced cross-attention mechanism.
[0055] ECG projection features Generate query vectors BVP projection features Generate key vectors Sum value vector Similarly, using BVP projection features Generate query vectors ECG projection features Generate key vectors Sum value vector :
[0056]
[0057] Then, use the scaling dot product attention formula. and The attention weight matrix is calculated. And using attention weight matrix right , The weighted auxiliary fusion features are obtained, and the specific formula is as follows:
[0058]
[0059] in, , This is the attention weight matrix. , To assist in fusing feature sequences, for The feature dimension is typically D.
[0060] Finally, the auxiliary feature sequence is scaled using learnable channels. , After scaling, the residual is compared with the projected features. , Adding them together yields the enhanced feature sequences of ECG and BVP. , :
[0061]
[0062] in, , The sequence channels representing the corresponding modalities can be learned scaling factors.
[0063] Step 3: Enhance the gating mechanism.
[0064] To further enhance the effectiveness of fusion, a gating mechanism is introduced to use enhanced feature sequences for adaptive fusion to control the strength of each modality in the fused feature representation. and After concatenation of the feature dimensions, a linear transformation is performed to reduce it to 1 dimension, and then... Shaped activation function Obtain the gating coefficient :
[0065] in, It is a non-linear activation. This indicates a feature dimension concatenation operation. The output fused feature sequence is obtained using the following formula. :
[0066] in, This indicates element-wise multiplication. A gating mechanism controls the intensity of enhancement information injection between the two modalities and highlights complementary information, ensuring that the fused features contain both ECG modal features and BVP modal information, ultimately outputting a fused feature sequence. .
[0067] Example 5 See Figure 4 Based on Example 1, this example provides a multimodal physiological signal feature fusion and emotion recognition method. The global temporal modeling module adopts a Transformer encoder structure as the global temporal modeling model scheme, and its input is the fused feature sequence. This method can fuse and aggregate time-series feature information. Specifically, it includes the following steps: Step 1, Convolution Position Encoding.
[0068] Positional encoding is obtained using convolutional positional encoding for the fused feature sequence. One-dimensional convolution is performed on the sequence dimension to obtain the position embedding, and then the position is fused with the feature sequence. Add:
[0069] The convolutional position encoding can employ grouped one-dimensional convolution and introduce weight normalization and the non-linear activation function GELU to improve training stability and expressive power.
[0070] Step 2: Concatenate the CLS token and input it into the Transformer encoder.
[0071] In sequence Prepend a learnable classification token (CLS):
[0072] Will The input consists of an M-layer Transformer encoder for global temporal information modeling. Each layer includes a multi-head self-attention mechanism and a feedforward connection network. GELU activation is used, along with residual connections and layer normalization. The Transformer output sequence is denoted as... Take the output vector corresponding to CLS:
[0073] Its tensor size is This serves as a feature for classifying emotions. (Emotion classification features) It can describe the vector representation of a sample in the emotion feature space, and thus serve as the input to a subsequent classifier for emotion recognition, classification, and prediction.
[0074] Example 6 This embodiment provides an emotion recognition system that integrates multimodal physiological signal features, including: The signal segmentation unit is used to segment the acquired original electrocardiogram (ECG) signal and original pulse signal of the subject to be tested, respectively, to obtain ECG signal segment sequences and pulse signal segment sequences; The feature extraction unit is used to extract features from the electrocardiogram signal segment sequence and the pulse signal segment sequence respectively, to obtain the local feature sequence of electrocardiogram and the local feature sequence of pulse. The feature fusion unit is used to fuse the local feature sequences of electrocardiogram and pulse to obtain a fused feature sequence; The temporal information fusion unit is used to perform global temporal information fusion on the obtained fusion feature sequence to obtain a global feature sequence; The emotion recognition unit is used to obtain emotion recognition prediction results based on the obtained global feature sequence.
[0075] Example 7 This embodiment also provides a computing device. The computing device includes a bus, a processor, a memory, and a communication interface. The processor, memory, and communication interface communicate with each other via the bus. The computing device can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memory in the computing device.
[0076] A bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, a bus can include a path for transmitting information between various components of a computing device (e.g., memory, processor, communication interfaces).
[0077] The processor may include any one or more of the following: central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), microprocessor (MP), or digital signal processor (DSP).
[0078] Memory can include volatile memory, such as random access memory (RAM). Processors can also include non-volatile memory. volatile memory, such as read-only memory (ROM). ROM (memory only), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0079] The memory stores executable program code, which the processor executes to implement the functions of the aforementioned units, thereby achieving, for example, the method described in Embodiment 1. That is, the memory may store instructions for the methods and functions relating to the computing device in any of the above embodiments.
[0080] The communication interface uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between computing devices and other devices or communication networks.
[0081] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for fusing multi-modal physiological signal features for emotion recognition, characterized in that, Includes the following steps: The original electrocardiogram (ECG) signal and the original pulse signal of the subject were segmented to obtain ECG signal segment sequences and pulse signal segment sequences, respectively. Feature extraction was performed on the electrocardiogram (ECG) signal segment sequence and the pulse signal segment sequence respectively to obtain the ECG local feature sequence and the pulse local feature sequence. The local feature sequences of electrocardiogram and pulse are fused to obtain a fused feature sequence; The obtained fused feature sequence is then fused with global temporal information to obtain a global feature sequence; The emotion recognition prediction result is obtained based on the obtained global feature sequence.
2. The emotion recognition method integrating multimodal physiological signal features according to claim 1, characterized in that, The original electrocardiogram (ECG) signal and the original pulse signal of the subject were segmented using a sliding window to obtain ECG signal segment sequences and pulse signal segment sequences.
3. The emotion recognition method integrating multimodal physiological signal features according to claim 1, characterized in that, The local feature perception module is used to extract features from the electrocardiogram signal segment sequence and the pulse signal segment sequence, respectively, to obtain the local feature sequence of electrocardiogram and the local feature sequence of pulse.
4. The emotion recognition method integrating multimodal physiological signal features according to claim 3, characterized in that, The local feature perception module includes a waveform perception submodule, a phase perception submodule, and a perception feature fusion submodule, wherein: The waveform sensing submodule is used to perform multi-scale one-dimensional convolution parallel extraction on electrocardiogram signal segment sequences or pulse signal segment sequences to obtain waveform features; The phase sensing submodule is used to perform parallel dilated convolution on the electrocardiogram signal segment sequence or pulse signal segment sequence to obtain phase features; The sensing feature fusion submodule is used to fuse the obtained waveform features and phase features to obtain an electrocardiogram local feature sequence or a pulse local feature sequence.
5. The emotion recognition method integrating multimodal physiological signal features according to claim 4, characterized in that, The waveform sensing submodule includes multiple parallel convolutional layers, and the outputs of the multiple convolutional layers are connected to the channel attention layer. The phase-aware submodule includes multiple parallel dilated convolutional layers, and the outputs of the multiple parallel dilated convolutional layers are connected to the channel attention layer. The perceptual feature fusion submodule includes a serial channel fusion convolutional layer, multiple residual convolutional blocks, and a feature aggregation pooling layer.
6. The emotion recognition method based on the fusion of multimodal physiological signal features according to claim 1, characterized in that, The multimodal fusion module is used to fuse the local feature sequences of electrocardiogram and pulse to obtain the fused feature sequence.
7. The emotion recognition method integrating multimodal physiological signal features according to claim 6, characterized in that, The multimodal fusion module is used to fuse local feature sequences of electrocardiogram and pulse to obtain a fused feature sequence. The specific method is as follows: Linear projections were performed on the local feature sequences of electrocardiogram and pulse respectively to obtain the electrocardiogram projection features and pulse projection features. Generate ECG query vectors using ECG projection features and generate ECG key vectors and ECG value vectors using pulse projection features; A pulse query vector is generated using pulse projection features, and a pulse key vector and pulse value vector are generated using electrocardiogram projection features. Based on the ECG query vector, ECG key vector, pulse query vector, and pulse key vector, the attention weight matrix corresponding to ECG and pulse is calculated. The ECG value vector and pulse value vector are weighted using the attention weight matrix to obtain ECG-assisted fusion features and pulse-assisted fusion features respectively; The ECG-assisted fusion features and pulse-assisted fusion features are scaled and then added to the corresponding ECG projection features and pulse projection features respectively using the residual method to obtain the ECG enhancement feature sequence and the pulse enhancement feature sequence. The ECG enhancement feature sequence and the pulse enhancement feature sequence are concatenated in feature dimension and then linearly transformed to one-dimensional features. One-dimensional features are processed The gating coefficients are obtained by using a shaped activation function; The ECG enhancement feature sequence and the pulse enhancement feature sequence are fused based on the gating coefficient to obtain the fused feature sequence.
8. The emotion recognition method integrating multimodal physiological signal features according to claim 1, characterized in that, The obtained fused feature sequence is then fused with global temporal information to obtain a global feature sequence. The specific method is as follows: The fused feature sequence is convolved in one dimension in the sequence dimension to obtain the position embedding; The obtained position embedding and fused feature sequences are added together to obtain a sequence; A learnable classification token is appended to the beginning of the obtained sequence to obtain the token sequence; By modeling the token sequence using global temporal information, we can obtain emotion classification and discrimination features.
9. An emotion recognition system integrating multimodal physiological signal features, characterized in that, include: The signal segmentation unit is used to segment the acquired original electrocardiogram (ECG) signal and original pulse signal of the subject to be tested, respectively, to obtain ECG signal segment sequences and pulse signal segment sequences; The feature extraction unit is used to extract features from the electrocardiogram signal segment sequence and the pulse signal segment sequence respectively, to obtain the local feature sequence of electrocardiogram and the local feature sequence of pulse. The feature fusion unit is used to fuse the local feature sequences of electrocardiogram and pulse to obtain a fused feature sequence; The temporal information fusion unit is used to perform global temporal information fusion on the obtained fusion feature sequence to obtain a global feature sequence; The emotion recognition unit is used to obtain emotion recognition prediction results based on the obtained global feature sequence.
10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 8.