Emotion recognition method based on cross fusion of multiple time-frequency graphs

Abstract

The invention relates to an emotion recognition method based on cross fusion of multiple time-frequency diagrams. Single-path electrocardiosignals are subjected to fine grit decomposition and converted into multi-dimensional time-frequency diagrams so as to construct a multi-view signal description system. Through multi-time-frequency graph fusion and a local-global feature cross fusion mechanism, information of signals under different time scales and frequency resolutions is dynamically integrated, so that complex dynamic states of rapid fluctuation and slow adjustment of the autonomic nervous system are comprehensively captured. Meanwhile, heart rate variability is used as key neuroscience priori knowledge and is directly embedded into a model input and feature learning process. In the aspect of feature extraction, a convolution block and a Transform block are crossed and fused to realize collaborative perception of local subtle changes and global function states, and deep features with higher robustness and higher discrimination are obtained with lower calculation cost.

Description

Technical Field

[0001] This invention belongs to the field of affective computing in human-computer interaction, specifically the application of artificial intelligence and deep learning in biomedical signal processing, focusing on extracting meaningful features from physiological signals such as electrocardiograms and establishing classification models. Background Technology

[0002] Currently, emotion recognition has become a research hotspot in fields such as affective computing and human-computer interaction, and will play an increasingly crucial role in the future to meet the urgent needs of various fields for accurate emotion recognition. Emotion recognition methods aim to capture and analyze the physiological reactions and behavioral expressions of individuals when faced with external stimuli. Although it is quite challenging for computers to directly understand human emotional fluctuations, by using advanced technologies such as machine vision and speech analysis, it is possible to effectively identify and analyze external expressions of emotion such as facial expressions and tone of voice. Unlike external expressions, physiological signals are not easily faked. By analyzing these signals, the emotional characteristics hidden within physiological signals can be revealed, better reflecting an individual's true emotional state.

[0003] Heart rate variability (HRV), as the gold standard for quantifying the precision of autonomic nervous system regulation of the heart, naturally becomes a key breakthrough in decoding emotional states. However, traditional HRV analysis relies on manually extracted time-domain and frequency-domain statistical features, which are often static and generalized, making it difficult to capture the subtle and transient rhythmic patterns in the dynamic changes of emotions.

[0004] The rise of deep learning, particularly convolutional neural networks (CNNs) and Transformers, has made it possible to learn multi-level, deep feature representations directly from raw or transformed signals. CNNs excel at extracting local textures, while Transformers excel at modeling global dependencies. Applying this idea to physiological signal analysis, researchers have begun to transform one-dimensional time-series signals into two-dimensional time-frequency maps, thus transforming the time-series analysis problem into an image analysis problem, fully leveraging the powerful representation learning capabilities of computer vision. Both the local instantaneous fluctuations and overall changes in emotion are of significant value. Therefore, there is an urgent need for a method that, within a unified framework, captures both the local detailed textures in time-frequency maps and understands their overall structure and long-range dependencies, thereby improving model performance and ultimately enabling emotion recognition. Summary of the Invention

[0005] The purpose of this invention is to design a neuroscience prior and data model that integrates electrocardiogram (ECG), propose a method for extracting ECG features that characterize fluctuations in the nervous system, establish an ECG signal emotion recognition model, and realize emotion recognition based on ECG signals.

[0006] The technical solution adopted by this invention to achieve the above objectives is: an emotion recognition method based on multi-time-frequency graph cross-fusion, comprising the following steps:

[0007] Extracting the intercardiac interval signal from the electrocardiogram signal: The original continuous electrocardiogram waveform signal is preprocessed to obtain the time sequence signal;

[0008] Signal decomposition and dynamic feature extraction: For time-series signals, a serial architecture of empirical mode decomposition and wavelet transform time-frequency analysis is adopted to transform them into two-dimensional time-frequency graphs that reflect the dynamic evolution of energy over time and different physiological mechanisms.

[0009] Multi-time-frequency graph feature extraction: Convolutional neural networks are used to extract local features within the two-dimensional time-frequency graph, while global features within the time-frequency graph are extracted using Transformer;

[0010] Feature cross-fusion: Through spatial dimension alignment, local features and global features are mapped to the same semantic space and fused to obtain a fused and enhanced feature map;

[0011] Classifier design: Based on the fused and enhanced feature maps, an adaptive multi-level pooling strategy is adopted to output the classification results.

[0012] The extraction of the cardiac interval signal from the electrocardiogram signal includes the following steps:

[0013] The ECG waveform signal was filtered, and the R-wave peak value in each QRS complex was detected to obtain a timestamp. N represents the total number of detected R waves; the time difference between adjacent R waves is calculated to obtain the cardiac interval sequence. ;

[0014] Use timestamps and Cubic spline interpolation is performed on a uniform time grid to obtain the cardiac interval timing signal.

[0015] The signal decomposition and dynamic feature extraction include the following steps:

[0016] (1) Empirical Mode Decomposition: The time series signal of the cardiac interval is decomposed by EMD to obtain the IMF component and residual;

[0017] (2) Select IMF components: Perform Hilbert transform on each IMF component, calculate its average frequency, and filter out the IMF components within the set average frequency range;

[0018] (3) Wavelet transform time-frequency analysis: Perform continuous wavelet transform on the selected IMF components, select complex Morlet wavelet as mother wavelet, and obtain the wavelet coefficient matrix that simultaneously encodes time-frequency energy and phase information by calculating the convolution of IMF components and wavelet basis functions at different scales.

[0019] (4) Generation of time-frequency spectrum: First, the wavelet coefficient matrix is ​​squared by taking the modulus and then converted into a scale map. Then, the time-frequency energy matrix is ​​constructed by the scale-frequency mapping relationship based on the mother wavelet parameters. Finally, the processed time-frequency energy matrix is ​​used as two-dimensional image data. With time as the horizontal axis and frequency as the vertical axis, a continuous color spectrum is used for mapping to generate a two-dimensional time-frequency spectrum.

[0020] The multi-time-frequency map feature extraction includes the following steps:

[0021] (1) Preprocessing and standardization: The two-dimensional time-frequency map is uniformly adjusted to a fixed size and normalized to obtain a standardized time-frequency map, which is used as input to the convolutional block branch; the time-frequency map is serialized and positionally encoded to obtain an embedded sequence containing the visual content and spatiotemporal structure information of the time-frequency map, which is used as input to the transformer branch;

[0022] (2) Convolutional block branch: The normalized time-frequency map is input to five sequential convolutional blocks to extract features at different scales and output local features; each convolutional block consists of a two-dimensional convolutional layer, a downsampling layer, a ReLU activation function and a spatial attention layer, while residual connections are used to alleviate gradient vanishing.

[0023] (3) Transformer branch: The input embedding sequence is processed through 6 layers of Transformer encoders for feature extraction, and finally the processed global features are output.

[0024] The feature cross-fusion includes the following steps:

[0025] First, through spatial dimension alignment, the local features extracted by the convolutional block branches and the global features captured by the transformer branches are mapped to the same semantic space, and then they are fused together to output a fused and enhanced feature map:

[0026] (1) Spatial dimension alignment: The vector of global features output by the transformer branch is expanded in spatial dimension to obtain the expanded global features. The local features extracted from the convolutional block branches are converted into two-dimensional matrices, and then linearly projected through convolution operations to map them to the same feature dimension as the global features, thus obtaining local features with the same dimension as the expanded global features. ;

[0027] (2) Bidirectional feature cross-attention mechanism:

[0028] Local-to-global cross-attention: global features Generate a global query vector through linear transformation. , local features Local key vectors are generated through two different linear transformations. and local value vector Then calculate the global query vector. With local key vectors The similarity score is used to obtain the attention score, and finally the attention score is used to apply the local value vector. Perform a weighted summation and update the global features;

[0029] Global-to-local cross-attention: local features Local query vectors are generated through linear transformation. global features Generate global key vectors using two different linear transformations. and global value vector Then calculate the local query vector. With global key vector The similarity score is used to obtain the attention score, and finally the attention score is used to apply the global value vector. Perform weighted summation and update local features;

[0030] The updated global features and updated local features are fused using weighted summation, and the result of feature fusion is output.

[0031] (3) Gated fusion: global features after input expansion Local features generated after processing local features of convolutional block branches A gated fusion strategy is used to adaptively integrate branch features and generate a unified deep feature representation as the gated fusion feature.

[0032] (4) Feature fusion and processing: The feature fusion result is then cross-fused with the gated fusion feature. Layer normalization and GELU activation function are applied to the fused feature to output the enhanced feature map.

[0033] The classifier design includes the following steps:

[0034] ① For the enhanced feature map, features are extracted step by step through three layers of convolution to obtain a 128-channel feature map;

[0035] ② Pooling Processing: Input a 128-channel feature map and employ a spatial pyramid pooling hierarchical gridding strategy to perform pooling operations on the feature map at multiple scales; simultaneously, attention pooling is used to redistribute the importance of spatial locations through learnable weights, outputting a one-dimensional feature vector.

[0036] ③ Fully connected layer: Input a one-dimensional feature vector and use two fully connected layers to complete the classification decision.

[0037] An emotion recognition system based on multi-time-frequency graph cross-fusion includes:

[0038] The ECG signal preprocessing module is used to preprocess the original continuous ECG waveform signal to obtain a time-series signal;

[0039] The signal decomposition and dynamic feature extraction module is used to transform time-series signals into two-dimensional time-frequency graphs that reflect the dynamic evolution of energy over time and different physiological mechanisms, using a serial architecture of empirical mode decomposition and wavelet transform time-frequency analysis.

[0040] The multi-time-frequency graph feature extraction module is used to extract local features within a two-dimensional time-frequency graph using a convolutional neural network, while simultaneously extracting global features within the time-frequency graph using a Transformer.

[0041] The feature cross-fusion module is used to map local and global features to the same semantic space through spatial dimension alignment operations, and then fuse them to obtain a fused and enhanced feature map.

[0042] The classifier is used to output classification results based on the fused and enhanced feature maps using an adaptive multi-level pooling strategy.

[0043] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned emotion recognition method based on multi-time-frequency graph cross-fusion.

[0044] The present invention has the following beneficial effects and advantages:

[0045] 1. By combining EMD with wavelet transform, it is possible to adaptively capture full-scale rhythmic information ranging from millisecond-level instantaneous fluctuations to minute-level trend changes, overcoming the limitations of fixed time windows in traditional methods.

[0046] 2. Multi-perspective information fusion: Generate and fuse time-frequency maps of multiple physiological frequency bands (HF, LF, etc.), which is equivalent to observing heart rhythm from multiple "physiological perspectives" in a coordinated manner, avoiding the omission of information from a single perspective and constructing a more complete emotional physiological fingerprint.

[0047] 3. By constructing a cross-fusion architecture of CNN and Transformer, we can simultaneously capture the subtle local change patterns and overall structural dependencies in the time-frequency graph, thus achieving a deep understanding of complex physiological patterns.

[0048] 4. The refined pooling strategy and multi-level regularization design force the model to learn more robust and essential feature representations, rather than random noise in the training data, so that the model performs more stably when faced with individual differences, different acquisition devices or environmental changes. Attached Figure Description

[0049] Figure 1 Main flowchart of the technical solution of this invention;

[0050] Figure 2 Flowchart of the generation of time-frequency graphs in this invention;

[0051] Figure 3 The feature fusion and processing flowchart of the present invention. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0053] This invention relates to an emotion recognition method based on multi-time-frequency graph cross-fusion. Addressing the problem that current traditional linear and nonlinear features contain limited information about nervous system fluctuations, resulting in an inability to comprehensively reflect emotional states, this project aims to integrate neuroscience prior knowledge of electrocardiograms (ECG) with deep learning models to propose an emotion recognition method based on multi-time-frequency graph cross-fusion. This method overcomes the limitations of traditional representation techniques for the nonlinear coupling relationship between the sympathetic and parasympathetic nervous systems, achieving intelligent representation and emotion recognition of ECG signals. The method involves fine-grained decomposition and conversion of a single ECG signal into a multi-dimensional time-frequency graph to construct a multi-perspective signal description system. Through multi-time-frequency graph fusion and a local-global feature cross-fusion mechanism, information from signals at different time scales and frequency resolutions is dynamically integrated, thereby comprehensively capturing the complex dynamics of rapid fluctuations and slow adjustments in the autonomic nervous system. Simultaneously, heart rate variability is used as key neuroscience prior knowledge, directly embedded into the model input and feature learning process. In feature extraction, convolutional blocks and Transformer blocks are cross-fused to achieve collaborative perception of subtle local changes and global functional states, obtaining more robust and discriminative deep features with lower computational cost.

[0054] An emotion recognition method based on multi-time-frequency graph cross-fusion includes the following steps:

[0055] Extracting the intercardiac interval signal from the electrocardiogram signal: transforming the original, continuous electrocardiogram waveform into core physiological indicators that can quantitatively reflect the dynamic balance of the autonomic nervous system.

[0056] Signal decomposition and dynamic feature extraction: A serial architecture of empirical mode decomposition and wavelet transform time-frequency analysis is adopted to decompose, filter and visualize the original rhythmic fluctuations from bottom to top in order to extract dynamic features that can be directly mapped to different autonomic neural regulation mechanisms.

[0057] Multi-time-frequency map feature extraction: The processed time-frequency map is used as the input for the feature extraction part. The dynamic correlation between local details and global context is calculated to obtain branch features.

[0058] Feature cross-fusion: Features extracted from the two pathways are deeply interacted and integrated, forming a comprehensive and in-depth representation of the ECG time-frequency dynamics, laying a solid foundation for accurate emotion recognition.

[0059] Classifier design: An adaptive multi-level pooling strategy is adopted, and progressive feature compression and information preservation are performed. Finally, the classification result is output by a fully connected layer.

[0060] Extracting the cardiac interval signal from the electrocardiogram signal:

[0061] The sequential fluctuations of the heart rate variability (HRV) signal are entirely driven by the modulation of the sinoatrial node by the autonomic nervous system (sympathetic and parasympathetic nervous systems). The intercardiac interval signal, a series of time intervals measured in milliseconds, is the sole and essential input data for calculating HRV. Without a high-precision, reliable intercardiac interval sequence, reliable HRV analysis is impossible.

[0062] The detailed steps are as follows:

[0063] (1) Use a filter with a passband frequency of about 5-15 Hz, then differentiate the filtered signal to highlight the fast-changing part of the signal, square it to amplify the R wave and suppress other waveform components.

[0064] (2) The Pan-Tompkins algorithm was used to detect the R-wave peak value in each QRS group, and a series of timestamps were obtained. N represents the total number of R waves detected.

[0065] (3) Calculate the time difference between adjacent R waves to obtain the heartbeat interval sequence (RR interval sequence). .

[0066] (4) Set a threshold based on common sense in physiology and retain only the interval between 300ms and 2000ms to remove artifacts.

[0067] (5) Use the original R-wave timestamp and Cubic spline interpolation is performed on a uniform time grid to obtain a continuous, smooth, and uniform cardiac interval timing signal.

[0068] Signal decomposition and dynamic feature extraction:

[0069] The purpose of signal decomposition and dynamic feature extraction is to transform the above-mentioned cardiac intermittent time sequence signal into a two-dimensional time-frequency spectrum that reflects the dynamic evolution of energy over time and different physiological mechanisms.

[0070] The specific method is as follows:

[0071] (1) Empirical Mode Decomposition (EMD): The time series signal of the heartbeat interval is decomposed by EMD to obtain a series of intrinsic mode function (IMF) components and a residual.

[0072] (2) Selecting IMF components: Perform Hilbert transform on each IMF to calculate its average frequency. Then select IMF components based on the classical physiological frequency band, i.e., screen out IMFs with an average frequency in the range of 0.15 Hz to 0.40 Hz and IMFs with an average frequency in the range of 0.04 Hz to 0.15 Hz.

[0073] (3) Wavelet transform time-frequency analysis: Perform continuous wavelet transform on the selected IMF components, select complex Morlet wavelet as mother wavelet, and obtain the wavelet coefficient matrix that simultaneously encodes time-frequency energy and phase information by calculating the convolution of IMF components and wavelet basis functions at different scales.

[0074] (4) Generation of time-frequency spectrum: First, the wavelet coefficient matrix is ​​squared by taking the modulus and then converted into a scale map. Then, the time-frequency energy matrix is ​​constructed by the scale-frequency mapping relationship based on the mother wavelet parameters. Finally, the processed time-frequency energy matrix is ​​used as two-dimensional image data. With time as the horizontal axis and frequency as the vertical axis, a continuous color spectrum is used for mapping and rendering to generate a high-resolution two-dimensional time-frequency map.

[0075] Multi-time-frequency graph feature extraction:

[0076] After obtaining the time-frequency maps, a convolutional neural network is used to extract local neighborhood features within each time-frequency map, while a Transformer is used to extract global dependency features within the time-frequency maps.

[0077] The specific method is as follows:

[0078] (1) Preprocessing and standardization: Before inputting the convolutional block, each time-frequency map is uniformly adjusted to a fixed size and normalized to standardize the pixel value range to the [0,1] interval, and the output is a standardized time-frequency map. Before inputting the transformer block, the time-frequency map is serialized and positionally encoded to obtain an embedded sequence containing the visual content and spatiotemporal structure information of the time-frequency map.

[0079] (2) Convolutional Block Branching: Five convolutional blocks are used to sequentially extract features at different scales from the input standardized time-frequency map, and finally the extracted feature map is output. Each convolutional block consists of two dimensions. It consists of convolutional layers, downsampling layers, ReLU activation functions, and spatial attention layers, while residual connections are used to alleviate gradient vanishing and promote deep network training.

[0080] (3) Transformer Branch: The input embedded sequence is processed through 6 layers of Transformer encoders for feature extraction, and finally outputs the processed global features. Each Transformer encoder consists of layer normalization, multi-head attention, residual connections, layer normalization, feedforward network, and residual connections. Finally, the global context information of the entire sequence is aggregated to form a highly abstract global feature that contains the complex relationships of all elements within the entire time-frequency graph.

[0081] Feature cross-fusion:

[0082] The module first uses spatial dimension alignment to map the feature maps extracted by the convolutional blocks, which contain rich details and textures, and the global features captured by the Transformer, which carry the overall structure and long-range dependencies, to the same semantic space. Then, it fuses these features to output a fused and enhanced feature map. The specific method is as follows:

[0083] (1) Spatial dimension alignment: The vector of global features output by Transformer is expanded in spatial dimension to obtain the expanded global features. The local features extracted from the convolutional block branches are transformed into a two-dimensional matrix. This matrix is ​​then linearly projected using a learnable 1×1 convolution operation, mapping it to the same feature dimension as the global features. This results in local features with the same dimension as the expanded global features. .

[0084] (2) Bidirectional feature cross-attention mechanism:

[0085] Local-to-global cross-attention: global features from the Transformer output Generate query vectors These are the learnable parameters of the model, used to map global features to the query space; from local features... Generate key-value pairs = = ,in , These are the learnable parameters of the model, used to map local features to the key and value spaces; attention calculation: ,in This is the scaling factor.

[0086] Global-to-local cross-attention: generating query vectors from global features These are the learnable parameters of the model, used to map local features to the query space; generating key-value pairs from local features. = = ,in , These are the learnable parameters of the model, used to incorporate global features. Mapping to key and value spaces; attention computation: .

[0087] The output features of the two bidirectional features are fused using weighted summation, and the result of the output feature fusion is obtained.

[0088] (3) Gated fusion: Input the global features generated by the Transformer branch and the local features generated by the CNN branch, and adopt the gated fusion strategy to adaptively integrate the branch features to generate a unified deep feature representation with both detail resolution and overall consistency, and output the gated fusion feature.

[0089] (4) Feature Fusion and Processing: The feature fusion result is then cross-fused with the gating fusion result. Layer normalization and GELU activation function are applied to the fused features to stabilize the training process. This increases the feature representation capability, and finally, the enhanced feature map is output.

[0090] Classifier Design:

[0091] The specific steps are as follows:

[0092] (1) Features are extracted step by step from low level to high level through three convolutional layers. The input feature map is passed through the first convolutional block to obtain a 32-channel half-size feature map, then through the second convolutional block to obtain a 64-channel quarter-size feature map, and then through the third convolutional block to obtain a 128-channel feature map.

[0093] (2) Pooling: The feature map after the input convolution block is processed. The spatial pyramid pooling layered grid strategy is adopted to perform pooling operations on the feature map at multiple scales. At the same time, attention pooling is used to redistribute the importance of spatial positions through learnable weights and output a one-dimensional feature vector.

[0094] Fully connected layer: Input a one-dimensional feature vector and use two fully connected layers to complete the classification decision.

[0095] Figure 1 This is the main flowchart of the technical solution of the present invention, which includes the following steps:

[0096] (1) Extracting heart rate variability data: By filtering the signal and detecting the R wave, the time interval between consecutive heartbeats is accurately calculated to form a heartbeat interval sequence, which is the direct data source for heart rate variability analysis.

[0097] (2) Fine-grained decomposition: The non-stationary intercardia sequence is input into the Empirical Mode Decomposition (EMD) algorithm. Through an adaptive sieving process, the original signal is decomposed into a series of intrinsic mode functions (IMFs) arranged from high frequency to low frequency.

[0098] (3) Select the key IMF components. Apply continuous wavelet transform to each selected IMF component to convert the one-dimensional time-series oscillation into a two-dimensional time-frequency energy spectrum.

[0099] (4) Preliminary local feature extraction: The generated time-frequency map is input into a convolutional neural network (CNN) to effectively capture the texture, edge and energy aggregation patterns in the local area.

[0100] (5) Preliminary global feature extraction: The time-frequency map is divided into a sequence of image blocks and input into the Transformer encoder to capture the long-range dependencies and global structural patterns within the entire time-frequency map.

[0101] (6) Feature fusion: The feature cross-fusion module enables bidirectional interaction and deep fusion of local and global features.

[0102] (7) Classifier: Emotion classification is completed through pooling layer, fully connected layer and output layer.

[0103] Figure 2 The flowchart for generating the time-frequency graph of this invention is as follows:

[0104] (1) Perform frequency domain filtering on the original signal using a filter with a passband frequency of approximately 5-15 Hz.

[0105] (2) Pan-Tompkins algorithm for detecting R-wave peaks: The signal is differentiated, squared, and integrated through a sliding window to enhance the steepness of the R-wave and smooth the waveform. Then, the highest and sharpest R-wave peaks in each QRS complex are identified and located by using adaptive thresholds and refractory period rules, and their corresponding timestamps are recorded.

[0106] (3) Calculate the time difference between adjacent R waves to obtain the original heartbeat interval sequence.

[0107] (4) Outlier removal: Based on common sense in physiology, a threshold is set to retain only the interval between 300ms and 2000ms to remove artifacts.

[0108] (5) Spline interpolation: On a uniform time grid, a continuous signal curve is reconstructed using cubic spline interpolation. This step outputs a clean, continuous, and uniformly sampled time series of heart rate intervals.

[0109] (6) Empirical Mode Decomposition: Local extrema are identified through an iterative sieving process, upper and lower envelopes are constructed, the mean is calculated, and oscillation modes are extracted. The original signal is adaptively decomposed into a series of intrinsic mode functions (IMFs) from high frequency to low frequency.

[0110] (7) Select relevant IMF components: By calculating the average instantaneous frequency of each IMF, it is mapped to the classic heart rate variability physiological frequency band, and the oscillating components related to the physiological mechanism are screened.

[0111] (8) Time-frequency analysis using wavelet transform: Perform continuous wavelet transform on each selected physiologically relevant IMF component. The wavelet coefficient matrix is ​​calculated by shifting the wavelet function along the time axis at different scales (corresponding to different frequencies) and convolving it with the signal.

[0112] (9) Generate time-frequency spectrum: Take the modulus of the wavelet coefficient matrix and square it to obtain the time-frequency energy density spectrum. Plot a two-dimensional time-frequency spectrum with time as the horizontal axis and frequency or period as the vertical axis by mapping energy.

[0113] Figure 3 The flowchart illustrating the feature fusion and processing in this invention is shown below, with detailed steps as follows:

[0114] (1) Parallel processing of independent branches: The local features are reshaped into 2D matrices and then processed by convolutional mapping. The global feature vector is expanded so that the features of the two branches are fully aligned in space and feature dimension.

[0115] (2) Information Interaction: The feature cross-fusion module enables bidirectional interaction and deep fusion of local and global features. Interaction is achieved through a bidirectional cross-attention mechanism: global features are used as queries to retrieve relevant details from local features; local features are used as queries to obtain context from global features; finally, a weighted summation is used to fuse the output features from the bidirectional feature cross-fusion, outputting the feature fusion result. Simultaneously, strategies such as gating fusion are used to adaptively integrate branch features, generating a unified deep feature representation that combines detail resolution with overall consistency.

[0116] (3) Progressive integration: The result of feature fusion is then cross-fused with the result of gating fusion. The fused features are then normalized, residuals are connected, and the GELU activation function is applied.

[0117] (4) Output features: After the above processing, the output features are enhanced by deep cross-fusion and sent to the subsequent classifier for emotion classification and recognition.

Claims

1. An emotion recognition method based on multi-time-frequency graph cross-fusion, characterized in that, Includes the following steps: Extracting the intercardiac interval signal from the electrocardiogram signal: The original continuous electrocardiogram waveform signal is preprocessed to obtain the time sequence signal; Signal decomposition and dynamic feature extraction: For time-series signals, a serial architecture of empirical mode decomposition and wavelet transform time-frequency analysis is adopted to transform them into two-dimensional time-frequency graphs that reflect the dynamic evolution of energy over time and different physiological mechanisms. Multi-time-frequency graph feature extraction: Convolutional neural networks are used to extract local features within the two-dimensional time-frequency graph, while global features within the time-frequency graph are extracted using Transformer; Feature cross-fusion: Through spatial dimension alignment, local features and global features are mapped to the same semantic space and fused to obtain a fused and enhanced feature map; Classifier design: Based on the fused and enhanced feature maps, an adaptive multi-level pooling strategy is adopted to output the classification results.

2. The emotion recognition method based on multi-time-frequency graph cross-fusion according to claim 1, characterized in that, The extraction of the cardiac interval signal from the electrocardiogram signal includes the following steps: The ECG waveform signal was filtered, and the R-wave peak value in each QRS complex was detected to obtain a timestamp. N represents the total number of detected R waves; the time difference between adjacent R waves is calculated to obtain the cardiac interval sequence. ; Use timestamps and Cubic spline interpolation is performed on a uniform time grid to obtain the cardiac interval timing signal.

3. The emotion recognition method based on multi-time-frequency graph cross-fusion according to claim 1, characterized in that, The signal decomposition and dynamic feature extraction include the following steps: (1) Empirical Mode Decomposition: The time series signal of the cardiac interval is decomposed by EMD to obtain the IMF component and residual; (2) Select IMF components: Perform Hilbert transform on each IMF component, calculate its average frequency, and filter out the IMF components within the set average frequency range; (3) Wavelet transform time-frequency analysis: Perform continuous wavelet transform on the selected IMF components, select complex Morlet wavelet as mother wavelet, and obtain the wavelet coefficient matrix that simultaneously encodes time-frequency energy and phase information by calculating the convolution of IMF components and wavelet basis functions at different scales. (4) Generation of time-frequency spectrum: First, the wavelet coefficient matrix is ​​squared by taking the modulus and then converted into a scale map. Then, the time-frequency energy matrix is ​​constructed by the scale-frequency mapping relationship based on the mother wavelet parameters. Finally, the processed time-frequency energy matrix is ​​used as two-dimensional image data. With time as the horizontal axis and frequency as the vertical axis, a continuous color spectrum is used for mapping to generate a two-dimensional time-frequency spectrum.

4. The emotion recognition method based on multi-time-frequency graph cross-fusion according to claim 1, characterized in that, The multi-time-frequency map feature extraction includes the following steps: (1) Preprocessing and standardization: The two-dimensional time-frequency map is uniformly adjusted to a fixed size and normalized to obtain a standardized time-frequency map, which is used as input to the convolutional block branch; the time-frequency map is serialized and positionally encoded to obtain an embedded sequence containing the visual content and spatiotemporal structure information of the time-frequency map, which is used as input to the transformer branch; (2) Convolutional block branch: The normalized time-frequency map is input to five sequential convolutional blocks to extract features at different scales and output local features; each convolutional block consists of a two-dimensional convolutional layer, a downsampling layer, a ReLU activation function and a spatial attention layer, while residual connections are used to alleviate gradient vanishing. (3) Transformer branch: The input embedding sequence is processed through 6 layers of Transformer encoders for feature extraction, and finally the processed global features are output.

5. The emotion recognition method based on multi-time-frequency graph cross-fusion according to claim 1, characterized in that, The features are cross-fused. Includes the following steps: First, through spatial dimension alignment, the local features extracted by the convolutional block branches and the global features captured by the transformer branches are mapped to the same semantic space, and then they are fused together to output a fused and enhanced feature map: (1) Spatial dimension alignment: The vector of global features output by the transformer branch is expanded in spatial dimension to obtain the expanded global features. The local features extracted from the convolutional block branches are converted into two-dimensional matrices, and then linearly projected through convolution operations to map them to the same feature dimension as the global features, thus obtaining local features with the same dimension as the expanded global features. ; (2) Bidirectional feature cross-attention mechanism: Local-to-global cross-attention: global features Generate a global query vector through linear transformation. , local features Local key vectors are generated through two different linear transformations. and local value vector Then calculate the global query vector. With local key vectors The similarity score is used to obtain the attention score, and finally the attention score is used to apply the local value vector. Perform a weighted summation and update the global features; Global-to-local cross-attention: local features Local query vectors are generated through linear transformation. global features Generate global key vectors using two different linear transformations. and global value vector Then calculate the local query vector. With global key vector The similarity score is used to obtain the attention score, and finally the attention score is used to apply the global value vector. Perform weighted summation and update local features; The updated global features and updated local features are fused using weighted summation, and the result of feature fusion is output. (3) Gated fusion: global features after input expansion Local features generated after processing local features of convolutional block branches A gated fusion strategy is used to adaptively integrate branch features and generate a unified deep feature representation as the gated fusion feature. (4) Feature fusion and processing: The feature fusion result is then cross-fused with the gated fusion feature. Layer normalization and GELU activation function are applied to the fused feature to output the enhanced feature map.

6. The emotion recognition method based on multi-time-frequency graph cross-fusion according to claim 1, characterized in that, The classifier design includes the following steps: ① For the enhanced feature map, features are extracted step by step through three layers of convolution to obtain a 128-channel feature map; ② Pooling Processing: Input a 128-channel feature map and employ a spatial pyramid pooling hierarchical gridding strategy to perform pooling operations on the feature map at multiple scales; simultaneously, attention pooling is used to redistribute the importance of spatial locations through learnable weights, outputting a one-dimensional feature vector. ③ Fully connected layer: Input a one-dimensional feature vector and use two fully connected layers to complete the classification decision.

7. An emotion recognition system based on multi-time-frequency graph cross-fusion, characterized in that, include: The ECG signal preprocessing module is used to preprocess the original continuous ECG waveform signal to obtain a time-series signal; The signal decomposition and dynamic feature extraction module is used to transform time-series signals into two-dimensional time-frequency graphs that reflect the dynamic evolution of energy over time and different physiological mechanisms, using a serial architecture of empirical mode decomposition and wavelet transform time-frequency analysis. The multi-time-frequency graph feature extraction module is used to extract local features within a two-dimensional time-frequency graph using a convolutional neural network, while simultaneously extracting global features within the time-frequency graph using a Transformer. The feature cross-fusion module is used to map local and global features to the same semantic space through spatial dimension alignment operations, and then fuse them to obtain a fused and enhanced feature map. The classifier is used to output classification results based on the fused and enhanced feature maps using an adaptive multi-level pooling strategy.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements an emotion recognition method based on multi-time-frequency graph cross-fusion as described in any one of claims 1-6.

Images