Emotion recognition method and device based on electrocardiosignal and respiratory signal multi-modal feature extraction and medium

Abstract

The invention discloses an emotion recognition method and device based on electrocardiosignal and respiratory signal multi-mode feature extraction and a medium, and relates to the field of emotion recognition, the method comprises the steps that an electrocardiosignal and a respiratory signal are denoised and positioned, and a denoised electrocardiosignal, a denoised respiratory signal, an RR interval sequence and a BB interval sequence are obtained; performing alignment and feature extraction on the RR interval sequence and the BB interval sequence to obtain heart-lung cooperation features, inputting the heart-lung cooperation features to the trained convolutional bidirectional attention network, and outputting heart-lung cooperation feature vectors; processing the denoised electrocardiosignals and the denoised respiratory signals to obtain electrocardiosignal time-frequency diagrams and respiratory signal time-frequency diagrams, inputting the electrocardiosignal time-frequency diagrams and the respiratory signal time-frequency diagrams into the trained multi-scale space and frequency domain enhancement network, and outputting electrocardiosignal feature vectors and respiratory feature vectors; the electrocardio feature vector and the respiration feature vector are fused firstly and then pass through a full-connection classifier with the cardiopulmonary collaborative feature vector, and the emotional state is obtained. According to the invention, the accuracy of emotion recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, in particular to an emotion recognition method based on multi-modal feature extraction of electrocardiogram signals and respiratory signals, a device and a medium. BACKGROUND

[0002] In recent years, with the rapid development of human-computer interaction, intelligent health monitoring and emotion recognition, physiological signals, as important information carriers reflecting the activity of the autonomic nervous system of the human body, have attracted more and more attention from researchers. Electrocardiogram signals and respiratory signals, as two typical physiological signals, can sensitively reflect the autonomic nervous regulation changes of individuals in different emotional states, and thus have important application value in emotion recognition tasks.

[0003] In some cases, the emotion recognition method still has obvious deficiencies in deep feature modeling: on the one hand, for the RR interval and BB interval and other interval features, the usual method is to extract statistical indicators and then directly input them into the classifier for discrimination, which stays at the level of shallow utilization or simple splicing, lacks modeling of the correlation and global dependence between feature dimensions by deep networks, and is difficult to reveal the complex dynamic patterns of the autonomic nervous system in different emotional states; on the other hand, the network used for time-frequency graph feature extraction is mostly a general traditional convolutional neural network structure (such as VGG16, ResNet), which lacks a multi-scale mechanism and attention regulation in the spatial domain, making it difficult to capture both the transient texture of the electrocardiogram signal and the low-frequency rhythm of the respiratory signal. In addition, these methods often ignore the domain modeling, lack adaptive enhancement of key frequency band components, and are easily disturbed by redundant frequency bands. The above limitations make it difficult for existing models to fully extract and highlight the key information highly related to emotion discrimination, limiting the further improvement of recognition performance and generalization ability. SUMMARY

[0004] The purpose of the present application is to provide an emotion recognition method based on multi-modal feature extraction of electrocardiogram signals and respiratory signals, which can improve the accuracy of emotion recognition.

[0005] To achieve the above purpose, the present application provides the following solutions.

[0006] In a first aspect, the present application provides an emotion recognition method based on multi-modal feature extraction of electrocardiogram signals and respiratory signals, comprising: denoising and positioning the electrocardiogram signal and the respiratory signal respectively to obtain a denoised electrocardiogram signal, a denoised respiratory signal, an RR interval sequence and a BB interval sequence; The RR interval sequence and the BB interval sequence were aligned using cubic spline interpolation, and features were extracted from the aligned RR interval sequence and BB interval sequence to obtain the cardiopulmonary coordination features of the RR interval sequence and BB interval sequence. The cardiopulmonary coordination features are input into a trained convolutional bidirectional attention network, which outputs a cardiopulmonary coordination feature vector. The convolutional bidirectional attention network includes a convolutional layer, a bidirectional long short-term memory layer, and an attention mechanism layer connected in sequence. The denoised ECG signal and the denoised respiratory signal were respectively downsampled, wavelet transformed and frequency clipped to obtain the time-frequency graph of the ECG signal and the time-frequency graph of the respiratory signal. The time-frequency plots of the electrocardiogram (ECG) signal and the respiratory signal are respectively input into a trained multi-scale spatial and frequency domain enhancement network, which outputs ECG feature vectors and respiratory feature vectors. The multi-scale spatial and frequency domain enhancement network includes multiple multi-scale spatial-frequency domain attention blocks. Each multi-scale spatial-frequency domain attention block includes a multi-scale spatial pyramid attention module and a frequency domain attention module. The multi-scale spatial pyramid attention module includes a hierarchical phantom convolution module and a spatial pyramid recalibration module. The electrocardiogram feature vector and the respiratory feature vector are fused to obtain a fused feature vector; The cardiopulmonary coordination feature vector and the fused feature vector are input into a fully connected classifier to obtain the emotional state.

[0007] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals.

[0008] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals.

[0009] According to the specific embodiments provided in this application, this application has the following technical effects: This application generates ECG and respiratory feature vectors through a trained multi-scale spatial and frequency domain augmentation network, and fuses them to obtain a fused feature vector. Simultaneously, a trained convolutional bidirectional attention network generates a cardiopulmonary coordination feature vector. The cardiopulmonary coordination feature vector and the fused feature vector are then passed through a fully connected classifier to obtain the emotional state. The entire emotional state recognition process, through the collaborative application of the multi-scale spatial and frequency domain augmentation network and the convolutional bidirectional attention network, achieves multi-level, multi-scale, and cross-dimensional fine modeling of ECG and respiratory signals, thereby obtaining accurate, efficient, and robust emotional state recognition performance. For the multi-scale spatial and frequency domain enhancement network, the time-frequency map is segmented in the spatial domain by the hierarchical phantom convolution module in the multi-scale spatial pyramid attention module, and recalibrated with the spatial pyramid recalibration module in the multi-scale spatial pyramid attention module through context awareness. This achieves adaptive enhancement of multi-scale features in the spatial domain. Combined with the frequency domain attention module, it can amplify key frequency bands related to emotions and suppress redundant components in the frequency domain, thus forming a complementary enhancement mechanism between the spatial and frequency domains, significantly improving the representation quality of ECG and respiratory feature vectors. For the convolutional bidirectional attention network, the convolutional layer is used to mine the local correlation between feature dimensions, and the bidirectional long short-term memory layer is combined to model cross-dimensional global dependencies. The attention mechanism adaptively emphasizes key dimensions, breaking through the shallow approach of simply splicing features in the traditional approach. This significantly improves the representation quality of cardiopulmonary coordination feature vectors, thereby ensuring the accuracy of emotional state recognition. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating an emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals, provided as an embodiment of this application.

[0012] Figure 2 This is a detailed flowchart illustrating an emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals, provided as an embodiment of this application.

[0013] Figure 3 This is a schematic diagram of the structure of a convolutional bidirectional attention network provided in an embodiment of this application.

[0014] Figure 4 This is a schematic diagram of a multi-scale spatial and frequency domain enhancement network structure provided in an embodiment of this application.

[0015] Figure 5 This is a schematic diagram of a multi-scale spatial-frequency domain attention block structure provided in an embodiment of this application.

[0016] Figure 6 This is a schematic diagram of a multi-scale spatial pyramid attention module structure provided in an embodiment of this application.

[0017] Figure 7 This is a schematic diagram of the structure of a layered phantom convolution module provided in an embodiment of this application.

[0018] Figure 8 This is a schematic diagram of the spatial pyramid recalibration module structure provided in one embodiment of this application.

[0019] Figure 9 This is a schematic diagram of the frequency domain attention module structure provided in an embodiment of this application.

[0020] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] In one exemplary embodiment, such as Figure 1 and Figure 2 As shown, an emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals is provided. The method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is described using a server as an example, including the following steps S1 to S8.

[0024] Step S1: Denoise and locate the ECG and respiratory signals respectively to obtain the denoised ECG signal, denoised respiratory signal, RR interval sequence and BB interval sequence.

[0025] Step S2: The RR interval sequence and the BB interval sequence are aligned using cubic spline interpolation, and features are extracted from the aligned RR interval sequence and the BB interval sequence to obtain the cardiopulmonary coordination features of the RR interval sequence and the BB interval sequence.

[0026] Step S3: Input the cardiopulmonary coordination features into the trained convolutional bidirectional attention network and output the cardiopulmonary coordination feature vector; the convolutional bidirectional attention network includes a convolutional layer, a bidirectional long short-term memory layer and an attention mechanism layer connected in sequence.

[0027] Step S4: Perform downsampling and wavelet transform on the denoised ECG signal and the denoised respiratory signal respectively to obtain the time-frequency graph of the ECG signal and the time-frequency graph of the respiratory signal.

[0028] Step S5: Input the time-frequency plots of the ECG signal and the respiratory signal into the trained multi-scale spatial and frequency domain enhancement network, and output the ECG feature vector and the respiratory feature vector; the multi-scale spatial and frequency domain enhancement network includes multiple multi-scale spatial-frequency domain attention blocks; the multi-scale spatial-frequency domain attention blocks include a multi-scale spatial pyramid attention module and a frequency domain attention module; the multi-scale spatial pyramid attention module includes: a hierarchical phantom convolution module and a spatial pyramid recalibration module.

[0029] Step S6: Fuse the ECG feature vector and the respiratory feature vector to obtain the fused feature vector.

[0030] Step S7: Input the cardiopulmonary coordination feature vector and the fused feature vector into the fully connected classifier to obtain the emotional state.

[0031] Further, step S1 includes: steps S11 to S15: Step S11: Use a third-order Butterworth bandpass filter to denoise the ECG signal to obtain the denoised ECG signal.

[0032] Step S12: Use a fourth-order Butterworth low-pass filter to denoise the respiratory signal to obtain the denoised respiratory signal.

[0033] Step S13: The Pan-Tompkins detection algorithm is used to locate the denoised ECG signal and obtain the R wave.

[0034] Step S14: Use the local maximum detection algorithm to locate the denoised respiratory signal and obtain the peak position of the respiration.

[0035] Step S15: Calculate the RR interval sequence and BB interval sequence based on the peak positions of the R wave and respiration, respectively; the RR interval sequence includes the R wave of each heartbeat; two adjacent R waves constitute one RR interval; the BB interval sequence includes the peak position of each respiration; the peak positions of two adjacent respirations constitute one BB interval.

[0036] Specifically, the ECG and respiratory signals are denoised and feature points are identified: The ECG and respiratory signals are denoised by using a third-order Butterworth bandpass filter to eliminate high-frequency noise and low-frequency drift in the ECG signal, a fourth-order Butterworth lowpass filter to eliminate high-frequency components in the respiratory signal, the Pan-Tompkins detection algorithm for R-wave localization, and the local maximum detection algorithm for respiratory peak localization. The RR interval sequence and BB interval sequence are calculated and resampled to maintain the consistency of temporal characteristics and frequency.

[0037] In the ECG signal processing, a third-order Butterworth bandpass filter was used, with a passband frequency range of 0.5Hz to 50Hz. This filter removed low-frequency drift noise below 0.5Hz and high-frequency electromyographic interference above 50Hz, while retaining the main energy components of the QRS complex. The lower limit (0.5Hz) and upper limit (50Hz) were normalized by dividing the sampling frequency by 2 and multiplying by 2, respectively. For respiratory signal processing, a fourth-order Butterworth low-pass filter was used, with a cutoff frequency set to 0.4Hz. The low-pass cutoff frequency was calculated by normalizing the set cutoff frequency with the Nyquist frequency (half the sampling rate). This fourth-order Butterworth low-pass filter effectively suppressed high-frequency noise, retained the main low-frequency components of respiratory movements, and extracted the trend of respiratory rhythm changes.

[0038] For ECG signal feature point detection, the Pan-Tompkins algorithm is used for R-wave localization. First, the denoised ECG signal is localized and differentiated to highlight the rapid changes in the QRS complex. Then, energy features are enhanced through sliding integral. Combining dynamic thresholding and a fixed search radius, each initially identified R-wave position is locally corrected to ensure the stability and accuracy of R-wave detection. If the amplitude difference between the first and last R-waves exceeds 0.6, it is judged as a boundary anomaly and removed, thereby improving the stability and localization accuracy of the R-wave sequence. For respiratory signal feature point detection, a local maximum detection algorithm is used, combined with minimum peak spacing, amplitude filtering, and time interval adjustment mechanisms to dynamically correct the respiratory peak position and extract high-confidence respiratory peak sequence. During detection, the minimum peak spacing is set to approximately 2 seconds. Finally, by traversing and filtering all maxima points that meet the conditions, the complete respiratory peak sequence is extracted.

[0039] After the R wave and respiratory peak are identified, the RR interval and BB interval sequences are calculated respectively, and the cubic spline interpolation algorithm is used for smooth reconstruction. A uniformly sampled continuous interval time series is generated at a frequency of 2Hz, providing a standardized basis for subsequent feature extraction and modeling.

[0040] Cubic spline interpolation first constructs a cumulative time axis based on the time difference between adjacent R waves or respiratory peaks, arranging the original interval values ​​in chronological order. Then, a set of third-order polynomials is constructed between every two adjacent data points to fit the curve within that interval. During the reconstruction process, a series of equally spaced new time points are generated at a frequency of 2Hz, and the corresponding RR interval and BB interval values ​​are calculated at each time point using the aforementioned fitted polynomials. Finally, a reconstructed sequence with continuous, smooth, and equally spaced sampling under a unified time axis is obtained.

[0041] .

[0042] in, The time difference between two adjacent periods. and It is at point and The second derivative value of the interphase change.

[0043] The denoising preprocessing of ECG and respiratory signals was completed, accurately locating the R wave of each heartbeat and the peak position of each breath. Then, the RR interval sequence and BB interval sequence were calculated. Through cubic spline interpolation, the two types of intervals were continuously and smoothly reconstructed on a unified time axis, generating high-quality time series data with a unified sampling frequency, time alignment, and suitability for subsequent feature analysis. This lays the foundation for subsequent time-frequency feature extraction and multimodal fusion analysis.

[0044] Furthermore, the central lung synergistic features in step S2 include time-domain features, frequency-domain features, nonlinear features, and coupling features.

[0045] Specifically, in order to extract statistical features of electrocardiogram and respiratory signals that can reflect the regulatory effect of changes in emotional state on the autonomic nervous system, for example, in a state of emotional excitement or stress, sympathetic nerve activity is enhanced, often manifested as increased heart rate and rapid breathing, which in turn leads to a shortening of the RR interval and BB interval and a decrease in their variability; while in a calm and relaxed state, the parasympathetic nervous system is dominant, the heart rate slows down, breathing becomes slower, and the variability of the intervals is enhanced.

[0046] Time-domain features include: the mean, first standard deviation (SDNN), root mean square (RMSSD), and second standard deviation of the RR interval and BB interval sequences. The first standard deviation is based on Determined; the second standard deviation is based on Certain; among them, The first standard deviation; This represents the total number of RR intervals or BB intervals. For the first One RR interval or BB interval; The mean of the RR interval or BB interval; The second standard deviation; This is used to calculate the number of consecutive RR intervals or BB intervals where the difference exceeds 20 ms.

[0047] Frequency domain characteristics include the oscillation energy distribution characteristics of RR interval sequences and BB interval sequences in different frequency bands.

[0048] Specifically, the oscillation energy distribution characteristics are extracted based on the Welch power spectrum estimation method, which extracts features such as very low frequency power (VLF), low frequency power (LF), high frequency power (HF), total power, normalized low frequency power (nLF), normalized high frequency power (nHF), and low-high frequency power ratio (LF / HF Ratio) of the RR interval sequence and the BB interval sequence. This reveals the oscillation energy distribution characteristics of heart rate oscillation energy at different frequency components and reflects the relative strength of sympathetic and parasympathetic nerve regulation.

[0049] Nonlinear features include: the standard deviation of points perpendicular to the diagonal in the Poincaré plot at different time ranges, the standard deviation of points along the diagonal in the Poincaré plot at different time ranges, and the approximate entropy (ApEn), fuzzy entropy (FuzzyEn), and sample entropy (SampEn) of the RR interval sequence and BB interval sequence; nonlinear features are used to describe the regularity of heart rate and respiratory sequences, and to capture changes in the nonstationarity of the autonomic nervous system induced by emotional states.

[0050] The coupling features include the joint entropy of the RR interval sequence and the BB interval sequence, as well as the cross-fuzzy entropy of the RR interval sequence and the BB interval sequence; the coupling features are used to assess the synergistic relationship between the cardiopulmonary system and the regulatory effect of emotional state on the cardiopulmonary interaction.

[0051] Further, the convolutional layer includes: a one-dimensional convolutional layer, a first one-dimensional max pooling layer, and a second one-dimensional max pooling layer connected in sequence; step S3 includes steps S31-S35: Step S31: Input the cardiopulmonary coordination features into a one-dimensional convolutional layer to obtain the convolutional cardiopulmonary coordination features.

[0052] Step S32: Input the convolutional cardiopulmonary coordination features into the first one-dimensional max pooling layer to obtain the cardiopulmonary coordination features after the first pooling.

[0053] Step S33: Input the cardiopulmonary coordination features after the first pooling into the second one-dimensional max pooling layer to obtain the cardiopulmonary coordination features after the second pooling.

[0054] Step S34: Input the cardiopulmonary coordination features after the second pooling into the bidirectional long short-term memory layer to obtain the global cardiopulmonary coordination features.

[0055] Step S35: Input the global cardiopulmonary coordination features into the attention mechanism layer to obtain the cardiopulmonary coordination feature vector.

[0056] Specifically, convolutional layers are used to discover local correlation patterns between feature dimensions, bidirectional long short-term memory layers are used to capture global dependencies across feature dimensions, and attention mechanisms further highlight feature subsets that are more critical for emotion discrimination.

[0057] Among them, the cardiopulmonary coordination features are input into a convolutional bidirectional attention network (CBANet), and the network structure is as follows: Figure 3 As shown, this network delves into the inter-dimensional correlations and global dependencies in the statistical features of the RR and BB intervals. Specifically, the convolutional layer first performs convolution operations on the input cardiopulmonary coordination features to capture local correlation patterns between adjacent feature dimensions, and enhances the discriminative power of these local combinations through nonlinear mapping and pooling compression. Subsequently, the cardiopulmonary coordination features after the second pooling are fed into a bidirectional long short-term memory layer, which uses a bidirectional recurrent structure to model cross-dimensional global dependencies under the serialized representation of feature dimensions, revealing the potential interactions and overall combination patterns between different feature subsets (temporal features, frequency features, nonlinear features, and coupled features). Finally, the attention mechanism layer adaptively weights the output of the bidirectional long short-term memory layer, highlighting the feature dimensions that contribute most to emotion discrimination, thereby obtaining a compact and highly discriminative global feature representation. Overall, the convolutional bidirectional attention network achieves deep structural modeling of RR and BB interval features by mining local dimensional correlations through convolutional layers, capturing cross-dimensional dependencies through bidirectional recurrent layers in the bidirectional long short-term memory layer, and focusing on key features through the attention mechanism layer. This breaks through the limitations of traditional methods that only perform feature splicing or shallow statistics, and provides more accurate feature representation for emotion recognition based on cardiopulmonary signals.

[0058] Further, in step S4, the denoised ECG signal and the denoised respiratory signal are downsampled, wavelet transform is performed on the selected signal segment, and key frequency band components are retained by frequency interval pruning to obtain the time-frequency diagram of the ECG signal and the time-frequency diagram of the respiratory signal.

[0059] Specifically, based on the frequency component characteristics of the signals, the ECG and respiratory signals were downsampled separately. The ECG signal was downsampled to 360Hz, and the respiratory signal was downsampled to 100Hz. To maintain the waveform continuity and smoothness of the signal during the downsampling process, Fourier interpolation reconstruction was used to smooth the signal, so that the downsampled signal could still accurately reflect the overall shape and energy distribution characteristics of the original signal in the time domain. Wavelet transform mapped the time-domain signal to the time-frequency domain, extracting the energy distribution characteristics at different time scales. Furthermore, by performing modulus square operation on the wavelet transform coefficients, the energy distribution map of the signal in both time and frequency dimensions was obtained. Finally, effective frequency domain extraction was achieved through pseudo-frequency mapping and logical mask clipping, thereby generating the time-frequency maps of the ECG and respiratory signals.

[0060] Furthermore, in step S5, the hierarchical phantom convolution module includes: a channel partitioning layer, a hierarchical convolution processing layer, and a feature splicing layer.

[0061] Furthermore, in step S5, the frequency domain attention module includes: a fast Fourier transform layer, an amplitude calculation layer, a 1×1 convolution + sigmoid activation function layer, a frequency weighting layer, a ModReLU activation function layer, and an inverse fast Fourier transform layer connected in sequence.

[0062] Specifically, the time-frequency maps of the electrocardiogram and the respiratory signal are input into the Multi-Scale Spatial–Frequency Attention Fusion Network (MSPAFNet) proposed in this application, and its network structure is as follows: Figure 4 As shown, this network introduces a multi-scale spatial-frequency domain attention block (MSPA-FFTBlock) into the residual structure, and its structure is as follows: Figure 5 As shown, the multi-scale spatial pyramid attention module in the multi-scale spatial-frequency domain attention block (its structure is as follows) Figure 6 As shown, the Hierarchical Pyramid Convolution Module (HPCModule) is used (its structure is as follows). Figure 7 (as shown) and the Spatial Pyramid Recalibration Module (SPR Module) (its structure is as shown) Figure 8As shown, this achieves adaptive enhancement of multi-scale features in the spatial domain. The frequency domain attention module (Fast Fourier Transform Attention Module, FFT-Attention Module) in the multi-scale spatial-frequency domain attention block (its structure is shown below) Figure 9 (As shown) This approach amplifies key frequency bands related to emotion and suppresses redundant components in the frequency domain, thereby forming complementary modeling in the spatial and frequency domains. Compared to traditional convolutional neural networks, this application can more comprehensively and accurately capture local transient textures and global frequency patterns in ECG and respiratory time-frequency maps, significantly improving the discriminativeness and robustness of features.

[0063] The time-frequency plots of ECG and respiratory signals are input into the multi-scale spatial and frequency domain enhancement network, respectively. Two branches process the ECG and respiratory time-frequency plots independently, with completely symmetrical structures and no shared weights, ensuring that the feature learning processes of ECG and respiratory signals do not interfere with each other, and each captures its unique time-frequency pattern in its spectral image. The multi-scale spatial and frequency domain enhancement network adopts a [2,2,2,2] hierarchical layout, with an initial feature extraction structure (composed of 7×7 convolution, batch normalization, ReLU activation function, and 3×3 max pooling) at the front end for initial feature extraction and spatial downsampling. In the backbone, each residual unit uses a multi-scale spatial-frequency domain attention block (3×3 convolution → multi-scale spatial pyramid attention module → frequency domain attention module), as shown below. Figure 4 As shown, complementary enhancements in the spatial and frequency domains are achieved.

[0064] The 3×3 convolution placed at the beginning of each residual unit serves to perform preliminary local modeling and feature transformation on the input features. This convolution receives the feature map output from the previous residual unit as input and reorganizes the input feature channels and spatial patterns to better suit the subsequent multi-scale spatial pyramid attention module. Simultaneously, it combines batch normalization and non-linear activation to further enhance the discriminativeness and stability of the features. When downsampling is required, this convolution can also reduce the feature map size while preserving local discriminative information as much as possible, thus providing high-quality input for subsequent spatial and frequency domain enhancement.

[0065] In the spatial domain, the Multi-Scale Spatial Pyramid Attention Module (MSPA Module) first employs a hierarchical phantom convolution module (HPC Module) for multi-scale feature extraction: the input features are segmented into three subspaces by channel partitioning at the channel partitioning layer, and then extracted through hierarchical convolution processing layers. Specifically, each subspace undergoes 3×3 convolution and two-dimensional batch normalization to extract local features. A hierarchical residual connection is used to add the normalized output of the convolution of the previous branch to the input of the next branch, so that the prior information of the lower-level branches is progressively passed to the branches with larger receptive fields, thereby constructing multi-scale candidate features of "detail-texture-background" layer by layer. Finally, the features are concatenated through a feature stitching layer. Specifically, the channel stitching explicitly retains the weighted output of the three subspaces, providing more discriminative input for subsequent channel recalibration and attention allocation. Specifically, the small receptive field branch in the hierarchical phantom convolution module highlights the fast transient energy stripes and fine neighborhood textures related to QRS complexes in the ECG time-frequency map, and depicts local energy fluctuations within the respiratory cycle in the respiratory time-frequency map; the medium receptive field branch captures local P / T wave patterns in the ECG and harmonic expansion near the main rhythm band in the respiratory signal; and the large receptive field branch emphasizes the rhythm-related slowly varying background in the ECG and the global trend of the low-frequency rhythm energy band in the respiratory signal evolving over time. This multi-scale modeling approach can simultaneously take into account the fast transients in the ECG and the slow rhythm band features in the respiratory signal, forming a multi-scale candidate representation with rich context, providing high-quality input for subsequent channel attention and cross-scale enhancement.

[0066] For each branch, the embedded Spatial Pyramid Recalibration Module (SPR Module) further models the global and local statistical relationships between channels based on these multi-scale features. It extracts global and local features through a joint 1×1 first adaptive 2D average pooling layer and a 2×2 second adaptive 2D average pooling layer. Cross-channel nonlinear interactions are introduced via 1×1 convolution and a linear rectified function, and channel-wise attention weights are generated using a Sigmoid activation function to adaptively enhance multi-scale features. This global-local joint approach highlights the spatial cues that the branch excels at. These weights achieve branch-level soft selection through a Softmax activation function in the branch dimension. After element-wise weighting of the three features, branch summation is not performed; instead, the weighted output of each branch is concatenated in the channel dimension. Thus, the Spatial Pyramid Recalibration Module can highlight the energy patterns of QRS complexes and their neighborhoods on ECG time-frequency maps and enhance the dominant frequency band and slowly evolving respiratory rhythms on respiratory time-frequency maps, achieving complementary enhancement of global-local and detail-background aspects. It outputs multi-scale features after channel recalibration, laying the foundation for subsequent spatial and frequency domain co-modeling.

[0067] In the frequency domain, the Frequency-Domain Attention Module (FFT-Attention Module) further enhances the discriminative power in the frequency dimension by spectrally weighting the feature map after the multi-scale spatial pyramid attention module. Specifically, this module first transforms the input spatiotemporal domain feature map to the spectral domain using a two-dimensional Fast Fourier Transform (FFT) to explicitly separate amplitude and phase components. Subsequently, a 1×1 convolution is used to perform a channel-by-channel linear transformation on the spectral amplitude map, which is equivalent to performing adaptive feature recombination between channels in the frequency domain. This is combined with a Sigmoid activation function to generate frequency-domain attention weights, achieving adaptive selection and suppression of different frequency components. Next, the frequency-weighted complex features are input to the ModReLU activation function. Biased cosine modulation and ReLU activation introduce nonlinear modulation in the phase space, thereby highlighting the main frequency band that contributes more to the signal energy and weakening noise interference bands. Finally, the weighted spectral features are restored to the spatiotemporal domain through an Inverse Fourier Transform (IFFT). The output maintains the same number of channels as the input, but with significantly enhanced frequency selectivity and discriminative power. In this way, the module can enhance the energy patterns of QRS complexes and their harmonic bands in the electrocardiogram time-frequency diagram, and highlight the rhythmic evolution of the main frequency band and low frequency over time in the respiratory time-frequency diagram, thereby achieving frequency domain supplementary modeling of spatiotemporal characteristics.

[0068] The multi-scale spatial and frequency domain enhancement network proposed in this application constitutes a mutually reinforcing dual-domain enhancement mechanism in the spatial and frequency domains. Unlike the direct connection scheme of "ordinary convolution → frequency domain module," the multi-branch channel segmentation of the multi-scale spatial and frequency domain enhancement network performs branch-level soft selection and channel recalibration before entering the frequency domain. Specifically, through channel segmentation and hierarchical branching, the spatial pyramid recalibration module uses global and local statistical weighting to first filter out spatial redundancy and highlight key information at different scales and channels, making the features fed into the frequency domain attention module more task-relevant. Subsequently, the frequency domain attention module implements learnable gating and adaptive filtering on important frequency bands in the frequency domain, explicitly amplifying key frequency bands and suppressing irrelevant bandwidth. This two-end collaboration is equivalent to a cascaded selection process of "spatial optimization followed by frequency domain purification": the spatial side captures multi-scale textures and structural patterns, while the frequency side enhances discriminative bandwidth, effectively avoiding the frequency domain module's mis-amplification of spatial redundancy, thereby significantly improving the discriminative power of the features. Each residual unit in the network retains the identity shortcut to ensure training stability. After the four stages, an adaptive global average pooling is used to generate a 672-dimensional aggregated representation.

[0069] The emotion recognition method proposed in this application, based on multimodal feature extraction of ECG and respiratory signals, completes multimodal feature extraction. Multimodal feature extraction, as the core component of the overall architecture, aims to achieve multi-dimensional joint modeling of both ECG and respiratory signals. Specifically, multimodal feature extraction includes two main branches: First, a cardiopulmonary coordination feature extraction branch based on RR and BB intervals, which uses a convolutional bidirectional attention network to perform deep structural modeling of the temporal, frequency, nonlinear, and coupling features between heart rate and respiratory rhythm, uncovering local correlations and cross-dimensional dependencies between feature dimensions; Second, a spatial-frequency domain feature extraction branch based on the time-frequency maps of denoised ECG and respiratory signals, which utilizes a multi-scale spatial and frequency domain enhancement network to perform complementary enhancement in the spatial and frequency domains, extracting highly discriminative time-frequency features. The two branches are jointly optimized at a high level through a feature fusion module to form a cross-modal fusion representation, achieving multi-dimensional perception and refined recognition of emotional states, thus constituting a complete multimodal feature extraction system.

[0070] The beneficial effects of the emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals proposed in this application are mainly reflected in the following aspects.

[0071] 1. The core idea of ​​this application is based on the dual-path collaboration of "interval statistical branch + time-frequency image branch", which simultaneously models across feature dimensions and in both spatial and frequency domains. Specifically: the time-domain, frequency-domain, nonlinear, and coupling features of the interpolated RR and BB intervals are input into a convolutional bidirectional attention network. One-dimensional convolution captures the local correlations between feature dimensions, while a bidirectional long short-term memory network characterizes the global dependencies across dimensions. The attention mechanism adaptively highlights key features, enabling deep mining of relationships among multiple statistical features. In the image branch, the time-frequency images of ECG and respiration are input into the proposed multi-scale spatial and frequency domain enhancement network, respectively. A multi-scale spatial pyramid attention module is introduced into the residual unit, forming a multi-branch structure through channel segmentation. Combined with a context-aware channel weighting mechanism, the discriminative cues of different spatial scales and channels are adaptively enhanced. At the same time, the frequency domain attention module is integrated to perform learnable gating and adaptive filtering on key frequency bands in the frequency domain, forming a complementary modeling mechanism of "first selecting the best in the spatial domain and then refining in the frequency domain". Through the above-mentioned multi-branch, cross-dimensional, and dual-domain collaborative design, this invention can more accurately capture the subtle differences in cardiopulmonary physiological signals under different emotional states, improve the accuracy and robustness of emotional state recognition, and provide strong technical support for complex emotion modeling.

[0072] 2. Two multi-scale spatial and frequency domain enhancement network branches (processing ECG time-frequency maps and respiratory time-frequency maps respectively) each output a 672-dimensional feature vector. These are first concatenated to form a 1344-dimensional feature vector, which is then compressed to a 512-dimensional representation via a fully connected layer. Subsequently, the 512-dimensional feature vectors output by the convolutional bidirectional attention network branch are concatenated to obtain the final 1024-dimensional fused feature, which is then input into a fully connected classifier to predict emotional states. This hierarchical fusion strategy effectively combines the cross-dimensional dependency information of the multi-scale spatial-frequency features of the time-frequency image branch and the interval feature branch, thereby improving the model's discriminative ability.

[0073] 3. By combining multi-scale spatial and frequency domain enhancement networks with convolutional bidirectional attention networks, multi-level, multi-scale, and cross-dimensional fine modeling of ECG and respiratory signals is achieved, thereby obtaining efficient and robust emotion recognition performance.

[0074] 4. Multi-scale spatial and frequency domain enhancement network with built-in self-designed multi-scale spatial-frequency domain attention block: In the spatial domain, a hierarchical phantom convolution module is used to divide the channel into a multi-branch structure and recalibrate it with the context-aware channel of the spatial pyramid recalibration module to adaptively highlight discriminative cues of different scales and channels; in the frequency domain, a frequency domain attention module is introduced to implement learnable gating and adaptive filtering on key frequency bands related to emotion to suppress redundant interference, thereby forming a dual-domain complementary enhancement mechanism of "first selecting the best in the spatial domain and then refining in the frequency domain", which significantly improves the representation quality of time-frequency map features.

[0075] 5. The convolutional bidirectional attention network utilizes convolutional layers to mine local correlations between feature dimensions, combines bidirectional long short-term memory layers to model cross-dimensional global dependencies, and adaptively emphasizes key dimensions with an attention mechanism, breaking through the shallow approach of traditional feature concatenation. The two feature paths are fused at the semantic level, enabling the model to form complementary representations in both image space—frequency features and interval dimension features—taking into account both minute transient changes and macroscopic control patterns. Experiments show that this design achieves significant improvements in accuracy, stability, and robustness, fully demonstrating the technical value of this invention in network structure innovation and physiological signal emotion recognition applications.

[0076] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores electrocardiogram (ECG) and respiratory signals. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements an emotion recognition method based on multimodal feature extraction from ECG and respiratory signals.

[0077] Those skilled in the art will understand that Figure 10 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0078] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0080] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0081] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals, characterized in that, The emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals includes: Denoising and localization were performed on the electrocardiogram (ECG) signal and respiratory signal, respectively, to obtain the denoised ECG signal, denoised respiratory signal, RR interval sequence and BB interval sequence; The RR interval sequence and the BB interval sequence were aligned using cubic spline interpolation, and features were extracted from the aligned RR interval sequence and BB interval sequence to obtain the cardiopulmonary coordination features of the RR interval sequence and BB interval sequence. The cardiopulmonary coordination features are input into a trained convolutional bidirectional attention network, which outputs a cardiopulmonary coordination feature vector. The convolutional bidirectional attention network includes a convolutional layer, a bidirectional long short-term memory layer, and an attention mechanism layer connected in sequence. The denoised ECG signal and the denoised respiratory signal were respectively downsampled, wavelet transformed and frequency clipped to obtain the time-frequency graph of the ECG signal and the time-frequency graph of the respiratory signal. The time-frequency plots of the electrocardiogram (ECG) signal and the respiratory signal are respectively input into a trained multi-scale spatial and frequency domain enhancement network, which outputs ECG feature vectors and respiratory feature vectors. The multi-scale spatial and frequency domain enhancement network includes multiple multi-scale spatial-frequency domain attention blocks. Each multi-scale spatial-frequency domain attention block includes a multi-scale spatial pyramid attention module and a frequency domain attention module. The multi-scale spatial pyramid attention module includes a hierarchical phantom convolution module and a spatial pyramid recalibration module. The electrocardiogram feature vector and the respiratory feature vector are fused to obtain a fused feature vector; The cardiopulmonary coordination feature vector and the fused feature vector are input into a fully connected classifier to obtain the emotional state.

2. The emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals according to claim 1, characterized in that, Denoising and localization were performed on the electrocardiogram (ECG) and respiratory signals respectively, resulting in denoised ECG signals, denoised respiratory signals, RR interval sequences, and BB interval sequences, specifically including: The ECG signal was denoised using a third-order Butterworth bandpass filter to obtain the denoised ECG signal. The respiratory signal was denoised using a fourth-order Butterworth low-pass filter to obtain a denoised respiratory signal. The Pan-Tompkins detection algorithm was used to locate the R wave in the denoised ECG signal; The local maximum detection algorithm is used to locate the peak position of the respiration signal after denoising. Based on the peak positions of the R wave and respiration, the RR interval sequence and BB interval sequence are calculated respectively; the RR interval sequence includes the R wave of each heartbeat; two adjacent R waves constitute one RR interval; the BB interval sequence includes the peak position of each respiration; the peak positions of two adjacent respirations constitute one BB interval.

3. The emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals according to claim 1, characterized in that, The cardiopulmonary coordination characteristics include time-domain characteristics, frequency-domain characteristics, nonlinear characteristics, and coupling characteristics; The time-domain features include: the mean, first standard deviation, root mean square, and second standard deviation of the RR interval and BB interval sequences; the first standard deviation is based on... Determined; the second standard deviation is based on Certain; among them, The first standard deviation; This represents the total number of RR intervals or BB intervals. For the first One RR interval or BB interval; The mean of the RR interval or BB interval; The second standard deviation; To calculate the number of consecutive RR intervals or BB intervals where the difference exceeds 20 ms; The frequency domain features include: the oscillation energy distribution characteristics of the RR interval sequence and the BB interval sequence in different frequency bands; The nonlinear features include: the standard deviation of points perpendicular to the diagonal in the Poincaré plot within different time ranges, the standard deviation of points along the diagonal in the Poincaré plot within different time ranges, and the approximate entropy, fuzzy entropy, and sample entropy of the RR interval sequence and the BB interval sequence; the nonlinear features are used to describe the regularity of heart rate and respiratory sequences, and to capture changes in the nonstationarity of the autonomic nervous system induced by emotional states; The coupling features include the joint entropy of the RR interval sequence and the BB interval sequence, and the cross-fuzzy entropy of the RR interval sequence and the BB interval sequence; the coupling features are used to assess the synergistic relationship between the cardiopulmonary system and the regulatory effect of emotional state on the cardiopulmonary interaction.

4. The emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals according to claim 1, characterized in that, The convolutional layer comprises: a one-dimensional convolutional layer, a first one-dimensional max pooling layer, and a second one-dimensional max pooling layer connected in sequence. Specifically, the cardiopulmonary coordination features are input into a trained convolutional bidirectional attention network, and the cardiopulmonary coordination feature vector is output, including: The cardiopulmonary coordination features are input into the one-dimensional convolutional layer to obtain the convolutional cardiopulmonary coordination features. The convolutional cardiopulmonary coordination features are input into the first one-dimensional max pooling layer to obtain the first pooled cardiopulmonary coordination features. The cardiopulmonary coordination features after the first pooling are input into the second one-dimensional max pooling layer to obtain the cardiopulmonary coordination features after the second pooling. The cardiopulmonary coordination features after the second pooling are input into the bidirectional long short-term memory layer to obtain global cardiopulmonary coordination features; The global cardiopulmonary coordination features are input into the attention mechanism layer to obtain the cardiopulmonary coordination feature vector.

5. The emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals according to claim 1, characterized in that, The layered phantom convolution module includes: a channel partitioning layer, a layered convolution processing layer, and a feature splicing layer.

6. The emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals according to claim 1, characterized in that, The spatial pyramid recalibration module includes: a first adaptive two-dimensional average pooling layer and a second adaptive two-dimensional average pooling layer.

7. The emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals according to claim 1, characterized in that, The frequency domain attention module includes: a fast Fourier transform layer, an amplitude calculation layer, a 1×1 convolution + sigmoid activation function layer, a frequency weighting layer, a ModReLU activation function layer, and an inverse fast Fourier transform layer connected in sequence.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the emotion recognition method based on multimodal feature extraction of electrocardiogram and respiratory signals as described in any one of claims 1-7.

Images