Multi-modal heterogeneous neural stress recognition model
By using a multimodal heterogeneous neural stress recognition model, which utilizes a modality-specific encoder, dynamic gating cross-attention, and a bidirectional feature pyramid module, the problem of insufficient exploration of intermodal correlations in existing technologies is solved, thereby improving the accuracy and generalization ability of neural stress recognition.
Patent Information
- Application Number
- CN202511530728.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-08-20
- Filing Date
- 2025-10-24
- Publication Date
- 2026-02-27
AI Technical Summary
Existing multimodal fusion methods fail to fully explore the deep correlations between modalities and lack consideration for the dynamic importance of information from each modality, resulting in limited recognition accuracy and generalization ability of neural stress recognition models.
A multimodal heterogeneous neural stress recognition model is adopted. Features are extracted using a modality-specific encoder, cross-modal interaction and fusion are achieved through a dynamic gating cross-attention module, further processing is performed using a bidirectional feature pyramid module, and finally the neural stress recognition result is output using a classification module.
It improves the accuracy and generalization ability of neural stress recognition, realizes adaptive information interaction and fusion of multimodal heterogeneous data, enhances the expressive power of the model, and provides a more reliable basis for clinical diagnosis and health monitoring.
Smart Images
Figure CN121570124A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural stress recognition technology, specifically a multimodal heterogeneous neural stress recognition model. Background Technology
[0002] Neurological stress is a pervasive physiological and psychological phenomenon closely related to the occurrence and development of various diseases. Traditional stress assessment methods typically rely on subjective questionnaires or single-modality physiological signals (such as heart rate and skin conductance), making it difficult to comprehensively and objectively capture complex neurological stress states. In recent years, the development of various biomedical measurement technologies, such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and serum biochemical indicators, has provided new opportunities for a deeper understanding and identification of neurological stress from multimodal heterogeneous data.
[0003] However, effectively fusing these heterogeneous data with different spatiotemporal characteristics and expressions, and extracting robust stress discrimination features from them, remains a challenging problem. Existing multimodal fusion methods often employ simple splicing or weighted averaging, failing to fully explore the deep correlations between modalities and lacking consideration for the dynamic importance of information from each modality. This results in limited model recognition accuracy and generalization ability. Therefore, there is an urgent need for a new model and method that can adaptively perform cross-modal information interaction and fusion to achieve more accurate neural stress recognition. Summary of the Invention
[0004] The purpose of this invention is to provide a multimodal heterogeneous neural stress recognition model to solve the problem that existing multimodal fusion methods often use simple splicing or weighted averaging, which fail to fully explore the deep correlation between modalities and lack consideration of the dynamic importance of each modality's information, resulting in limited model recognition accuracy and generalization ability.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal heterogeneous neural stress recognition model, comprising:
[0006] Acquire electroencephalogram (EEG) data, functional magnetic resonance imaging (fMRI) data, and serum biochemical data of the individual to be identified;
[0007] The EEG data, fMRI data, and serum biochemical data are encoded using a modality-specific encoder, and feature representations of each modality are extracted, namely EEG features, fMRI features, and serum biochemical features.
[0008] A dynamic gated cross-attention module is used to perform cross-modal interaction and fusion of the EEG features, the fMRI features, and the serum biochemical features to generate fused cross-modal attention features;
[0009] The fused cross-modal attention features are further processed using a bidirectional feature pyramid module to generate the final fused feature representation.
[0010] The classification module receives the final fused feature representation and outputs the neural stress recognition result of the individual.
[0011] Preferably, the steps for cross-modal interaction and fusion using a dynamic gating cross-attention module specifically include:
[0012] The EEG features, fMRI features, and serum biochemical features are projected into the query (Q), key (K), and value (V) spaces, respectively;
[0013] Calculate the cross-attention score based on projection features between any two different modalities;
[0014] The value vector is weighted and summed based on the cross-attention scores to obtain the attention output of each modality from other modalities;
[0015] Based on the original features of each mode, a dynamic gating signal is generated through a gating network;
[0016] The dynamic gating signal is multiplied by the sum of the attention outputs obtained by the corresponding mode from other modes to obtain the attention features after dynamic gating.
[0017] The attention features of each modality that have undergone dynamic gating are spliced together;
[0018] Perform linear projection on the stitched features;
[0019] The linearly projected features are added to the original EEG features, fMRI features, and serum biochemical features using a residual concatenation method.
[0020] The summed features are normalized by applying layer normalization to obtain the fused cross-modal attention features.
[0021] Preferably, the step of generating the dynamic gating signal specifically includes:
[0022] The original features of each modality are input into a linear layer;
[0023] The Sigmoid activation function is applied to the output of the linear layer to obtain the dynamic gating signal.
[0024] Preferably, the step of further processing the fused cross-modal attention features using the bidirectional feature pyramid module specifically includes:
[0025] The fused cross-modal attention features are projected through a linear layer to generate the final fused feature representation.
[0026] Preferably, the step of outputting the individual's neural stress recognition result using the classification module specifically includes:
[0027] The final fused feature representation is then input into a fully connected layer;
[0028] Apply the Softmax activation function to the output of the fully connected layer to obtain the probability distribution of the individual belonging to different stress categories;
[0029] The final predicted category is determined based on the probability distribution and used as the neural stress recognition result.
[0030] Preferably, the system includes:
[0031] The data acquisition module is used to acquire electroencephalogram (EEG) data, functional magnetic resonance imaging (fMRI) data, and serum biochemical data of the individual to be identified.
[0032] A modal encoder module is used to encode the EEG data, the fMRI data, and the serum biochemical data respectively, and extract the feature representation of each modality;
[0033] The dynamic gated cross-attention module is used to perform cross-modal interaction and fusion of the modal features extracted by the modal-specific encoder module to generate fused cross-modal attention features.
[0034] The bidirectional feature pyramid module is used to further process the fused cross-modal attention features to generate the final fused feature representation.
[0035] The classification module is used to receive the final fused feature representation and output the neural stress recognition result of the individual.
[0036] Compared with existing technologies, the beneficial effects of this invention are: it effectively integrates multimodal heterogeneous data such as EEG, fMRI, and serum biochemistry, making full use of the complementary information of each modality; through a dynamic gating cross-attention mechanism, it achieves adaptive information interaction and fusion between modalities, highlighting important features and suppressing noise; by utilizing a bidirectional feature pyramid module, it enhances the multi-scale representation capability of features and improves the expressive power of the model; and it improves the accuracy and generalization ability of neural stress recognition, providing a more reliable basis for clinical diagnosis and health monitoring. Attached Figure Description
[0037] Figure 1 This is a schematic diagram of the overall architecture of the present invention;
[0038] Figure 2 This is a schematic diagram of the data flow of the present invention;
[0039] Figure 3 This is a schematic diagram of the EEG spatiotemporal encoder structure of the present invention;
[0040] Figure 4 This is a schematic diagram of the fMRI dynamic image encoder structure of the present invention;
[0041] Figure 5 This is a schematic diagram of the serum biochemical encoder structure of the present invention;
[0042] Figure 6 This is a schematic diagram of the bidirectional feature pyramid structure of the present invention. Detailed Implementation
[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] A multimodal heterogeneous neural stress recognition model, MHN-StressNet, employs a modular design with an overall architecture consisting of four main layers: an input layer, a modality-specific encoder layer, a cross-modal interaction layer, and an inference layer. The input layer receives multimodal data such as EEG signals, fMRI data, and serum biochemical indicators. The modality-specific encoder layer contains three dedicated encoders, each processing data from a different modality. The cross-modal interaction layer achieves intermodal information exchange through dynamic gating cross-attention and a bidirectional feature pyramid. The inference layer uses causal inference units and an uncertainty calibration module to predict the final stress state. This hierarchical design not only ensures the effective extraction of features from each modality but also enables adaptive fusion between modalities, providing a reliable foundation for the final stress recognition.
[0045] The overall model processes three different types of input data: EEG signals, fMRI data, and serum biochemical indicators. The EEG signal is a 64-channel × 600,000 time-point matrix with a sampling rate of 1000 Hz and a total duration of 10 minutes. Each channel represents the brain activity at an electrode location, and the time points represent continuous EEG signal sampling. The fMRI data is a 61 × 73 × 61 voxel × 300 time-point matrix with a sampling rate of 0.5 Hz (one image every 2 seconds) and a total duration of 10 minutes. Each voxel represents brain activity at a 3D spatial location, and the time points represent continuous fMRI scans. The serum biochemical indicators are a 20-dimensional vector containing 20 different biochemical indicators, each representing a specific physiological state.
[0046] The modal encoder layer transforms three different types of input data into a unified 512-dimensional feature vector. This transformation process preserves the key information of the original data while achieving dimensionality reduction and standardization. The EEG encoder extracts features of EEG signals through time-frequency analysis and spatial attention mechanisms; the fMRI encoder extracts features of brain functional networks through dynamic functional connectivity and graph attention networks; and the serum biochemical encoder extracts features of biochemical indicators through feature selection and factor decomposition.
[0047] The cross-modal interaction layer receives three 512-dimensional feature vectors and achieves intermodal information fusion through dynamic gating cross-attention and a bidirectional feature pyramid. The dynamic gating cross-attention module controls the information flow between different modalities through a learnable gating matrix to achieve adaptive feature fusion, while the bidirectional feature pyramid module achieves multi-scale feature fusion through a bottom-up and top-down dual-path design, ultimately outputting a 1024-dimensional fused feature.
[0048] The multi-level joint inference layer processes 1024-dimensional fused features and achieves the final stress state prediction through causal inference and uncertainty calibration. The causal inference unit analyzes the causal effects of different modal features through a counterfactual inference framework and identifies key biomarkers. The uncertainty calibration module quantifies the uncertainty of the model prediction through an evidence deep learning framework and provides a reliable confidence estimate. The final output includes a binary stress state prediction (0 represents the normal state and 1 represents the stress state) and an uncertainty estimate between 0 and 1.
[0049] The three modal encoders in this model are implemented in different ways:
[0050] EEG encoder: based on 3D convolution and multi-scale spatiotemporal attention mechanism;
[0051] fMRI encoder: Based on graph neural network (GNN), implemented through dynamic functional connections and GATv2 attention mechanism;
[0052] Serum biochemical encoder: based on feature selection gating and factor decomposition embedding;
[0053] The EEG spatiotemporal encoder is one of the core components of this model, specifically designed to process the spatiotemporal and spectral features of EEG signals. This encoder uses an improved Morlet wavelet transform as its foundation, combined with 3D convolution and a multi-scale spatiotemporal attention mechanism, to achieve multi-level feature extraction of EEG signals.
[0054] During the time-frequency conversion stage, the encoder first processes the input EEG signal. The signal is processed, where C is the number of channels and T is the number of time points. An improved Morlet wavelet transform is used to convert the time-domain signal into a time-frequency representation. Specifically, for each time point t and channel c, its time-frequency representation at different scales a is calculated.
[0055]
[0056] in:
[0057] The parameter 'a' is a scaling parameter that controls the scaling degree of the wavelet function; a larger value of 'a' corresponds to lower frequencies, and a smaller value of 'a' corresponds to higher frequencies.
[0058] Parameter b is a translation parameter that determines the position of the wavelet function on the time axis, corresponding to a specific time point.
[0059] ψ * Represents the complex conjugate of wavelet basis functions
[0060] To improve the accuracy and efficiency of time-frequency conversion, the encoder employs optimized scaling parameter settings. Specifically, the scaling parameter a... k Sampling is performed on a logarithmic scale:
[0061]
[0062] Where a0 = 2 / f s The base scale is defined as Δs = 1 / 8, where Δs is the scale step. In practical implementation, the sampling frequency f is set... s The time-frequency representation is set to 1000Hz, with a center frequency ω0 of 5.0, a scale number K of 64, and a time window of 256ms. These parameters were selected through extensive experimental verification, ensuring both computational efficiency and accuracy in time-frequency representation.
[0063] In the 3D convolution stage, the encoder uses a fixed-size convolution kernel for feature extraction. Specifically, it uses a 3×3×3 convolution kernel with a stride of 2 and padding of 1 to ensure that the feature map size is halved. The convolution operation can be represented as:
[0064] Y=Conv3D(W,K,stride=(2,2,2),padding=(1,1,1))
[0065] in It uses a fixed convolution kernel.
[0066] The multi-scale spatiotemporal attention mechanism is an innovation of EEG encoders. This mechanism achieves multi-scale feature extraction of EEG signals by simultaneously considering attention in three dimensions: time, frequency, and space. Specifically, for input features... Calculate the attention weights for each of the three dimensions:
[0067] Time attention:
[0068] A t =Softmax(W t ·Y)
[0069] in This is the time attention weight matrix.
[0070] Frequency attention:
[0071] A f =Softmax(W f ·Y)
[0072] in This is the frequency attention weight matrix.
[0073] Spatial attention:
[0074] A s =Softmax(W s ·Y)
[0075] in This is the spatial attention weight matrix.
[0076] The final attention output is obtained by multiplying the attention weights in three dimensions:
[0077] H attn =A t ⊙A f ⊙A s ⊙Y
[0078] Here, ⊙ represents element-wise multiplication. The advantages of this multi-scale spatiotemporal attention mechanism are: it can simultaneously capture features of time, frequency, and spatial dimensions, adaptively highlight important features through attention weights, has high computational efficiency, and is easy to implement in parallel.
[0079] To further improve the model's expressive power, the encoder employs a multi-head attention design with eight attention heads, each with a feature dimension of 64. Simultaneously, the introduction of residual connections and layer normalization effectively mitigates the gradient vanishing problem in deep networks.
[0080] H eeg =LayerNorm(Y+H) attn )
[0081] The fMRI dynamic map encoder is another core component of this model, specifically designed to process the spatiotemporal features of functional magnetic resonance imaging data. This encoder achieves efficient feature extraction from fMRI data through dynamic functional connections and an improved GATv2 attention mechanism.
[0082] During the dynamic functional connectivity construction phase, the encoder first processes the input fMRI signal. The process is performed, where V is the number of voxels and T is the number of time points. The functional connectivity matrix between different time points is calculated using the sliding window technique:
[0083]
[0084] in:
[0085] The fMRI signal at time point i is represented by a V-dimensional vector, where each element corresponds to the activity value of a voxel.
[0086] x(i) T It is the transpose of x(i), with dimension .
[0087] x(i)·x(i) T The calculated result is a V×V matrix, representing the functional connectivity strength between all voxel pairs at time point i.
[0088] W represents the window size, set to 30 time points. Since the fMRI data sampling rate is 0.5Hz (data is acquired every 2 seconds), the 30 time points correspond to a 60-second time window. Specifically, for any two voxels v1 and v2, their functional connectivity strength at time point i is calculated as follows:
[0089]
[0090] in and Let v1 and v2 represent the activity values of voxels at time point i, respectively. The average functional connectivity strength between these two voxels is obtained by averaging over the entire time window.
[0091]
[0092] Within the selected time window (W = 30 time points, corresponding to 60 seconds), calculate the correlation coefficient r between voxel pairs. ij First, calculate their covariance and standard deviation:
[0093]
[0094] in:
[0095] μ i and μ j These are the average values of voxels i and j within the time window [tW / 2, t+W / 2].
[0096] x i (t) and xj (t) is the activity value of the voxel at time point t.
[0097] W=30 is the size of the time window, corresponding to a time span of 60 seconds.
[0098] Then, the correlation coefficient r ij Calculated using the following formula:
[0099]
[0100] Here, r ij The Pearson correlation coefficient between voxels i and j within the time window is represented, with a value ranging from [-1, 1]. The correlation coefficient r ij Functional connection strength FC ij The relationship is:
[0101] FC ij Directly reflects the activity intensity between voxel pairs
[0102] r ij This reflects the correlation of activity between voxel pairs.
[0103] Both are important metrics for measuring functional connectivity, but they have different focuses.
[0104] To improve the reliability of functional connections, the encoder employs an adaptive window adjustment strategy. Specifically, for each time point t, the local rate of change of the signal near that time point is first calculated:
[0105]
[0106] Where V is the number of voxels, x v (t) represents the activity value of voxel v at time point t. The time window size is dynamically adjusted based on the local rate of change.
[0107]
[0108] in:
[0109] W min =15 (corresponding to 30 seconds) is the minimum window size
[0110] W max =45 (corresponding to 90 seconds) is the maximum window size
[0111] W base =30 (corresponding to 60 seconds) is the base window size
[0112] θ high and θ low The high and low thresholds are set to the 75th and 25th quantiles of the signal rate of change distribution, respectively.
[0113] This adaptive window adjustment mechanism is closely related to the calculation of the functional connectivity matrix FC(t). Specifically, for each time point t, the functional connectivity matrix is recalculated using the adjusted window size W(t):
[0114]
[0115] The advantages of this dynamic adjustment strategy are: using a smaller window when the signal changes drastically to maintain a high time resolution, using a larger window when the signal is relatively stable to improve statistical reliability, and smoothly transitioning in intermediate states to avoid abrupt changes in window size.
[0116] During the dynamic edge weight calculation phase, the encoder first sparsifies the functional connection matrix, retaining only the top k strongest connections for each node. Specifically, for each node i, its top k = 20 neighboring nodes are selected, and the dynamic edge weights are calculated:
[0117]
[0118] Where α is a learnable temperature parameter. Let be the set of neighbors of node i. To ensure the stability of the graph structure, the encoder employs a dynamic thresholding strategy, adaptively adjusting the threshold based on the global connectivity strength:
[0119] θ(t) = μ(t) + β·σ(t)
[0120] Where μ(t) and σ(t) are the mean and standard deviation of all connection strengths at time point t, respectively, and β is an adjustable parameter.
[0121] Dynamic threshold θ(t) and dynamic edge weight A ij The relationship between (t) is as follows:
[0122] First, for each node i, based on the functional connection strength FC ij Filtering neighbor nodes using (t) and dynamic threshold θ(t):
[0123]
[0124] This ensures that only edges with connection strength exceeding the threshold are preserved.
[0125] Then, in the filtered neighbor set Calculate the dynamic edge weight A ij (t). This two-step calculation process can be represented as:
[0126]
[0127] The adaptive adjustment mechanism of the dynamic threshold θ(t) ensures that:
[0128] When the connection strength is generally high, μ(t) is large, and θ(t) increases accordingly, preserving stronger connections.
[0129] When the connection strength varies significantly, σ(t) is larger, and θ(t) increases accordingly, reducing noise in the connection.
[0130] The parameter β controls the sensitivity of the threshold to changes in connection strength.
[0131] This design allows the graph structure to adapt to dynamic changes in the strength of functional connections, maintaining the sparsity of the network while ensuring the preservation of important connections.
[0132] The GATv2 attention mechanism is a core innovation of the fMRI encoder, used to handle dynamic functional connectivity networks. This mechanism receives the previously computed dynamic edge weight matrix. As input, V represents the number of voxels (61×73×61), and t represents the time point. The entire processing flow is as follows:
[0133] Node feature initialization
[0134] First, the fMRI signal of each voxel is used as the initial node feature. For time point t, the feature vector of node i is... Defined as:
[0135] h i (t)=x i (t)
[0136] Where x i (t) represents the fMRI signal value of the voxel at time point t, and F represents the feature dimension (initially 1).
[0137] Feature transformation
[0138] To enhance feature representation, the node features are first subjected to a linear transformation:
[0139] h i ′(t)=W·h i (t)+b
[0140] in:
[0141] F is the learnable weight matrix, and F′ is the transformed feature dimension (set to 64).
[0142] For learnable bias vectors
[0143] This step maps the features of each node from the original signal space to a high-dimensional feature space.
[0144] Attention coefficient calculation
[0145] For each pair of adjacent nodes (i,j), calculate the attention coefficient e. ij (t):
[0146] e ij (t)=LeakyReLU(W2·LeakyReLU(W1·[h i ′(t)||h j ′(t)]))
[0147] in:
[0148] and The weight matrix is a learnable matrix;
[0149] [h i ′(t)| |h j [′(t)] represents concatenating the feature vectors of two nodes;
[0150] LeakyReLU is a ReLU activation function with leakage, used to introduce nonlinear transformations;
[0151] This two-level nonlinear transformation enhances the model's ability to capture complex node relationships;
[0152] Dynamic attention weight calculation:
[0153] The attention coefficient is compared with the previously calculated dynamic edge weight A. ij Combining (t), we obtain the final attention weights:
[0154]
[0155] in: Let A be the set of neighbors of node i, defined by dynamic edge weights A. ij (t) is determined; A ij (t) represents the dynamic edge weights calculated earlier, used to control the connection strength between nodes.
[0156] Multi-head attention mechanism: To improve the model's expressive power, a multi-head attention design is adopted. For each attention head k (a total of 8 heads), attention weights and feature aggregation are calculated independently.
[0157]
[0158] in:
[0159] K=8 is the number of attention heads.
[0160] The weight matrix for the k-th head
[0161] The bias vector of the k-th head
[0162] σ is a non-linear activation function (using GeLU).
[0163] Each head learns a different feature subspace independently, and the results are then concatenated.
[0164] Feature fusion and normalization
[0165] Finally, the final node features are obtained through residual connections and layer normalization:
[0166] h i "′(t)=LayerNorm(h i ′(t)+h i "(t))
[0167] in:
[0168] Residual connection h i ′(t)+h i "(t) helps alleviate the vanishing gradient problem"
[0169] Layer normalization ensures stable feature distribution and accelerates training convergence.
[0170] Output features
[0171] After the above processing, the node features h at each time point t are obtained. i "′(t). To obtain the final fMRI feature representation, global pooling is performed on the features of all nodes:
[0172]
[0173] Where: GlobalPool is the global pooling operation (using average pooling); the final output
[0174] This represents the global features of fMRI.
[0175] A serum biochemical encoder is specifically designed to process serum biochemical index data. Through feature selection gating and factor decomposition embedding, it achieves efficient representation of biochemical indicators. This encoder receives a 20-dimensional vector of serum biochemical indicators. As input, after a series of processing steps, the final output is a 512-dimensional feature representation.
[0176] In the feature selection gating mechanism, the encoder first calculates the importance score for each biochemical indicator using a two-layer gating network. The first gating layer maps the 20-dimensional input features to a 128-dimensional hidden space, introducing a non-linear transformation using the ReLU activation function. The second gating layer maps the 128-dimensional hidden features back to 20 dimensions, normalizing the importance score to the [0,1] interval using the Sigmoid activation function. Specifically, for each biochemical indicator x... k Its importance score g k The calculation process can be represented as:
[0177] h k =ReLU(W g ·x k +b g )
[0178]
[0179] in:
[0180] The first-layer gate weight matrix
[0181] The first layer bias vector
[0182] The second-layer gate weight matrix
[0183] For the second layer bias scalar
[0184] In the middle hidden state
[0185] Features mapped back
[0186] σ is the Sigmoid function
[0187] To ensure the stability of feature selection, the encoder introduces L1 regularization term ||g||1 and L2 regularization term. By adjusting the regularization coefficient λ g =0.01 balances the sparsity of feature selection with the expressive power of the model. Based on the calculated importance scores, the encoder adaptively filters features, retaining those with importance scores greater than 0.7 and setting other features to zero.
[0188]
[0189] This adaptive threshold design effectively identifies and retains important biochemical indicators while filtering out noise and redundant features. The final output of the feature selection gating mechanism... It can be represented as:
[0190] X gate =[x1′,x2′,...,x 20 ′]
[0191] In the factorization embedding stage, the encoder uses three fully connected layers to embed X... gate Each x is converted into an embedding. The hidden layer dimensions of the 3 fully connected layers are [256, 128, 64], and a nonlinear transformation is introduced using the GeLU activation function. For each fundamental factor x... i The learning process of ′ can be represented as:
[0192]
[0193] in:
[0194] The first layer weight matrix
[0195] The second layer weight matrix
[0196] The third layer weight matrix
[0197] For each layer bias vector
[0198] In the feature representation stage, the encoded e i The embeddings are then merged to form a unified embedding. Finally, a linear transformation layer maps the embedding representations to the final 512-dimensional feature space.
[0199] H ss =W out ·[e1,e2,...,e 20 ]+b out
[0200] in:
[0201] The output layer weight matrix (1280 = 20 indicators × 64-dimensional embedding)
[0202] For the output layer bias vector
[0203] To represent the final serum biochemical characteristics
[0204] The dynamic gated cross-attention module is one of the core innovations of this model. It achieves adaptive fusion of multimodal features through innovative dynamic convolutional gating and adaptive attention decomposition mechanisms. This module receives feature vectors from three modalities: EEG, fMRI, and serum biochemistry, with each modality having a feature dimension of 512.
[0205] First, a projection transformation is performed on the features of the three modalities. For the feature vector of each modality, it is mapped to the same feature space through linear projection:
[0206]
[0207] The query (Q), key (K), and value (V) vectors are obtained using a learnable projection matrix:
[0208]
[0209] in It is a learnable projection matrix. To calculate the attention score, the QK vectors need to be expanded into matrix form. Specifically, each vector is expanded into a 512×512 matrix through an outer product operation:
[0210]
[0211] The V matrix is extended to a second dimension, forming...
[0212]
[0213] in It is an all-one vector. Therefore, the attention score matrix can be calculated:
[0214]
[0215] To enhance the effectiveness of the attention mechanism, a dynamic convolutional gating mechanism was designed. First, the feature vectors from the three modalities are concatenated:
[0216]
[0217] Then, a two-layer fully connected network is used to map the spliced features to the appropriate dimension:
[0218]
[0219] in It is a learnable weight matrix. It is the bias vector.
[0220] Next, the projected features are reconstructed into feature maps. To maintain the validity of the features, we choose to reconstruct them into 32×32 feature maps:
[0221]
[0222] The generation process of dynamic convolution kernels is as follows: First, a global representation of the feature map is obtained through global average pooling:
[0223]
[0224] Then, the parameters of the dynamic convolutional kernel are generated through a three-layer fully connected network:
[0225]
[0226]
[0227] in It is a learnable weight matrix, b conv1 ,b conv2 It is the corresponding bias vector. These are the weight matrices used to generate the initialization parameters for 3×3, 5×5, and 7×7 convolutional kernels, respectively. 3 ×3 conv3 b 5×5 conv3 b 7×7 conv3 This corresponds to the bias. After obtaining it, it is transformed into the corresponding convolution kernel initialization parameter matrix through reshape transformation.
[0228] Multi-scale feature extraction is achieved through three parallel convolutional paths:
[0229]
[0230] in:
[0231] The paradigm is passed as the initialization kernel parameter, and parameters are learned based on the actual results. Adaptive attention decomposition is implemented through a three-branch network structure. Each branch contains a 1×1 convolutional layer and a spatial attention module. Specifically, for multi-scale input features... First, it is decomposed into three sub-feature maps:
[0232] G1,G2,G3=Split(G multi )
[0233] in These correspond to feature maps at different scales.
[0234] Then, attention decomposition is performed on each sub-feature map separately:
[0235]
[0236] in:
[0237] Conv1x1 represents a 1×1 convolution operation, used to adjust the number of feature channels.
[0238] SpatialAttention is a spatial attention module, and its calculation process is as follows:
[0239] SpatialAttention(X)=X⊙σ(W attn ·AvgPool(X)+b attn )in:
[0240] It is the input feature map
[0241] AUgPool is a global average pooling operation that compresses the feature map into a 1-dimensional vector.
[0242] and It is a learnable parameter.
[0243] σ is the Sigmoid activation function.
[0244] ⊙ indicates element-wise multiplication.
[0245] In this way, the attention module of each branch can adaptively focus on feature patterns at different scales:
[0246] A1 focuses on local feature relationships through a 3×3 convolutional path.
[0247] A2 focuses on medium-range feature dependencies through a 5×5 convolutional path.
[0248] A3 focuses on global feature interactions through a 7×7 convolutional path.
[0249] Finally, the attention features of the three branches are concatenated:
[0250]
[0251] After concatenating the attention features from the three branches, dynamic gating weights are generated through a 1×1 convolutional layer:
[0252]
[0253] in:
[0254] Conv1x1 represents a 1×1 convolution operation.
[0255] The sigmoid function is used to map the concatenated features (32×32×3=3072 dimensions) to a 512-dimensional array. It compresses the output values to the [0,1] interval and is used to generate gate weights.
[0256] This is the generated dynamic gating weight vector, where each element has a value range of [0,1].
[0257] Combining dynamic gating weights with the attention score matrix:
[0258]
[0259] Calculate the final attention output:
[0260]
[0261] The final output is obtained through residual connections and layer normalization:
[0262]
[0263] The following optimization strategies were adopted during the implementation process:
[0264] 1. Use Dropout to prevent overfitting, with a dropout rate set to 0.1.
[0265] 2. Use batch normalization layers to accelerate training convergence.
[0266] 3. Use the AdamW optimizer with a learning rate set to 1e-4.
[0267] 4. Employ a cosine annealing learning rate scheduling strategy.
[0268] The bidirectional feature pyramid module achieves multi-scale feature fusion through a bottom-up and top-down dual-path design. This module receives output features from the dynamically gated cross-attention module. By gradually expanding the feature dimensions through a 5-layer pyramid structure, a 1024-dimensional fused feature is finally output.
[0269] In the bottom-up path Each layer contains one convolution operation and one max pooling operation. The convolution operation uses a 3×3 kernel with a stride of 1 and padding of 1 to ensure the feature map size remains constant. The max pooling operation uses a 2×2 kernel.
[0270] With a step size of 2, the feature map size is halved. The bottom-up feature map of the l-th layer... The calculation process can be represented as:
[0271]
[0272] in For learnable convolutional kernels, C in and C out These represent the number of input and output channels, respectively.
[0273] In the top-down path, each layer contains a transposed convolution operation and a thinning convolution operation. The transposed convolution operation uses a 2×2 kernel with a stride of 2, doubling the feature map size. The thinning convolution operation uses a 3×3 kernel with a stride of 1 and padding of 1, used to optimize feature quality. The top-down feature map of layer l... The calculation process can be represented as:
[0274] The last one to get It is to calculate the first one The input of and It is a learnable convolutional kernel.
[0275] The feature fusion process employs an adaptive weighting strategy. For each layer, the bottom-up features are first weighted... and top-down features The data is concatenated and then passed through a 1×1 convolutional layer to generate fusion weights. The fusion weights are normalized to the [0,1] interval using a sigmoid function. The final fusion features are then obtained. We obtain the result through weighted summation:
[0276]
[0277] The final pyramid output feature H fusion By adding the fused features of all layers, we obtain:
[0278]
[0279] The bidirectional feature pyramid module achieves multi-scale feature extraction and fusion through a bottom-up and top-down bidirectional path. This module receives output features from the dynamically gated cross-attention module and transforms them into a 16×16×2 feature map through feature reshaping.
[0280] The classification layer receives output features from the bidirectional feature pyramid module. A fully connected layer maps it to a 2D space, corresponding to the two categories of stress and non-stress. The specific calculation process is as follows:
[0281] z = W·H fusion +b
[0282] p = Softmax(z)
[0283] in:
[0284] For learnable weight matrix
[0285] For bias vector
[0286] logits after linear transformation
[0287] Class probabilities after Softmax activation
[0288] The Softmax activation function transforms logits into a probability distribution:
[0289]
[0290] Where p i This represents the predicted probability of the i-th category. The final output includes: Predicted category: Category probability: p = [p1, p2].
[0291] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A multimodal heterogeneous neural stress recognition model, characterized in that, include: Acquire electroencephalogram (EEG) data, functional magnetic resonance imaging (fMRI) data, and serum biochemical data of the individual to be identified; The EEG data, fMRI data, and serum biochemical data are encoded using a modality-specific encoder, and feature representations of each modality are extracted, namely EEG features, fMRI features, and serum biochemical features. A dynamic gated cross-attention module is used to perform cross-modal interaction and fusion of the EEG features, the fMRI features, and the serum biochemical features to generate fused cross-modal attention features; The fused cross-modal attention features are further processed using a bidirectional feature pyramid module to generate the final fused feature representation. The classification module receives the final fused feature representation and outputs the neural stress recognition result of the individual.
2. The multimodal heterogeneous neural stress recognition model according to claim 1, characterized in that: The specific steps for cross-modal interaction and fusion using a dynamic gating cross-attention module include: The EEG features, fMRI features, and serum biochemical features are projected into the query (Q), key (K), and value (V) spaces, respectively; Calculate the cross-attention score based on projection features between any two different modalities; The value vector is weighted and summed based on the cross-attention scores to obtain the attention output of each modality from other modalities; Based on the original features of each mode, a dynamic gating signal is generated through a gating network; The dynamic gating signal is multiplied by the sum of the attention outputs obtained by the corresponding mode from other modes to obtain the attention features after dynamic gating. The attention features of each modality that have undergone dynamic gating are spliced together; Perform linear projection on the stitched features; The linearly projected features are added to the original EEG features, fMRI features, and serum biochemical features using a residual concatenation method. The summed features are normalized by applying layer normalization to obtain the fused cross-modal attention features.
3. The multimodal heterogeneous neural stress recognition model according to claim 2, characterized in that: The specific steps for generating dynamic gating signals include: The original features of each modality are input into a linear layer; The Sigmoid activation function is applied to the output of the linear layer to obtain the dynamic gating signal.
4. The multimodal heterogeneous neural stress recognition model according to claim 3, characterized in that: The steps for further processing the fused cross-modal attention features using the bidirectional feature pyramid module specifically include: The fused cross-modal attention features are projected through a linear layer to generate the final fused feature representation.
5. A multimodal heterogeneous neural stress recognition model according to claim 4, characterized in that: The steps for outputting the individual's neural stress recognition results using the classification module specifically include: The final fused feature representation is then input into a fully connected layer; Apply the Softmax activation function to the output of the fully connected layer to obtain the probability distribution of the individual belonging to different stress categories; The final predicted category is determined based on the probability distribution and used as the neural stress recognition result.
6. A multimodal heterogeneous neural stress recognition model according to claim 5, characterized in that: Its system includes: The data acquisition module is used to acquire electroencephalogram (EEG) data, functional magnetic resonance imaging (fMRI) data, and serum biochemical data of the individual to be identified. A modal encoder module is used to encode the EEG data, the fMRI data, and the serum biochemical data respectively, and extract the feature representation of each modality; The dynamic gated cross-attention module is used to perform cross-modal interaction and fusion of the modal features extracted by the modal-specific encoder module to generate fused cross-modal attention features. The bidirectional feature pyramid module is used to further process the fused cross-modal attention features to generate the final fused feature representation. The classification module is used to receive the final fused feature representation and output the neural stress recognition result of the individual.