Depression recognition method and device based on state space model and multi-modal fusion

Abstract

The application relates to the technical field of artificial intelligence and medical health, and discloses a depression recognition method and device based on a state space model and multi-modal fusion. Multi-channel electroencephalogram data is acquired, preprocessed and segmented to obtain electroencephalogram segments of fixed lengths of each channel; time-space features and frequency domain features are respectively acquired based on multi-modal feature extraction; a state space model is used to model the time sequence context of the time-space features to obtain context perception features; a cross-modal fusion mechanism is used to perform deep fusion on the context perception features and the frequency domain features to obtain a fusion feature sequence; multi-task decision is performed based on the fusion feature sequence to obtain a depression classification result, a depression severity total score and a brain function network activation intensity related to depression; and based on an explainability visualization technology, a clinical explainability atlas corresponding to the depression classification result is generated. The application fully excavates complementary information of time-space-frequency three-dimensional features, and improves the auxiliary recognition accuracy of depression.

Description

Technical Field

[0001] This application relates to the field of interdisciplinary technology of artificial intelligence and medical health, specifically to a method and device for depression identification based on state space model and multimodal fusion. Background Technology

[0002] Currently, deep learning-based EEG signal analysis methods have been applied to some extent in the identification of depression. Mainstream methods include using convolutional neural networks to extract local features or using Transformer models to capture long-range dependencies. For example, existing technologies often use CNNs (Convolutional Neural Networks) combined with RNNs (Recurrent Neural Networks) or self-attention mechanisms to build classification models for time-series modeling and classification of multi-channel EEG signals.

[0003] However, existing technologies have the following significant drawbacks:

[0004] First, in terms of long sequence processing, existing models based on self-attention mechanisms have a computational complexity of O(L²) (where L is the sequence length) when processing EEG signals with more than 1,000 sampling points, resulting in GPU memory usage exceeding 16GB, making it impossible to process long-term EEG records collected clinically.

[0005] Second, in terms of feature extraction, existing methods only utilize frequency domain features in simple power spectrum calculations and fail to achieve deep feature fusion in the three dimensions of time, space and frequency. As a result, the accuracy on public datasets is difficult to exceed 90%.

[0006] Third, in terms of model interpretability, existing visualization methods are limited to displaying attention weights and cannot provide clinically meaningful brain region localization and frequency band analysis, making it difficult to meet diagnostic needs.

[0007] Fourth, in terms of model generalization, the accuracy of existing single-task learning models drops by more than 15% in cross-center validation, which seriously restricts their clinical application.

[0008] Therefore, there is an urgent need to develop a depression identification method that can efficiently process long-sequence EEG signals, deeply integrate multi-dimensional features, and has strong clinical interpretability and generalization ability, in order to overcome the shortcomings of existing technologies and promote intelligent assisted diagnosis of depression. Summary of the Invention

[0009] To address the aforementioned issues, this application provides a method and apparatus for depression identification based on state-space model and multimodal fusion. This method processes long-sequence EEG signals with linear complexity, fully explores the complementary information of spatiotemporal and frequency three-dimensional features, improves the accuracy of depression auxiliary identification, and provides interpretable evidence to provide reliable support for clinical auxiliary diagnosis.

[0010] The technical solution adopted in this application is as follows:

[0011] Firstly, this application provides a method for depression identification based on state-space model and multimodal fusion, including:

[0012] Acquire multi-channel EEG data, perform preprocessing and segmentation on the EEG data to obtain EEG segments of fixed duration for each channel;

[0013] Multimodal feature extraction was performed on EEG segments from each channel to obtain spatiotemporal and frequency domain features, respectively.

[0014] A state-space model is used to perform temporal context modeling on spatiotemporal features to obtain context-aware features;

[0015] A deep fusion of context-aware features and frequency domain features is performed through a cross-modal fusion mechanism to obtain a fused feature sequence;

[0016] Multi-task decision-making is performed based on fused feature sequences to obtain depression classification results, total depression severity score and activation intensity of depression-related brain functional networks;

[0017] Based on interpretability visualization technology, a clinical interpretability atlas corresponding to the depression classification results is generated.

[0018] Secondly, this application also provides a depression recognition device based on state-space model and multimodal fusion, comprising:

[0019] The data preprocessing module is used to acquire multi-channel EEG data, and to perform preprocessing and segmentation on the EEG data to obtain EEG segments of fixed duration for each channel.

[0020] The multimodal feature extraction module is used to perform multimodal feature extraction on EEG segments from each channel, acquiring spatiotemporal features and frequency domain features respectively;

[0021] The temporal context modeling module is used to perform temporal context modeling on spatiotemporal features using a state-space model to obtain context-aware features.

[0022] The cross-modal fusion module is used to perform deep fusion of context-aware features and frequency domain features through a cross-modal fusion mechanism to obtain a fused feature sequence;

[0023] The multi-task decision module is used to perform multi-task decisions based on the fused feature sequence to obtain the depression classification results, the total score of depression severity, and the activation intensity of depression-related brain functional networks.

[0024] The interpretability visualization module is used to generate clinical interpretability maps corresponding to the depression classification results based on interpretability visualization technology.

[0025] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described depression recognition method based on state-space model and multimodal fusion.

[0026] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the depression recognition method based on state-space model and multimodal fusion described above.

[0027] The above-mentioned technical solution adopted in this application can achieve the following beneficial effects:

[0028] First, an enhanced multi-scale parallel convolution was designed. Through parallel processing of three types of convolution kernels (3×1, 5×1, and 7×1) and combined with a coordinate attention mechanism, the feature extraction accuracy was improved by 25% at key electrode positions such as Pz, Fz, and Cz.

[0029] Second, a temporal modeling method for EEG based on a selective state space model was proposed. Through a selective scanning mechanism and hardware sensing algorithm, real-time processing of 4000-point sequences was achieved with a hidden layer dimension of 512 and a state dimension of 16, reducing memory usage by more than 60%.

[0030] Third, a frequency domain feature extraction branch is introduced, and deep fusion of temporal-spatial-frequency features is achieved through cross-modal attention fusion. The deep alignment of temporal and frequency domain features is achieved through the Query-Key-Value mapping mechanism, which improves the feature complementarity effect in the Alpha and Theta bands by 40%.

[0031] Fourth, a multi-task interpretable output framework is designed to simultaneously perform classification, regression, and semantic auxiliary tasks. It integrates attention weights and gradient-weighted class activation mapping techniques. By jointly optimizing classification accuracy, regression error, and semantic consistency, the model can provide diagnostic results while outputting explanatory evidence that is highly correlated with the activity of the Theta band in the prefrontal cortex and the Alpha band in the parietal cortex. Attached Figure Description

[0032] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0033] Figure 1 A flowchart illustrating a depression identification method based on state-space model and multimodal fusion according to an embodiment of this application is shown.

[0034] Figure 2 A schematic flowchart illustrating the principle of data preprocessing according to an embodiment of this application is shown;

[0035] Figure 3 A flowchart illustrating the principle of multimodal feature extraction according to an embodiment of this application is shown;

[0036] Figure 4 A flowchart illustrating the principle of timing context modeling according to an embodiment of this application is shown;

[0037] Figure 5 A flowchart illustrating the principle of cross-modal fusion according to an embodiment of this application is shown;

[0038] Figure 6 A flowchart illustrating the principle of multi-task decision-making according to an embodiment of this application is shown;

[0039] Figure 7 A schematic flowchart illustrating the interpretability of one embodiment of this application is shown;

[0040] Figure 8 A schematic diagram of the structure of a depression recognition device based on state-space model and multimodal fusion according to an embodiment of this application is shown;

[0041] Figure 9 A schematic diagram of the structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] Figure 1 A flowchart illustrating a depression identification method based on state-space model and multimodal fusion according to an embodiment of this application is shown. (Refer to...) Figure 1 As shown, this embodiment includes steps S110 to S160:

[0044] Step S110: Acquire multi-channel EEG data, perform preprocessing and segmentation on the EEG data to obtain EEG segments of fixed duration for each channel.

[0045] Data acquisition, data preprocessing, and data segmentation are fundamental steps in depression identification. The purpose of preprocessing is to remove various noises and artifacts while preserving the most effective physiological information in the EEG data, thus providing high-quality data for subsequent feature extraction. Preprocessing operations such as bandpass filtering, ICA (Independent Component Analysis) denoising, and 64-lead rereference can be performed on multi-channel EEG data, followed by segmentation to output clean, noise-free EEG segments of fixed duration.

[0046] Specifically, step S110 can be executed through the data preprocessing module.

[0047] Figure 2 A schematic flowchart illustrating the principle of data preprocessing according to an embodiment of this application is shown. (Refer to...) Figure 2 As shown, in some optional embodiments, step S110, acquiring multi-channel EEG data, performing preprocessing and segmentation on the EEG data to obtain fixed-duration EEG segments for each channel, includes: acquiring 64-channel EEG data with a sampling rate of 256Hz; performing bandpass filtering on the EEG data of each channel using an infinite impulse response filter to obtain filtered data for each channel; wherein the high-pass cutoff frequency is 0.5Hz and the low-pass cutoff frequency is 45Hz; performing denoising on the filtered data of each channel using an independent component analysis algorithm to obtain denoised data for each channel; performing 64-lead rereference on the denoised data of each channel using the whole-brain average reference method to generate corrected data for each channel; and cutting the corrected data of each channel into segments of 10 seconds in length, outputting EEG segments with dimensions [64, 2560]; wherein 64 represents the number of channels and 2560 represents the number of sampling points.

[0048] 1. Data acquisition.

[0049] The acquired data is multi-channel EEG data, i.e., multi-channel EEG time series, with the following specific requirements:

[0050] Number of channels: 64;

[0051] Sampling rate: 256Hz;

[0052] Storage formats: edf, set, fdt, mat, or pure binary files;

[0053] Key information: channel name, electrode location coordinates, etc.

[0054] 2. Data preprocessing.

[0055] 2.1 Bandpass filtering.

[0056] The purpose of bandpass filtering is to retain the frequency components most relevant to brain neural activity while removing irrelevant extreme high-frequency and low-frequency noise (such as electromyography artifacts and baseline drift). The specific implementation is as follows:

[0057] Filter selection: Infinite Impulse Response (IIR) filter;

[0058] Qualcomm cutoff frequency: 0.5Hz; used to eliminate slow baseline drift caused by factors such as sweating and breathing;

[0059] Low-pass cutoff frequency: 45Hz; used to eliminate mains interference (50Hz power frequency interference) and its harmonics, while fully preserving the meaningful Gamma band ([30, 45]Hz) in the EEG signal.

[0060] Execution method: Independent filtering is performed on the 64-channel EEG data to obtain filtered data for each channel.

[0061] 2.2 ICA noise reduction.

[0062] The purpose of ICA denoising is to identify and separate artifacts from non-brain-derived sources in the signal, such as electrooculography (blinking, eye movements, etc.), electromyography (muscle tension, etc.), and electrocardiography. The specific implementation is as follows:

[0063] Algorithm selection: Mainstream ICA algorithms such as Infomax and FastICA;

[0064] Artifact identification: The filtered data from each channel is input into the ICA algorithm to decompose it into multiple independent components. By analyzing the three core features of each independent component—topology (scalp distribution), time series, and power spectrum—components that match known artifact features are automatically or semi-automatically identified. For example, electrooculography artifacts have a strong weighted distribution in the prefrontal cortex region and the sampling points match blink events.

[0065] Artifact removal: The identified artifact components are set to zero, and the remaining brain-derived components are reconstructed back into the channel space data through the inverse mixing matrix to obtain denoised data.

[0066] 2.3. Important reference.

[0067] Since the voltage recorded by EEG is relative, the purpose of 64-lead rereference is to unify the reference standard across channels, thereby improving data interpretability and consistency. The specific implementation is as follows:

[0068] Method selection: Whole brain average reference method; This method uses the average potential of all electrodes as a reference benchmark, which can effectively reduce the deviation caused by poor positioning of a single reference electrode or noise interference.

[0069] Calculation method: The corrected voltage value for each channel and each sampling point is calculated according to the following formula (1):

[0070] , formula (1);

[0071] in, This represents the corrected voltage value for one sampling point in one channel. This indicates the original voltage value at this sampling point in this channel. This represents the average potential of this sampling point across all 64 channels.

[0072] Execution method: For each channel and each sampling point, calculate the difference between its original voltage value and the average potential of all channels at that sampling point, and then generate the corrected data for each channel.

[0073] 3. Data segmentation.

[0074] The purpose of data segmentation is to divide preprocessed data into segments of fixed duration, facilitating subsequent batch processing and time-series feature analysis by the model. The specific implementation is as follows:

[0075] Clip length: 10 seconds;

[0076] Overlap settings: Whether and how segments overlap can be flexibly set as needed; for example, overlapping segments for 5 seconds can increase the amount of data.

[0077] Output dimension: The dimension of each EEG segment is [64, 2560]; where 64 is the number of channels and 2560 is the number of sampling points (2560 sampling points = 10 seconds duration × 256Hz sampling rate).

[0078] Step S120: Perform multimodal feature extraction on the EEG segments of each channel to obtain spatiotemporal features and frequency domain features respectively.

[0079] In multimodal feature extraction, spatiotemporal and frequency domain features of EEG segments are extracted separately, and complementary information between the two is mined to provide strong feature support for subsequent decision-making. Spatiotemporal features of EEG segments can be extracted using enhanced multi-scale parallel convolution, and frequency domain features can be extracted using short-time Fourier transform and frequency band analysis, achieving preliminary mining of spatiotemporal and frequency three-dimensional features.

[0080] Specifically, step S120 can be performed by the multimodal feature extraction module. The multimodal feature extraction module may further include a spatiotemporal feature extraction unit and a frequency domain feature extraction unit.

[0081] Figure 3 A flowchart illustrating the principle of multimodal feature extraction according to an embodiment of this application is shown. (Refer to...) Figure 3As shown, in some optional implementations, step S120 involves performing multimodal feature extraction on the EEG segments of each channel to obtain spatiotemporal features and frequency domain features, including: taking EEG segments with dimensions [batch size, 64, 2560] as input, capturing multi-scale temporal patterns through parallel depthwise separable convolutions including 3×1, 5×1, and 7×1 kernels to obtain three parallel output feature maps; wherein each convolutional layer is followed by batch normalization and ReLU activation function; after concatenating the three output feature maps along the channel dimensions, global average pooling is performed along the temporal and spatial dimensions to generate temporal feature descriptors and spatial feature descriptors; the temporal feature descriptors and spatial feature descriptors are then combined. The descriptors are concatenated along the channel dimension, followed by a 1×1 convolution and ReLU activation function. They are then re-segmented and each segment is convolved with a 1×1 convolution before being merged to generate a channel weight vector with dimensions [batch size, 192, 1]. The temporal and spatial feature descriptors are element-wise summed, followed by a 1×1 convolution and a Sigmoid function to generate a two-dimensional spatial attention map with dimensions [batch size, 1, 2560]. The channel weight vector and the two-dimensional spatial attention map are multiplied element-wise to obtain the attention weights. The attention weights are then multiplied element-wise with the three concatenated output feature maps to obtain a weighted feature map. The weighted feature map is then convolved with a 1×1 convolution to output a spatiotemporal feature sequence with dimensions [batch size, 128, 2560].

[0082] 1. Spatiotemporal feature extraction.

[0083] 1.1 Input feature specifications.

[0084] The input features are EEG segments output by the data preprocessing module, and the dimensions can be represented as [Batch Size, Channels, Time Points]; Batch Size represents the batch size (i.e., the number of samples), Channels represents the number of channels, and Time Points represents the number of sampling points. Specifically, when the batch size is 32, the dimensions of the input features are [32, 64, 2560].

[0085] 1.2 Enhanced multi-scale parallel convolution.

[0086] Enhanced multi-scale parallel convolution aims to simultaneously capture local patterns at different time scales and explicitly model the spatial topological relationships of electrodes. Specifically, to simultaneously capture neural activity patterns at different time scales, three parallel, depthwise separable convolutional pathways are designed. The specific implementation is as follows:

[0087] Convolutional Path 1 (Fine-grained mode): Employs a one-dimensional convolution with a kernel size of 3 (i.e., a 3×1 convolutional kernel), a stride of 1, padding of 1, and 64 output channels; This convolutional path focuses on capturing sharp transient activities and high-frequency oscillations.

[0088] Convolutional Path 2 (Medium Granularity Mode): Employs a one-dimensional convolution with a kernel size of 5 (i.e., a 5×1 convolutional kernel), a stride of 1, padding of 2, and 64 output channels; This convolutional path is used to capture a medium range of temporal correlations.

[0089] Convolutional Path 3 (Coarse-grained mode): It adopts a one-dimensional convolution with a kernel size of 7 (i.e., a 7×1 convolutional kernel), a stride of 1, padding of 3, and an output channel count of 64; This convolutional path has a large receptive field and is suitable for extracting slowly changing potentials and background rhythms.

[0090] Technical details: All convolutional layers are implemented using depthwise separable convolutions, and all convolutional layers are followed by batch normalization and ReLU activation functions;

[0091] Execution result: The three parallel convolutional paths output feature maps respectively.

[0092] 1.3 Feature splicing.

[0093] The three output feature maps from the three parallel convolutional paths are concatenated along the channel dimension to obtain a concatenated feature map with dimensions [32, 192, 2560]; where 192 = 64 channels × 3 paths.

[0094] 1.4 Fusion of coordinate attention mechanism.

[0095] The coordinate attention mechanism can adaptively focus on key channels and key sampling points related to depression, improving the targeting and effectiveness of spatiotemporal feature extraction. The specific implementation is as follows:

[0096] 1.4.1 Global average pooling.

[0097] Global average pooling is performed on the stitched feature map along both the time and spatial dimensions to generate two feature descriptors with dimensions [32, 192, 1]. One feature descriptor is a time feature descriptor that captures global time information, and the other feature descriptor is a spatial feature descriptor that captures global channel information.

[0098] 1.4.2 Channel attention calculation.

[0099] After concatenating the two feature descriptors along the channel dimension, a vector with dimensions [32, 384, 1] is obtained;

[0100] Dimensionality reduction is achieved by using a 1×1 convolution, reducing the number of channels from 384 to 96, and outputting a vector with dimensions [32, 96, 1].

[0101] Nonlinear transformations are performed using the ReLU activation function to learn complex time-space relationships;

[0102] The vector of dimension [32, 96, 1] is then re-divided proportionally into two vectors of dimension [32, 48, 1] according to the channel dimension;

[0103] Each of the two vectors with dimensions [32, 48, 1] is assigned an independent 1×1 convolutional layer to restore the number of channels of each vector from 48 to 192, resulting in two vectors with dimensions [32, 192, 1].

[0104] The two vectors with dimensions [32, 192, 1] are added element by element to obtain the channel weight vector with dimensions [32, 192, 1].

[0105] 1.4.3 Spatial attention calculation.

[0106] The two feature descriptors are added element-wise to obtain a fused feature descriptor with dimensions [32, 192, 1].

[0107] A two-dimensional spatial attention map of dimension [32, 1, 2560] is generated by a 1×1 convolution and a sigmoid function; where the value at each position represents the importance of the corresponding sampling point to all channels; the mathematical expression is as follows: (2):

[0108] , formula (2);

[0109] in, This represents the splicing of feature maps. Represents a time feature descriptor. Represents spatial feature descriptors, This indicates element-wise addition. This represents a 1×1 convolution operation. This means mapping the output to the interval [0, 1]. This represents a two-dimensional attention map.

[0110] 1.4.4 Weighted fusion.

[0111] Multiply the channel weight vector and the two-dimensional attention map element by element to obtain the attention weights with dimensions [32, 192, 2560].

[0112] The attention weights of dimension [32, 192, 2560] are multiplied element-wise with the concatenated feature map of dimension [32, 192, 2560] to obtain a weighted feature map of dimension [32, 192, 2560]. In this way, the signal strength of specific electrodes related to depression at key sampling points can be adaptively enhanced.

[0113] 1.5 Feature dimensionality reduction.

[0114] The weighted feature map with dimensions [32, 192, 2560] is passed through a 1×1 convolution to reduce the number of channels from 192 to 128, and finally outputs a spatiotemporal feature sequence with dimensions [32, 128, 2560].

[0115] Reference Figure 3 As shown, in some optional implementations, step S120 involves performing multimodal feature extraction on the EEG segments of each channel to obtain spatiotemporal and frequency domain features, including: performing short-time Fourier transform on the EEG segments of each channel using a Hanning window to obtain the time-frequency spectrum of each channel; wherein the window length is 256 samples, the overlap length is 128 samples, and the number of Fourier transform points is 512; for each time window of each channel, the power is integrated on the five frequency bands on the time-frequency spectrum to obtain the integrated power of each frequency band, and the integrated power of each frequency band is normalized to the proportion of the sum of the integrated power of the five frequency bands to obtain the relative power of each frequency band. Power; where the five frequency bands include: Delta band [1, 4) Hz, Theta band [4, 8) Hz, Alpha band [8, 13) Hz, Beta band [13, 30) Hz, and Gamma band [30, 45] Hz; calculate the average relative power of each band of each channel over all time windows to obtain a frequency vector with dimension [batch size, 64, 5]; flatten the frequency vector in the dimensions of the number of channels and the number of bands, and project it into a frequency domain feature map with dimension [batch size, 320, 1]; where 320 = 64 channels × 5 bands.

[0116] 2. Frequency domain feature extraction.

[0117] The goal of frequency domain feature extraction is to extract the power features of rhythmic activities closely related to depression, forming a supplementary information source that runs parallel to spatiotemporal features.

[0118] 2.1 Short-time Fourier Transform (STFT).

[0119] STFT can convert a time-domain signal into a frequency-domain signal while retaining information in both the time and frequency dimensions. The specific implementation is as follows:

[0120] Window function: Hanning window; where the window length is 256 samples (corresponding to 1 second), the overlap length is 128 samples (corresponding to 0.5 seconds), and the FFT (Fast Fourier Transform) points are 512;

[0121] Execution method: STFT processing is performed on the EEG segments of each channel to obtain the time-spectrum diagram of each channel;

[0122] 2.2. Frequency band power feature extraction.

[0123] Frequency band division: According to the consensus in neuroscience, different frequency bands of EEG signals are closely related to the pathological mechanism of depression. Therefore, five key frequency bands are defined: Delta band [1, 4) Hz, Theta band [4, 8) Hz, Alpha band [8, 13) Hz, Beta band [13, 30) Hz, and Gamma band [30, 45] Hz.

[0124] Power integration: For the time spectrum of each channel and each time window, perform power integration (summation) in the above five frequency bands to obtain the integrated power of each frequency band;

[0125] Relative power: To avoid the influence of differences in integrated power between individuals, the relative power of each frequency band is calculated, that is, the ratio of the integrated power of any frequency band to the sum of the integrated power of the five frequency bands in a time window of a channel, based on the following formula (3):

[0126] , formula (3);

[0127] in, This represents the integrated power of the target frequency band. This represents the sum of the integrated power across the five frequency bands;

[0128] 2.3 Feature aggregation and formatting.

[0129] Time window averaging: For each channel, calculate the average relative power of each frequency band over all time windows, discard detailed time-varying information, simplify the feature dimensions, and obtain a vector with dimensions [32, 64, 5]; where 64 is the number of channels and 5 is the number of frequency bands;

[0130] Flattening: Flatten the vector of dimension [32, 64, 5] in terms of the number of channels and the number of frequency bands to obtain a frequency domain feature vector of dimension [32, 320]; where 320 = 64 channels × 5 frequency bands;

[0131] Projection: The frequency domain feature vector of dimension [32, 320] is projected into a frequency domain feature map of dimension [32, 320, 1] by the unsqueeze (-1) operation.

[0132] Step S130: Use a state-space model to perform temporal context modeling on the spatiotemporal features to obtain context-aware features.

[0133] Temporal context modeling aims to perform efficient and powerful global context modeling of long-sequence EEG spatiotemporal features with linear computational complexity, and dynamically focus on key information related to depression through selective mechanisms.

[0134] Specifically, step S130 is executed through the temporal context modeling module.

[0135] Figure 4 A flowchart illustrating the principle of timing context modeling according to an embodiment of this application is shown. (Refer to...) Figure 4 As shown, in some optional implementations, step S130 involves using a state-space model to perform temporal context modeling on the spatiotemporal features to obtain context-aware features, including: using four stacked Mamba blocks to form a hierarchical encoder; wherein the hidden dimension of the hierarchical encoder is 512, the expansion factor of the gated MLP is 2, the state dimension of the first two Mamba blocks is 8, and the state dimension of the last two Mamba blocks is 16; taking a spatiotemporal feature sequence with dimensions [batch size, 128, 2560] as input, after each Mamba block of the hierarchical encoder sequentially performs input projection, selective state-space calculation, gated MLP processing, normalization and residual connection operations, the output is a context-aware feature with dimensions [batch size, 512, 2560].

[0136] 1. Layered encoder architecture.

[0137] The hierarchical encoder uses four stacked Mamba blocks to form a hierarchical structure. The specific implementation is as follows:

[0138] Hidden Dimension (D): 512; This dimension is the primary dimension used when features are processed within the module.

[0139] Dimension Expansion (E): In a gated MLP, the expansion factor of the feedforward layer is 2, that is, the intermediate layer dimension is 1024 (512×2).

[0140] State Dimension (N): This is the dimension of the state vector within the State Space Model (SSM), a key hyperparameter controlling the model's memory capacity. The first two Mamba blocks (shallow, local pattern capturers) have a state dimension of 8, a relatively small state dimension designed to efficiently capture fine-grained temporal patterns in sequences (timescales typically less than 500ms), such as micro-dynamics like single spikes or short-range phase locks. The latter two Mamba blocks (deep, global context integrators) have a state dimension of 16, a larger state dimension, leveraging the global receptive field of the State Space Model (SSM) to integrate global long-range contextual dependencies spanning several seconds (greater than 2000ms), such as cross-band coupling between different brain region rhythms and slow fluctuations in brain states at rest. By dynamically allocating model capacity, the hierarchical encoder can simultaneously and accurately model local events and global oscillation patterns in EEG signals, which is crucial for capturing neurodynamic features related to depression that may be distributed across multiple timescales.

[0141] 2. Input and output.

[0142] The input is the spatiotemporal feature sequence output by the spatiotemporal feature extraction unit, with dimensions [32, 128, 2560].

[0143] The output is a context-aware feature with dimensions [32, 512, 2560].

[0144] Reference Figure 4 As shown, in some optional implementations, each Mamba block of the hierarchical encoder sequentially performs input projection, selective state space computation, gated MLP processing, normalization, and residual connection operations, including: converting the input dimension to [batch size, 512, 2560] through pre-layer normalization and a linear layer to obtain an extended input; projecting the extended input to a projected input with a dimension of [batch size, 1024, 2560], and splitting the projected input into two branch inputs, each with a dimension of [batch size, 512, 2560]; and single-stepping the input of one branch input. Three input dependency parameters are generated through three independent linear layers; the input dependency parameters include: step size parameter. Input matrix and output matrix State transition matrix Initialize the system to a HIPPO structured matrix and keep it fixed during training as a priori basis; use a zero-order preservation method to store the continuous system parameters. Discretize into The discretization formula is: , ,in, The identity matrix is ​​represented by [a matrix]. Based on the discretized system parameters, the state space is recursively calculated successively according to the sampling points to obtain a selective state space output with dimensions [batch size, 512, 2560]. The formula is as follows: , ,in, Indicates the first The hidden state of each sampling point Indicates the first The hidden state of each sampling point Indicates the first The single-step output of each sampling point; another branch input is passed through a branch linear layer and the SiLU activation function to generate a filter feature; the selective state space output is multiplied element-wise with the filter feature to obtain the enhanced feature; and the enhanced feature is connected to the extended input residual.

[0145] 3. Mamba block core algorithm.

[0146] 3.1 Layer normalization.

[0147] To ensure the stability of model training and the effective propagation of gradients, a pre-layer normalization is set within each Mamba block. Before the input is projected, the input is normalized to standardize the feature distribution of each sample, accelerate model convergence, and alleviate the gradient vanishing problem.

[0148] 3.2 Input projection.

[0149] By using a linear layer to map the input dimension from 128 to 512, we obtain an extended input with dimensions [batch size, 512, 2560].

[0150] The extended input projection is then extended to 1024, resulting in a final projection input with dimensions [32, 1024, 2560].

[0151] The projection input is split into two branch inputs: one branch input with dimensions [32, 512, 2560] is used for subsequent selective state space calculation, and the other branch input with dimensions [32, 512, 2560] is used for subsequent gated MLP processing.

[0152] 3.3 Selective state-space computation.

[0153] Selective state-space computation is the core of the Mamba block. By dynamically generating state-space model parameters, it enables selective focusing on key information and modeling of long sequence dependencies.

[0154] 3.3.1 Parameter generation.

[0155] Single-step input based on a branch input of dimension [32, 512, 2560]. By dynamically generating three key parameters of the input dependencies of the state-space model (SSM) through three independent linear layers, adaptive adjustment of the input dependency parameters is achieved.

[0156] Step size parameters The output of the linear layer is obtained by processing the Softplus activation function, and the formula is as follows: ; The degree of discretization of the control state evolution can be adaptively adjusted according to the dynamic characteristics of the input signal;

[0157] Input matrix : Output directly through a linear layer, the formula is as follows ; Determine the degree to which the input contributes to the state update;

[0158] Output matrix : Output directly through a linear layer, the formula is as follows , Determine the degree to which the current state contributes to the output;

[0159] State transition matrix Initialized as a HIPPO (High-order Polynomial Projection Operator) structured matrix, it remains fixed during training and serves as the prior basis for the system's long-term memory.

[0160] 3.3.2 Discretization.

[0161] Since EEG signals are discrete-time sequences, it is necessary to convert the continuous-time state-space model parameters... Discretized into discrete-time parameters Discretization is performed using the zero-order hold method, and the discretization formulas are as follows: (4) ~ (5):

[0162] , formula (4);

[0163] , formula (5);

[0164] in, Represents the identity matrix.

[0165] 3.3.3 Recursive computation of state space.

[0166] Based on the discretized parameters, state updates and output calculations are performed according to the following formulas (6) to (7) to achieve long sequence dependency modeling:

[0167] , formula (6);

[0168] , formula (7);

[0169] in, Indicates the first The hidden state of each sampling point Indicates the first The hidden state of each sampling point Indicates the first Single-step output of each sampling point.

[0170] 3.3.4 Selective state-space output.

[0171] The dimensions are [32, 512, 2560].

[0172] 3.4 Gated MLP processing.

[0173] Another branch input with dimensions [32, 512, 2560] is passed through a branched linear layer and the SiLU activation function to generate filtered features.

[0174] 3.5 Feature enhancement.

[0175] The enhanced features are obtained by element-wise multiplying the selective state space output with the filtering features, which serves to filter information and enhance nonlinearity. This is expressed by the following formula (8):

[0176] , formula (8);

[0177] in, This represents the branch input for selective state-space computation. Indicates selective state-space output. This represents the branch input of a gated MLP. This represents the linear transformation of the branch linear layer of a gated MLP. This represents the SiLU activation function. This indicates element-wise multiplication.

[0178] 3.6 Residual connection.

[0179] The extended input and enhanced features are added element-wise to form a residual path.

[0180] 4. Hardware perception algorithm.

[0181] The hierarchical encoder employs a hardware-aware algorithm to optimize computational efficiency during both the training and inference phases, adapting to different hardware environments.

[0182] Training mode (parallel scanning): Utilizing the parallel computing capabilities of the GPU, the recursive computation of the state space is expanded into parallel prefix sum operations, breaking through the sequence dependency limitation of traditional RNNs, achieving efficient parallel training of long sequences, and significantly improving training speed.

[0183] Inference Mode (Loop Mode): During inference deployment, the Markov property of Discrete SSM is utilized, and each step of the computation only requires maintaining the hidden state vector from the previous time step. (The dimension is [batch size, N], where N represents the state dimension). The computational cost is linearly related to the sequence length (O(L)), and the memory usage is constant and unaffected by the sequence length, making it very suitable for real-time EEG analysis on resource-constrained edge devices (such as clinical portable EEG analyzers).

[0184] Step S140: Deep fusion of context-aware features and frequency domain features is performed through a cross-modal fusion mechanism to obtain a fused feature sequence.

[0185] To achieve dimensional alignment and semantic matching of features from the two modalities, context-aware features and frequency domain features are preprocessed and projected.

[0186] Specifically, step S140 can be executed through the cross-modal fusion module.

[0187] Figure 5 A schematic flowchart illustrating the principle of cross-modal fusion according to an embodiment of this application is shown. (Refer to...) Figure 5 As shown, in some optional implementations, step S140 involves performing deep fusion of context-aware features and frequency domain features through a cross-modal fusion mechanism to obtain a fused feature sequence. This includes: reducing the dimensionality of the context-aware features with dimensions [batch size, 512, 2560] using a linear layer to obtain a query vector with dimensions [batch size, 128, 2560]; broadcasting the frequency domain feature map with dimensions [batch size, 320, 1] along the time dimension to obtain aligned frequency domain features with dimensions [batch size, 320, 2560]; reducing the dimensionality of the aligned frequency domain features using two independent linear layers to generate Key vectors and Value vectors with dimensions [batch size, 128, 2560] respectively; and employing a 4-head attention mechanism. The Query, Key, and Value vectors are split into four sub-vectors according to their feature dimensions, forming a multi-head vector set with dimensions [batch size, 4, 2560, 32]. Scaling dot product attention calculation and Softmax normalization are performed on each sub-vector in the multi-head vector set to obtain the attention weight matrix for the four heads. The attention weight matrix is ​​weighted and summed with the corresponding sub-Value vector to obtain four sets of attention features. The four sets of attention features are concatenated and rearranged in dimensions to obtain preliminary fused features with dimensions [batch size, 128, 2560]. Residual connections and layer normalization are performed on the preliminary fused features and the Query vector to output a fused feature sequence with dimensions [batch size, 128, 2560].

[0188] 1. Query vector generation.

[0189] The input is context-aware features with dimensions [32, 512, 2560].

[0190] By reducing the dimension from 512 to 128 through a linear layer, computational overhead is reduced and aligned to the fusion space dimension, resulting in a query vector with dimensions [32, 128, 2560].

[0191] 2. Key and Value vector generation.

[0192] The input is a frequency domain feature map with dimensions [32, 320, 1].

[0193] By broadcasting along the time dimension through the repeat operation, aligned frequency domain features with dimensions [32, 320, 2560] are generated. This establishes a potential correlation between global, time-invariant frequency domain characteristics and the spatiotemporal characteristics of each sampling point.

[0194] The aligned frequency domain features are projected into the Key space and Value space through two independent linear layers, respectively, to obtain Key vectors and Value vectors with dimensions [32, 128, 2560].

[0195] 3. Calculation of multi-head attention mechanism

[0196] A four-head attention mechanism is adopted to achieve deep fusion of features from two modalities through multi-view learning. The specific implementation is as follows:

[0197] 3.1 Vector segmentation.

[0198] The Query, Key, and Value vectors are uniformly divided into 4 sub-vectors along the feature dimension (128 dimensions), with each head having a dimension of 32 (128 / 4). The resulting multi-head vector set has dimensions [32, 4, 2560, 32].

[0199] 3.2 Calculation of attention for scaling dot product.

[0200] Scaling dot product attention is performed independently on the subvectors of each head to capture intra-modal and inter-modal dependencies;

[0201] Attention score calculation: The similarity score between the Query subvector and the Key subvector is calculated according to the following formula (9):

[0202] , formula (9);

[0203] in, For the Query sub-vector, For the Key subvector, For each head dimension, The dimensions are [32, 4, 2560, 2560];

[0204] 3.3 Softmax normalization.

[0205] right The last dimension applies the Softmax function to obtain a normalized attention weight matrix for the four heads. The weight values ​​are between [0,1], and the sum of each row is 1, representing the sum of the weights in the generation of the first attention weight matrix. The relative importance of the frequency domain features of all sampling points to the fusion features of each sampling point;

[0206] 3.4 Weighted summation.

[0207] The attention weight matrix is ​​multiplied by the value subvector, and the attention features of each head are obtained based on the following formula (10):

[0208] , formula (10);

[0209] in, This represents the Value subvector, with an output dimension of [32, 2560, 32].

[0210] 3.5 Feature splicing and dimension rearrangement.

[0211] The attention features of the four heads are concatenated along the feature dimension to restore a 128-dimensional feature vector, resulting in a concatenated tensor of shape [32, 2560, 128].

[0212] The concatenated tensor is rearranged back to the [32, 128, 2560] format through a linear projection layer, aligned with the dimensions of the residual input, to obtain preliminary fused features.

[0213] 4. Residual connection and output.

[0214] Residual connection: The query vector is added element-wise to the output of the attention mechanism (i.e., the initial fused features). This operation ensures that the core information of the spatiotemporal features is not diluted during the fusion process. The attention mechanism learns the incremental information of "modification" and "enhancement" of the spatiotemporal features, which stabilizes the training process and avoids model degradation due to fusion.

[0215] Layer normalization: The results after residual connection are subjected to layer normalization to standardize the feature distribution;

[0216] The final output is a fused feature sequence with dimensions [32, 128, 2560]. This feature sequence retains the long-range context information and spatial topological relationships of the spatiotemporal features, and incorporates the rhythmic activity information of the frequency domain features, thus achieving deep fusion of spatiotemporal and frequency three-dimensional features.

[0217] Step S150: Perform multi-task decision-making based on the fused feature sequence to obtain the depression classification result, the total score of depression severity, and the activation intensity of depression-related brain functional networks.

[0218] Multi-task decision-making uses three parallel task heads (classification task head, regression task head, and semantic auxiliary task head) to jointly optimize model parameters and output depression classification results, severity scores, and brain functional network activation intensity. This ensures classification accuracy while improving the model's generalization ability and clinical applicability.

[0219] Specifically, step S150 can be executed through the multi-task decision module.

[0220] Figure 6 A flowchart illustrating the principle of multi-task decision-making according to an embodiment of this application is shown. (Refer to...) Figure 6As shown, in some optional implementations, step S150, performing multi-task decision-making based on the fused feature sequence to obtain the depression classification result, the total score of depression severity, and the activation intensity of depression-related brain functional networks, includes: performing global average pooling on the fused feature sequence to obtain a global feature vector with dimension [batch size, 128]; inputting the global feature vector into the depression classification task head, which outputs the depression classification result through a fully connected layer, ReLU, Dropout, fully connected layer, and Softmax architecture; wherein, the depression classification task head uses the cross-entropy loss function; and inputting the global feature vector into the depression classification task head. The severity regression task head outputs a total score for depression severity through a fully connected layer, ReLU, and another fully connected layer architecture, employing the Huber loss function. The global feature vector is input into the semantic auxiliary task head, which outputs the activation intensity of depression-related brain function networks through a fully connected layer, Tanh, and another fully connected layer architecture, employing the mean squared error loss function. The depression classification task head, severity task head, and semantic auxiliary task head are jointly trained using a weighted total loss calculated from the cross-entropy loss function, the Huber loss function, and the mean squared error loss function.

[0221] 1. Feature compression.

[0222] The fused feature sequence (dimension [32, 128, 2560]) first passes through a global average pooling layer, which performs global average pooling along the time dimension to compress the time dimension, resulting in a global feature vector with dimensions [32, 128].

[0223] 2. Depression Classification Task Header (Main Task).

[0224] The core objective of the classification task head is to accurately distinguish between healthy individuals and patients with depression; this is the core task. The specific implementation is as follows:

[0225] Network architecture: Global feature vector → Fully connected layer (input dimension 128, output dimension 64) → ReLU activation function → Dropout layer (dropout probability 0.5) → Fully connected layer (input dimension 64, output dimension 2) → Softmax activation function;

[0226] Output: A two-dimensional vector of depression probability, such as [0.15, 0.85], representing the probability that the sample belongs to the "healthy" and "depressed" categories, respectively. The category with the highest probability value is the classification result of the model.

[0227] Loss function: The cross-entropy loss function is used to measure the difference between the predicted probability distribution and the true label (one-hot encoding), and is expressed as the following formula (11):

[0228] , formula (11);

[0229] in, Indicates the number of samples. Indicates sample In category The actual label on the label (1 indicates that it belongs to the category, and 0 indicates that it does not). Indicates sample In category The predicted probability.

[0230] 3. Severity is returned to the task header (auxiliary task 1).

[0231] The goal of the regression task head is to predict the total severity score of patients with depression (based on the HAMD-17 scale), providing a quantitative reference for clinical intervention. The specific implementation is as follows:

[0232] Network architecture: Global feature vector → Fully connected layer (input dimension 128, output dimension 64) → ReLU activation function → Fully connected layer (input dimension 64, output dimension 1).

[0233] Output: Scalar value representing the predicted HAMD-17 total score, typically ranging from 0 to 50, with higher scores indicating greater severity of depression;

[0234] Loss function: The Huber loss function (Smooth L1 Loss) is adopted, which is expressed as the following formula (12):

[0235] , formula (12);

[0236] in, Indicates the number of samples. Indicates sample The actual HAMD-17 total score, Indicates sample The predicted total score, This is a hyperparameter.

[0237] 4. Semantic auxiliary task header (auxiliary task 2).

[0238] The goal of the semantic auxiliary task head is to guide the model to learn the activation strength of brain functional networks related to the pathological mechanisms of depression. The specific implementation is as follows:

[0239] Network architecture: Global feature vector → Fully connected layer (input dimension 128, output dimension 32) → Tanh activation function → Fully connected layer (input dimension 32, output dimension 2).

[0240] Output: Two-dimensional vectors of relative activation intensities, representing the relative activation intensities (normalized to [-1, 1] or [0, 1]) of the Default Mode Network (DMN) and the Frontolimbic System (FL), respectively. These two brain functional networks are closely related to the core pathological mechanisms of depression. This task does not require precise reproduction of fMRI measurements, but rather guides the model to learn intermediate features consistent with known neuroscience consensus.

[0241] Supervision signals: Supervision signals in training data can come from two sources: one is a rough estimate of the activation intensity of brain functional networks based on source localization or functional connectivity analysis; the other is the use of the differences in brain functional network levels between healthy individuals and patients with depression reported in the literature as weak supervision signals.

[0242] Loss function: The mean squared error (MSE) loss function is used to measure the difference between the predicted activation intensity of the brain functional network and the actual supervision signal, and is expressed as the following formula (13):

[0243] , formula (13);

[0244] in, Indicates the number of samples. , Indicates sample The actual activation intensity of brain functional networks , For the sample The predicted activation intensity.

[0245] 5. Joint training and loss weighting.

[0246] To achieve collaborative optimization across multiple tasks, a weighted total loss function is designed to balance the training priorities of each task:

[0247] The total loss function is expressed as the following formula (14):

[0248] , formula (14);

[0249] Training process: The gradient of the total loss function is used to update all parameters of the model simultaneously through the backpropagation algorithm, so as to achieve the coordinated optimization of the three tasks and force the model to learn more general and robust feature representations.

[0250] Step S160: Based on interpretability visualization technology, generate a clinical interpretability atlas corresponding to the depression classification results.

[0251] Interpretable visualization is key to building trust between the model and clinicians. By integrating three techniques—attention weights, gradient-weighted class activation mapping (Grad-CAM), and fully connected layer weight analysis—a clinically meaningful interpretability atlas is generated, providing intuitive and credible explanatory evidence for depression classification results.

[0252] Specifically, step S160 can be executed through the interpretability visualization module.

[0253] Figure 7 A schematic flowchart illustrating the interpretability of one embodiment of this application is shown. (Refer to...) Figure 7 As shown, in some optional implementations, step S160, based on interpretability visualization technology, generates a clinical interpretability atlas corresponding to the depression classification results, including: extracting the attention weight matrix, averaging the attention weight matrix in the head dimension to obtain an aggregated attention matrix, initializing it with an identity matrix, performing recursive calculations based on the aggregated attention matrix, and then performing gradient backpropagation to obtain the contribution of each sampling point to the depression classification results, and generating a sequence importance heatmap based on the contribution; calculating the gradient of each channel of a sample in the spatiotemporal feature sequence of the output before the softmax of the depression classification task head, performing global average pooling of the gradient in the time dimension to obtain a weight vector, calculating the global average of a sample in the spatiotemporal feature sequence in the time dimension to obtain a feature importance vector, and combining the weight vector and the feature importance vector in a weighted manner through a... Each fully connected layer generates scores in the dimension [1, 64]. After normalization, the scores are interpolated onto a two-dimensional scalp plane to generate a brain region contribution heatmap. The weight parameters of the first fully connected layer in the depression classification task head are extracted. The Euclidean norm along the 64-dimensional output dimension is calculated for each row of the weight parameters and the average value is taken. The gradient of the average value with respect to the output probability of the depression category is calculated to obtain a 128-dimensional feature importance score. The projection connection weights of the fused feature sequence and frequency domain feature map are traced back. The feature importance scores are back-projected to each frequency band according to the projection connection weights to obtain the contribution score of each channel in each frequency band. The contribution scores of all channels in each frequency band are summed and averaged to obtain the relative contribution value of each frequency band. A frequency band importance bar chart is generated based on the relative contribution value. The sequence importance heatmap, brain region contribution heatmap, and frequency band importance bar chart are integrated into a clinical interpretability atlas.

[0254] 1. Sequence importance heatmap.

[0255] Sequence importance heatmaps can clearly identify the most critical time segments for depression classification results, helping doctors focus on diagnostically valuable periods of neural activity. The specific implementation is as follows:

[0256] Attention weight extraction: Extract the attention weight matrix of 4 heads from the cross-modal fusion module, with dimensions [32, 4, 2560, 2560];

[0257] Head-dimension average aggregation: The attention weight matrix is ​​averaged along the head dimension to obtain an aggregated attention matrix with dimensions [32, 2560, 2560], thus eliminating the influence of randomness of individual heads;

[0258] Global importance calculation: To calculate the global importance of each sampling point to the final decision, the exponential average of the product of the aggregated attention matrix and the identity matrix is ​​recursively calculated, which is simplified as the following formula (15):

[0259] , formula (15);

[0260] in, Indicates the number of heads. Represents the aggregated attention matrix. Represents the identity matrix. Represents the global attention matrix. The initial value is the identity matrix, and this operation can capture the cumulative contribution of each sampling point to the final decision through multi-step attention passing;

[0261] Gradient backpropagation: Output probability of the "depression" category with respect to the global attention matrix Backpropagation is performed on each column to obtain the gradient value corresponding to each column. The magnitude of the gradient value represents the contribution of the corresponding sampling point to the final classification result (positive value is positive contribution, negative value is negative contribution).

[0262] Map generation: The contribution of each sampling point is plotted as a sequence importance heatmap, with the horizontal axis representing time and the vertical axis representing the magnitude of contribution. The darker the color, the higher the contribution, clearly identifying the most critical time segments for the diagnosis of depression.

[0263] 2. Heat map of brain region contribution.

[0264] Brain region contribution heatmaps can visually show which brain regions (electrode locations) have the most significant activity in relation to depression classification results, thus reflecting the pathological mechanisms of depression. The specific implementation is as follows:

[0265] Gradient calculation: Calculate the gradient of each channel of the output before softmax of the depression classification task head relative to a sample (dimension [1, 128, 2560]) of the spatiotemporal feature sequence (dimension [32, 128, 2560]) output by the multimodal feature extraction module.

[0266] Global average pooling in the time dimension: The gradient is globally average pooled in the time dimension to obtain a weight vector of dimension [1, 128, 1], which represents the importance of each channel;

[0267] Feature importance vector calculation: Calculate the global average of a sample in the spatiotemporal feature sequence in the time dimension to obtain a feature importance vector with dimension [1, 128, 1], which reflects the average activity intensity of each channel;

[0268] Weighted combination and back projection: The weight vector and the feature importance vector are multiplied element-wise to obtain a weighted vector. This weighted vector is then back-projected through a fully connected layer (whose weights correspond to the mapping relationship from 128-dimensional features to 64 electrodes (channels), and are the inverse matrix of the weights of the multimodal feature extraction module from 64 channels to 128-dimensional spatiotemporal features) to generate scores of dimension [1, 64]. Each score corresponds to the contribution level of an electrode (channel).

[0269] Heatmap rendering: The contribution scores of 64 electrodes (channels) are normalized to the [0,1] interval. Based on the 10-20 EEG electrode localization system, the scores are interpolated onto the two-dimensional scalp plane to generate a heatmap of brain region contribution. The darker the color, the higher the contribution of the brain region to decision-making.

[0270] 3. Frequency band importance bar chart.

[0271] Frequency band importance histograms can quantify the relative contribution of different EEG frequency bands to classification decisions, revealing key frequency bands associated with depression. The specific implementation is as follows:

[0272] Weight matrix extraction: Extract the weight parameters w of the first fully connected layer (input dimension 128, output dimension 64) of the depression classification task head, with dimensions [128, 64]; these 128-dimensional inputs are directly derived from the global feature vector, and the global feature vector has a projection relationship with the frequency domain feature map (320 dimensions);

[0273] Feature importance calculation: For each row of the weight parameter w, calculate the Euclidean norm (L2 norm) along the 64 output dimensions (64 dimensions), calculate the norm average, calculate its gradient with respect to the output probability of the depression category, and obtain the 128-dimensional feature importance score.

[0274] Frequency band backprojection: trace the projection connection weights of the 128-dimensional fused features and the 320-dimensional original frequency domain features, and backproject the 128-dimensional feature importance scores to the 5 frequency bands (Delta, Theta, Alpha, Beta, Gamma) according to the projection weights.

[0275] Frequency band contribution aggregation: The contribution scores of all channels in each frequency band are summed and averaged to obtain the relative contribution values ​​of the 5 frequency bands;

[0276] Bar chart generation: Using the frequency band name as the horizontal axis and the contribution value as the vertical axis, a bar chart of frequency band importance is drawn to clearly show the relative importance of each frequency band to the decision on the classification of depression.

[0277] 4. Integration of clinically interpretable atlases.

[0278] The sequence importance heatmap, brain region contribution heatmap, and frequency band importance bar chart are integrated into the clinical visualization interface, and the three complement and corroborate each other.

[0279] Figure 8 A schematic diagram of a depression recognition device based on a state-space model and multimodal fusion according to an embodiment of this application is shown. (Refer to...) Figure 8 As shown, the depression recognition device 800 based on state-space model and multimodal fusion includes:

[0280] The data preprocessing module 810 is used to acquire multi-channel EEG data, and to perform preprocessing and segmentation on the EEG data to obtain EEG segments of fixed duration for each channel.

[0281] The multimodal feature extraction module 820 is used to perform multimodal feature extraction on EEG segments from each channel, and to obtain spatiotemporal features and frequency domain features respectively;

[0282] The temporal context modeling module 830 is used to perform temporal context modeling on spatiotemporal features using a state-space model to obtain context-aware features.

[0283] The cross-modal fusion module 840 is used to perform deep fusion of context-aware features and frequency domain features through a cross-modal fusion mechanism to obtain a fused feature sequence;

[0284] The multi-task decision module 850 is used to perform multi-task decision-making based on the fused feature sequence to obtain the depression classification result, the total score of depression severity, and the activation intensity of depression-related brain functional networks.

[0285] The interpretability visualization module 860 is used to generate a clinical interpretability atlas corresponding to the depression classification results based on interpretability visualization technology.

[0286] In some optional embodiments, in the above-described apparatus, the data preprocessing module 810 is used to: acquire 64-channel EEG data with a sampling rate of 256Hz; perform bandpass filtering on the EEG data of each channel using an infinite impulse response filter to obtain filtered data for each channel; wherein the high-pass cutoff frequency is 0.5Hz and the low-pass cutoff frequency is 45Hz; perform denoising on the filtered data of each channel using an independent component analysis algorithm to obtain denoised data for each channel; perform 64-lead rereference on the denoised data of each channel using the whole-brain average reference method to generate corrected data for each channel; and cut the corrected data of each channel into segments of 10 seconds in length, outputting EEG segments with dimensions [64, 2560]; wherein 64 represents the number of channels and 2560 represents the number of sampling points.

[0287] In some optional embodiments, in the above apparatus, the multimodal feature extraction module 820 is used to: take EEG segments of dimensions [batch size, 64, 2560] as input, capture multi-scale temporal patterns through parallel depthwise separable convolutions including three types of convolutional kernels (3×1, 5×1, and 7×1), and obtain three parallel output feature maps; wherein each convolutional layer is followed by batch normalization and ReLU activation function; after concatenating the three output feature maps along the channel dimension, global average pooling is performed along the temporal and spatial dimensions respectively to generate temporal feature descriptors and spatial feature descriptors; the temporal feature descriptors and spatial feature descriptors are concatenated along the channel dimension, 1 The data is processed by 1×1 convolution and ReLU activation function, then re-segmented and merged after each segment is convolved by 1×1 convolution to generate a channel weight vector with dimensions [batch size, 192, 1]. The temporal and spatial feature descriptors are element-wise summed, followed by 1×1 convolution and a Sigmoid function to generate a two-dimensional spatial attention map with dimensions [batch size, 1, 2560]. The channel weight vector and the two-dimensional spatial attention map are multiplied element-wise to obtain the attention weights. The attention weights are then multiplied element-wise with the three concatenated output feature maps to obtain a weighted feature map. The weighted feature map is then convolved by 1×1 to output a spatiotemporal feature sequence with dimensions [batch size, 128, 2560].

[0288] In some optional embodiments, in the above-described apparatus, the multimodal feature extraction module 820 is used to: perform short-time Fourier transform on the EEG segments of each channel using a Hanning window to obtain the time-spectrum map of each channel; wherein the window length is 256 samples, the overlap length is 128 samples, and the number of Fourier transform points is 512; for each time window of each channel, perform power integration on the five frequency bands on the time-spectrum map to obtain the integrated power of each frequency band, and normalize the integrated power of each frequency band to the proportion of the sum of the integrated power of the five frequency bands to obtain the relative power of each frequency band; wherein the five frequency bands include Includes: Delta band [1, 4) Hz, Theta band [4, 8) Hz, Alpha band [8, 13) Hz, Beta band [13, 30) Hz, and Gamma band [30, 45] Hz; calculate the average relative power of each band of each channel over all time windows to obtain a frequency vector with dimensions [batch size, 64, 5]; flatten the frequency vector in terms of the number of channels and the number of bands, and project it into a frequency domain feature map with dimensions [batch size, 320, 1]; where 320 = 64 channels × 5 bands.

[0289] In some optional embodiments, in the above apparatus, the temporal context modeling module 830 is used to: form a hierarchical encoder by stacking four Mamba blocks; wherein the hidden dimension of the hierarchical encoder is 512, the expansion factor of the gated MLP is 2, the state dimension of the first two Mamba blocks is 8, and the state dimension of the last two Mamba blocks is 16; taking a spatiotemporal feature sequence with dimensions [batch size, 128, 2560] as input, each Mamba block of the hierarchical encoder sequentially performs input projection, selective state space calculation, gated MLP processing, normalization and residual connection operations, and outputs context-aware features with dimensions [batch size, 512, 2560].

[0290] In some optional embodiments, in the above apparatus, the temporal context modeling module 830 is used to: convert the input dimension to [batch size, 512, 2560] through a pre-layer normalization and a linear layer to obtain an extended input; project the extended input to a projected input with dimensions [batch size, 1024, 2560], and split the projected input into two branch inputs, each with dimensions [batch size, 512, 2560]; and process the single-step input of one branch input. Three input dependency parameters are generated through three independent linear layers; the input dependency parameters include: step size parameter. Input matrix and output matrix State transition matrix Initialize the system to a HIPPO structured matrix and keep it fixed during training as a priori basis; use a zero-order preservation method to store the continuous system parameters. Discretize into The discretization formula is: , ,in, The identity matrix is ​​represented by [a matrix]. Based on the discretized system parameters, the state space is recursively calculated successively according to the sampling points to obtain a selective state space output with dimensions [batch size, 512, 2560]. The formula is as follows: , ,in, Indicates the first The hidden state of each sampling point Indicates the first The hidden state of each sampling point Indicates the first The single-step output of each sampling point; another branch input is passed through a branch linear layer and the SiLU activation function to generate a filter feature; the selective state space output is multiplied element-wise with the filter feature to obtain the enhanced feature; and the enhanced feature is connected to the extended input residual.

[0291] In some optional embodiments, in the above apparatus, the cross-modal fusion module 840 is used to: reduce the dimensionality of context-aware features of dimension [batch size, 512, 2560] through a linear layer to obtain a query vector of dimension [batch size, 128, 2560]; broadcast the frequency domain feature map of dimension [batch size, 320, 1] along the time dimension to obtain aligned frequency domain features of dimension [batch size, 320, 2560]; reduce the dimensionality of the aligned frequency domain features through two independent linear layers to generate key and value vectors of dimension [batch size, 128, 2560] respectively; and apply 4-head attention to the query, key, and value vectors. The ue vector is split into 4 sub-vectors according to the feature dimension, forming a multi-head vector set with dimensions [batch size, 4, 2560, 32]. Scaling dot product attention calculation and Softmax normalization are performed on each sub-vector in the multi-head vector set to obtain the attention weight matrix for the 4 heads. The attention weight matrix is ​​weighted and summed with the corresponding sub-value vector to obtain 4 sets of attention features. The 4 sets of attention features are concatenated and rearranged in dimensions to obtain preliminary fused features with dimensions [batch size, 128, 2560]. Residual connections and layer normalization are performed on the preliminary fused features and the Query vector to output a fused feature sequence with dimensions [batch size, 128, 2560].

[0292] In some optional embodiments, in the above apparatus, the multi-task decision module 850 is used to: perform global average pooling on the fused feature sequence to obtain a global feature vector of dimension [batch size, 128]; input the global feature vector into the depression classification task head, which outputs the depression classification result through a fully connected layer, ReLU, Dropout, fully connected layer, and Softmax architecture; wherein the depression classification task head uses the cross-entropy loss function; input the global feature vector into the severity regression task head, which outputs the total severity score of depression through a fully connected layer, ReLU, and fully connected layer architecture; wherein the severity regression task head uses the Huber loss function; input the global feature vector into the semantic auxiliary task head, which outputs the activation intensity of brain functional networks related to depression through a fully connected layer, Tanh, and fully connected layer architecture; wherein the semantic auxiliary task head uses the mean squared error loss function; and jointly train the depression classification task head, the severity task head, and the semantic auxiliary task head using a weighted total loss of the cross-entropy loss function, the Huber loss function, and the mean squared error loss function.

[0293] In some optional implementations, in the above apparatus, the interpretability visualization module 860 is used to: extract the attention weight matrix, average the attention weight matrix in the head dimension to obtain the aggregated attention matrix, initialize it with an identity matrix, perform recursive calculation based on the aggregated attention matrix, and then perform gradient backpropagation to obtain the contribution of each sampling point to the depression classification result, and generate a sequence importance heatmap based on the contribution; calculate the gradient of each channel of a sample of the spatiotemporal feature sequence of the output before the softmax of the depression classification task head, perform global average pooling of the gradient in the time dimension to obtain the weight vector, calculate the global average of a sample of the spatiotemporal feature sequence in the time dimension to obtain the feature importance vector, and generate a dimension of [1] after weighted combination of the weight vector and the feature importance vector through a fully connected layer. The scores of

[64] are normalized and interpolated onto a two-dimensional scalp plane to generate a brain region contribution heatmap. The weight parameters of the first fully connected layer in the depression classification task head are extracted. The Euclidean norm along the 64-dimensional output dimension is calculated for each row of the weight parameters and the average value is taken. The gradient of the average value with respect to the output probability of the depression category is calculated to obtain a 128-dimensional feature importance score. The projection connection weights of the fused feature sequence and frequency domain feature map are traced back. The feature importance scores are back-projected to each frequency band according to the projection connection weights to obtain the contribution score of each channel in each frequency band. The contribution scores of all channels in each frequency band are summed and averaged to obtain the relative contribution value of each frequency band. A frequency band importance bar chart is generated based on the relative contribution value. The sequence importance heatmap, brain region contribution heatmap and frequency band importance bar chart are integrated into a clinical interpretability atlas.

[0294] It should be noted that the depression recognition device 800 based on state space model and multimodal fusion described above can implement the aforementioned depression recognition method based on state space model and multimodal fusion, which will not be elaborated further.

[0295] Figure 9 This invention illustrates a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 9 As shown, the electronic device includes a processor, internal memory, a network interface, and a non-volatile storage medium connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external devices via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a depression recognition method based on a state-space model and multimodal fusion.

[0296] In one embodiment, the electronic device provided in this application includes a memory and a processor. The memory stores a database and a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of a depression recognition method based on a state-space model and multimodal fusion.

[0297] The above is as stated in this application. Figure 8The method for depression recognition based on state-space model and multimodal fusion disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0298] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of a depression identification method based on a state-space model and multimodal fusion.

[0299] It should be noted that the functions or steps that the above-mentioned electronic devices or computer-readable storage media can achieve can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0300] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0301] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0302] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for identifying depression based on state-space model and multimodal fusion, characterized in that, include: Acquire multi-channel EEG data, perform preprocessing and segmentation on the EEG data to obtain EEG segments of fixed duration for each channel; Multimodal feature extraction was performed on EEG segments from each channel to obtain spatiotemporal and frequency domain features, respectively. A state-space model is used to perform temporal context modeling on spatiotemporal features to obtain context-aware features; A cross-modal fusion mechanism is used to perform deep fusion of context-aware features and frequency domain features to obtain a fused feature sequence. Specifically, the context-aware features are used to obtain the Query vector, and the frequency domain features are used to obtain the Key vector and Value vector. A multi-head attention mechanism is used to calculate the Query vector, Key vector, and Value vector to obtain the fused feature sequence. Multi-task decision-making is performed based on fused feature sequences to obtain depression classification results, total depression severity score and activation intensity of depression-related brain functional networks; Based on interpretability visualization technology, a clinical interpretability atlas corresponding to the depression classification results is generated; The multi-task decision-making based on fused feature sequences yields depression classification results, total depression severity scores, and activation intensity of depression-related brain functional networks, including: Perform global average pooling on the fused feature sequence to obtain a global feature vector with dimension [batch size, 128]. The global feature vector is input into the depression classification task head, which outputs the depression classification result through a fully connected layer, ReLU, Dropout, fully connected layer, and Softmax architecture; the depression classification task head uses the cross-entropy loss function. The global feature vector is input into the severity regression task head, which outputs the total severity score of depression through a fully connected layer, ReLU, and another fully connected layer architecture; the severity regression task head uses the Huber loss function. The global feature vector is input into the semantic auxiliary task head, which outputs the activation intensity of brain functional networks related to depression through a fully connected layer, Tanh, and fully connected layer architecture; the semantic auxiliary task head uses the mean squared error loss function. The depression classification task head, severity task head, and semantic auxiliary task head are jointly trained using a weighted total loss calculated from the cross-entropy loss function, Huber loss function, and mean squared error loss function.

2. The depression identification method based on state-space model and multimodal fusion according to claim 1, characterized in that, The acquisition of multi-channel EEG data, including preprocessing and segmentation of the EEG data to obtain fixed-duration EEG segments for each channel, includes: Acquire 64-channel EEG data at a sampling rate of 256Hz; An infinite impulse response filter was used to perform bandpass filtering on the EEG data of each channel to obtain the filtered data of each channel; the high-pass cutoff frequency was 0.5Hz and the low-pass cutoff frequency was 45Hz. Independent component analysis algorithm is used to denoise the filtered data of each channel to obtain denoised data for each channel; The whole-brain average reference method was used to perform 64-lead rereference on the denoised data of each channel to generate corrected data for each channel. The corrected data from each channel is cut into segments of 10 seconds each, and the output EEG segments are of dimension [64, 2560]; where 64 represents the number of channels and 2560 represents the number of sampling points.

3. The depression identification method based on state-space model and multimodal fusion according to claim 2, characterized in that, The process of performing multimodal feature extraction on EEG segments from each channel to obtain spatiotemporal and frequency domain features includes: Using EEG segments of dimensions [batch size, 64, 2560] as input, multi-scale temporal patterns are captured through parallel depthwise separable convolutions with three types of convolutional kernels: 3×1, 5×1, and 7×1, resulting in three parallel output feature maps; each convolutional layer is followed by batch normalization and ReLU activation function. After concatenating the three output feature maps along the channel dimension, global average pooling is performed along the time and spatial dimensions respectively to generate temporal feature descriptors and spatial feature descriptors; The temporal and spatial feature descriptors are concatenated along the channel dimension, followed by 1×1 convolution and ReLU activation. The descriptors are then re-segmented and convolved separately with 1×1 convolution before being merged to generate a channel weight vector with dimensions [batch size, 192, 1]. Temporal and spatial feature descriptors are element-wise summed, 1×1 convolution is performed, and the sigmoid function is applied to generate a two-dimensional spatial attention map with dimensions [batch size, 1, 2560]. The attention weights are obtained by multiplying the channel weight vectors and the two-dimensional attention map element by element, and the weighted feature maps are obtained by multiplying the attention weights and the three concatenated output feature maps element by element. The weighted feature maps are convolved with 1×1 to output a spatiotemporal feature sequence with dimensions [batch size, 128, 2560].

4. The depression identification method based on state-space model and multimodal fusion according to claim 2, characterized in that, The process of performing multimodal feature extraction on EEG segments from each channel to obtain spatiotemporal and frequency domain features includes: The Hanning window was used to perform short-time Fourier transform on the EEG segments of each channel to obtain the time-frequency spectrogram of each channel; the window length was 256 samples, the overlap length was 128 samples, and the number of Fourier transform points was 512. For each time window of each channel, the power of each of the five frequency bands is integrated on the time-spectrum graph to obtain the integrated power of each frequency band. The integrated power of each frequency band is normalized to the proportion of the sum of the integrated power of the five frequency bands to obtain the relative power of each frequency band. The five frequency bands include: Delta band [1, 4) Hz, Theta band [4, 8) Hz, Alpha band [8, 13) Hz, Beta band [13, 30) Hz, and Gamma band [30, 45] Hz. Calculate the average relative power of each frequency band of each channel over all time windows to obtain a frequency vector with dimensions [batch size, 64, 5]. The frequency vector is flattened in terms of the number of channels and the number of frequency bands, and projected into a frequency domain feature map with dimensions [batch size, 320, 1]; where 320 = 64 channels × 5 frequency bands.

5. The depression identification method based on state-space model and multimodal fusion according to claim 3, characterized in that, The process of using a state-space model to perform temporal context modeling on spatiotemporal features yields context-aware features, including: A hierarchical encoder is formed by stacking four Mamba blocks; the hidden dimension of the hierarchical encoder is 512, the expansion factor of the gated MLP is 2, the state dimension of the first two Mamba blocks is 8, and the state dimension of the last two Mamba blocks is 16. Taking a spatiotemporal feature sequence with dimensions [batch size, 128, 2560] as input, each Mamba block of the hierarchical encoder sequentially performs input projection, selective state space computation, gated MLP processing, normalization and residual connection operations, and outputs context-aware features with dimensions [batch size, 512, 2560].

6. The depression identification method based on state-space model and multimodal fusion according to claim 5, characterized in that, Each Mamba block of the hierarchical encoder sequentially performs input projection, selective state-space computation, gated MLP processing, normalization, and residual connection operations, including: The input dimension is transformed to [batch size, 512, 2560] by a pre-layer normalization and a linear layer to obtain an expanded input; Extend the extended input projection to a projected input with dimensions [batch size, 1024, 2560], and split the projected input into two branch inputs, each with dimensions [batch size, 512, 2560]. Single-step input of a branch input Three input dependency parameters are generated through three independent linear layers; the input dependency parameters include: step size parameter. Input matrix and output matrix ; State transition matrix Initialize as a HIPPO structured matrix and keep it fixed during training as a prior basis; Using the zero-order hold method to obtain the parameters of the continuous system Discretize into The discretization formula is: ， ， in, Represents the identity matrix; Based on the discretized system parameters, a recursive state-space calculation is performed successively at each sampling point to obtain a selective state-space output with dimensions [batch size, 512, 2560]. The formula is as follows: ， ， in, Indicates the first The hidden state of each sampling point Indicates the first The hidden state of each sampling point Indicates the first Single-step output of each sampling point; Another branch input is passed through a branched linear layer and the SiLU activation function to generate filtered features. The selective state space output is multiplied element-wise with the filtered features to obtain enhanced features. The enhanced features are then concatenated with the extended input residual.

7. The depression identification method based on state-space model and multimodal fusion according to claim 5, characterized in that, The process involves performing deep fusion of context-aware features and frequency domain features through a cross-modal fusion mechanism to obtain a fused feature sequence, including: The context-aware features with dimensions [batch size, 512, 2560] are reduced in dimension by a linear layer to obtain a query vector with dimensions [batch size, 128, 2560]. Broadcasting the frequency domain feature map with dimensions [batch size, 320, 1] along the time dimension yields aligned frequency domain features with dimensions [batch size, 320, 2560]. The aligned frequency domain features are reduced in dimensionality through two independent linear layers, generating Key and Value vectors with dimensions [batch size, 128, 2560], respectively. Using 4-head attention, the Query, Key, and Value vectors are split into 4 groups of sub-vectors according to the feature dimension, forming a multi-head vector set with dimensions [batch size, 4, 2560, 32]. Scaling dot product attention calculation and Softmax normalization are performed on each sub-vector in the multi-head vector set to obtain the attention weight matrix of 4 heads; The attention weight matrix and the corresponding sub-value vector are weighted and summed to obtain four sets of attention features; The four attention features are concatenated and rearranged in dimensions to obtain preliminary fused features with dimensions [batch size, 128, 2560]. Residual connections and layer normalization are performed on the initial fused features and the Query vector to output a fused feature sequence with dimensions [batch size, 128, 2560].

8. The depression identification method based on state-space model and multimodal fusion according to claim 1, characterized in that, The generation of a clinically interpretable atlas corresponding to the depression classification results based on interpretable visualization technology includes: The attention weight matrix is ​​extracted, and the average of the attention weight matrix over the head dimension is calculated to obtain the aggregate attention matrix. After initialization with the identity matrix, the aggregate attention matrix is ​​recursively calculated and gradient backpropagation is performed to obtain the contribution of each sampling point to the depression classification result. A sequence importance heatmap is generated based on the contribution. The gradient of each channel of a sample in the spatiotemporal feature sequence is calculated before the softmax of the head in the depression classification task. The gradient is then globally averaged and pooled in the time dimension to obtain the weight vector. The global average of a sample in the spatiotemporal feature sequence in the time dimension is calculated to obtain the feature importance vector. The weight vector and the feature importance vector are weighted and combined and then passed through a fully connected layer to generate a score with a dimension of [1, 64]. The score is normalized and then interpolated onto a two-dimensional scalp plane to generate a heatmap of brain region contribution. The weight parameters of the first fully connected layer in the depression classification task head are extracted. For each row of the weight parameters, the Euclidean norm along the 64-dimensional output dimension is calculated and the average value is taken. The gradient of the average value with respect to the output probability of the depression category is calculated to obtain the 128-dimensional feature importance score. The projection connection weights of the fused feature sequence and the frequency domain feature map are traced back. The feature importance score is back-projected to each frequency band according to the projection connection weight to obtain the contribution score of each channel in each frequency band. The contribution scores of all channels in each frequency band are summed and averaged to obtain the relative contribution value of each frequency band. A frequency band importance histogram is generated based on the relative contribution value. The sequence importance heatmap, brain region contribution heatmap, and frequency band importance bar chart were integrated into a clinically interpretable atlas.

9. A depression recognition device based on state-space model and multimodal fusion, characterized in that, include: The data preprocessing module is used to acquire multi-channel EEG data, and to perform preprocessing and segmentation on the EEG data to obtain EEG segments of fixed duration for each channel. The multimodal feature extraction module is used to perform multimodal feature extraction on EEG segments from each channel, acquiring spatiotemporal features and frequency domain features respectively; The temporal context modeling module is used to perform temporal context modeling on spatiotemporal features using a state-space model to obtain context-aware features. The cross-modal fusion module is used to perform deep fusion of context-aware features and frequency domain features through a cross-modal fusion mechanism to obtain a fused feature sequence. Specifically, the query vector is obtained through the context-aware features, and the key and value vectors are obtained through the frequency domain features. A multi-head attention mechanism is used to calculate the query vector, key vector, and value vector to obtain the fused feature sequence. The multi-task decision module is used to perform multi-task decisions based on the fused feature sequence to obtain the depression classification results, the total score of depression severity, and the activation intensity of depression-related brain functional networks. The interpretability visualization module is used to generate clinical interpretability maps corresponding to the depression classification results based on interpretability visualization technology; The multi-task decision module is used to perform global average pooling on the fused feature sequence to obtain a global feature vector with dimensions [batch size, 128]. The global feature vector is input into the depression classification task head, which outputs the depression classification result through a fully connected layer, ReLU, Dropout, fully connected layer, and Softmax architecture; the depression classification task head uses the cross-entropy loss function. The global feature vector is input into the severity regression task head, which outputs the total severity score of depression through a fully connected layer, ReLU, and another fully connected layer architecture; the severity regression task head uses the Huber loss function. The global feature vector is input into the semantic auxiliary task head, which outputs the activation intensity of brain functional networks related to depression through a fully connected layer, Tanh, and fully connected layer architecture; the semantic auxiliary task head uses the mean squared error loss function. The depression classification task head, severity task head, and semantic auxiliary task head are jointly trained using a weighted total loss calculated from the cross-entropy loss function, Huber loss function, and mean squared error loss function.

Images