Multi-modal mood disorder classification method based on fMRI and voice

Abstract

The invention discloses a multi-mode mood disorder classification method based on fMRI and voice, and the method mainly aims at constructing an fMRI and voice interaction fusion network framework which is composed of three stages of network structures, namely, a double-branch feature extraction network used for carrying out the feature extraction of voice data and fMRI signals, and a multi-branch feature extraction network used for carrying out the feature extraction of the voice data and the fMRI signals. The feature interaction fusion network is used for converting the multi-modal features into fusion features containing key information of data of each modal, and the full connection layer network is used for mapping the fusion features to category probabilities. A brain network topological structure and voice features which are related to mood disorder classification tasks and have high discrimination capability are effectively extracted from fMRI and voice data; therefore, cross-modal causal association between the fMRI and the voice data is effectively mined, and high-precision and interpretable mood obstacle recognition and classification are realized.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of psychiatry and data analysis, specifically to a multimodal mood disorder classification method based on fMRI and speech. Background Technology

[0002] Mood disorders (MD) mainly include bipolar disorder and depression, with clinical symptoms including elevated or depressed mood accompanied by cognitive and behavioral changes. Depression is characterized by persistent low mood, reduced behavior, and anhedonia; bipolar disorder alternates between mania and depression, often accompanied by social cognitive dysfunction. The prevalence of MD is increasing year by year with increasing social pressure. Mood disorders are multifactorial and heterogeneous diseases. Currently, diagnosis mainly relies on questionnaires and interviews such as DSM-IV-TR, KSADS-PL (<18 years old), or SCID (≥18 years old). These methods are affected by the assessor's experience, participant cooperation, and questionnaire quality, and lack objective biological indicators. There is an urgent need for detection methods based on objective biomarkers.

[0003] Existing research has shown that fMRI data analysis has unique advantages in revealing functional connectivity (FC) disorders in the brain, but its highly complex structure and data heterogeneity make accurate classification difficult. This complexity is mainly reflected in the multi-layered structure of brain networks and the variability of neural activity, while heterogeneity stems from significant inter-individual differences and different measurement methods. In contrast, behavioral data, due to significant individual differences in subjects' emotional and behavioral performance—differences that are even visually discernible—usually achieves higher diagnostic accuracy in classification tasks. Speech, as a non-invasive, low-cost behavioral signal, is closely related to emotional states and can instantaneously reflect mood changes, making it an ideal behavioral biomarker. However, behavioral data shows significant limitations in reliability and persuasiveness in revealing the underlying biological causes of diseases. For example, influenced by daily environment, a control group may temporarily exhibit sadness, while patients with mood disorders may exhibit happiness or elevation at specific times.

[0004] Given that fMRI and speech respectively carry biological and behavioral information, multimodal fusion has become a trend. However, existing research mostly focuses on spatiotemporally aligned data such as speech-text-video, employing splicing or attention mechanisms for fusion. Since fMRI and speech data are not acquired simultaneously, existing splicing and attention fusion methods cannot capture the correlation and causal relationships between the two modalities. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems existing in the related art.

[0006] The purpose of this invention is to provide a multimodal mood disorder classification method based on fMRI and speech, which effectively mines the cross-modal causal association between fMRI and speech data, and achieves high-precision, interpretable mood disorder identification and classification.

[0007] To achieve the above objectives, the present invention provides a multimodal mood disorder classification method based on fMRI and speech, comprising the following steps:

[0008] S1. Acquire the subject's speech data and fMRI signals, and construct a dataset after preprocessing;

[0009] S2. Construct an fMRI and speech interaction fusion network framework. This fMRI and speech interaction fusion network framework consists of a three-stage network structure, namely: a two-branch feature extraction network for extracting features from speech data and fMRI signals, a feature interaction fusion network for converting multimodal features into fusion features containing key information of each modality, and a fully connected layer network for mapping fusion features to class probabilities.

[0010] S3. Use the constructed dataset to train the fMRI and voice interaction fusion network framework to obtain the trained multimodal mood disorder classification model;

[0011] S4. Acquire the subject's voice data and fMRI signal, and use the multimodal mood disorder classification model to predict the subject's mood disorder classification result.

[0012] A further preferred embodiment of the present invention is that, in step S1, the subject's speech data and fMRI signal are acquired, specifically as follows:

[0013] S11. Recruit subjects, including patients with mood disorders and healthy controls;

[0014] S12. Collect speech samples and fMRI signals from all subjects.

[0015] S13, Definition of the The voice samples of the subjects were ,in , The total number of subjects; Divide into several segments, represented as , Indicates the first The first subject Segment speech data, Represents the total number of speech segments;

[0016] S14, Definition of the The fMRI signal of the subjects was ,in It is the number of vertices. The signal length for each vertex; Divided into multiple time windows, represented as , Representing the The first subject Data for each time window, Indicates the total number of time windows;

[0017] S15, Definition of the The labels of the subjects were .

[0018] Preferably, the dual-branch feature extraction network includes an interpretable speech feature extraction network and a dynamic brain network feature extraction network; in step S2, feature extraction is performed on the speech data and fMRI signal using the dual-branch feature extraction network, specifically as follows:

[0019] S21. Based on the interpretable speech feature extraction network, firstly through the speech backbone network... From the first Voice samples from 10 subjects Extract the CLS token feature from all speech segments ,in Dimensions representing speech features These are the learnable parameters of the speech backbone network; then, a segment attention mechanism is used to capture and automatically extract the dynamic features of speech segments and select the most representative regions as the comprehensive features of the speech modality. ;

[0020] S22. Based on the dynamic brain network feature extraction network, firstly according to the... fMRI signals of the subjects The process involves generating interpretable brain network structures, extracting graph features from each network structure, and finally fusing the extracted features to obtain a comprehensive feature profile of the subject's brain network. .

[0021] Preferably, the dynamic brain network feature extraction network in step S22 includes a sparse graph structure learning module, a graph feature extraction module, and a brain network feature fusion module.

[0022] The sparse graph structure learning module is used to learn from the first... fMRI signals of the subjects This generates interpretable brain network structures; specifically:

[0023] S221. First, define an undirected brain network. ,in This represents the set of vertices, corresponding to the region of interest defined in the brain template. Represents the set of edges in the brain connectome. The adjacency matrix of the graph describes the connection strength between different regions;

[0024] S222, for the first 10 subjects Data for each time window The low-frequency fluctuation amplitude of each time window was extracted as a representative feature of brain region activity. The feature representation constructed from the subjects is as follows: , where 𝜁 represents the dimension of the feature;

[0025] S223. Define a learning function. Learning based on input feature matrix nodes and Functional connections between them, where 𝑚,𝑛∈{1,2, ,}, and Represent node characteristics; A learnable function obtained through neural network fitting, with learnable weights. ;

[0026] Learned graph structure , represented as:

[0027] ;

[0028] Among them, weight By minimizing the loss function Update, represented as:

[0029] ;

[0030] The constraint means that when and Euclidean distance between The larger, The smaller the value;

[0031] S224. Assume that a sparse matrix exists. Through this sparse matrix For adjacency matrix Perform sparsification iterations, adjacency matrix The next state is represented as sparsified. The sparsification iterative process is represented as a... Optimization issues:

[0032] ;

[0033] in, Indicates element-wise multiplication. It is a sparsity parameter used to control the number of non-zero elements;

[0034] Will be The optimization problem is relaxed into an unconstrained convex optimization problem, denoted as:

[0035] ;

[0036] in, This represents the loss constraint term;

[0037] for Add a non-constraint , represented as:

[0038] ;

[0039] By performing an approximate gradient descent method Incremental optimization yields the following optimization formula:

[0040] ;

[0041] in, Indicates the step size;

[0042] The adaptive sparse graph structure learning module dynamically provides features for all nodes. and an adjacency matrix , forming graph data ;

[0043] S225. The graph feature extraction module is used to extract graph features from each brain network structure. It employs the Chebyshev expansion method of the graph Laplacian to control convolutional locality and the range of information propagation, establishing... The Chebyshev polynomial is expressed as:

[0044] ;

[0045] in, The node feature matrix, For learnable parameters, It is the order of the polynomial. Represents the Chebyshev polynomial. Represents the Laplace matrix, It is a degree matrix; , It is the largest eigenvalue of the Laplace matrix. It is the identity matrix; Let represent the Chebyshev polynomial recursive term, where , ;

[0046] By approximating the expansion of Chebyshev polynomials, extracting the nodes centered on each node. Information about adjacent nodes;

[0047] S226. Obtain each brain network feature The data is input into the brain network feature fusion module to fit the dynamics of brain network features and to mine the correlations between time segments. The data update status is as follows:

[0048] Forgotten Gate: ,

[0049] Input Gate: ,

[0050] Candidate cell status: ,

[0051] Cell status update: ,

[0052] Output gate: ,

[0053] Hidden status update: ;

[0054] in, , , and For learnable weights, Indicates the bias term. It is the Sigmoid function;

[0055] Select the last time step Hidden state As a comprehensive feature of the subject's brain network .

[0056] Preferably, in step S21, a segment attention mechanism is used to capture and automatically extract the dynamic features of speech segments and select the most representative region as the comprehensive feature of the speech modality. Specifically:

[0057] A segment attention mechanism is defined, expressed as:

[0058] ;

[0059] ;

[0060] in, , , , Represents learnable parameters; Indicates the current input. This represents the fragment correlation matrix, which is dynamically calculated from the input of the current layer. Representing fragments and fragments The specific relationship between them;

[0061] By exploring the overall relationships between each speech segment, each segment is assigned a weight, and then the mean of all speech segments is obtained as the comprehensive feature of the speech modality. .

[0062] Preferably, in step S2, the feature interaction fusion network transforms the multimodal features into fused features containing key information from each modality's data, specifically as follows:

[0063] S231. Comprehensive features of speech modalities and the comprehensive characteristics of brain networks The features of the two modalities are uniformly divided into feature subspaces of fixed size, which are represented as follows: and , Indicates the number of characteristic subspaces. , These are the feature dimensions of each subspace;

[0064] S232. Concatenate the corresponding feature subspaces to obtain a joint representation of speech and brain features. ,in This represents the total number of dimensions of the spliced ​​features;

[0065] S233, Define the subspace correlation matrix of speech modalities. and practical feature space matrix ,in , Given a learnable weight matrix, calculate speech features. With joint characterization joint correlation matrix The calculation formula is:

[0066] ;

[0067] Effective modal information in the practical feature space is obtained by using a correlation matrix, and a learnable weight matrix is ​​used. The most relevant feature information for convergence and speech modality is represented as follows:

[0068] ;

[0069] in, This is the final representation of the speech modality;

[0070] S234. Obtain the final representation of the brain modality using the same method as in step S233. ;

[0071] S235, The final representation of the speech modality and the final representation of brain modalities By splicing the data, a fused feature containing key information from each modality is obtained. .

[0072] As a preferred approach, unsupervised contrastive prediction coding is introduced into the feature interaction fusion network to control the interdependence of extracted brain modality features and speech modality features, ensuring maximum correlation with the fused features; specifically:

[0073] The correlation between modalities is measured using a scoring function that acts on both the predicted and true vectors, expressed as:

[0074] ;

[0075] in, It is the fitting function of the neural network, which will As input, to generate pairs The prediction;

[0076] Will Incorporating it into the noise contrast estimation framework for speech modality selection A set of random negative samples and a positive sample ,use To predict the h-th future step of the speech modality Then the constraints of the speech modality and the fusion modality are defined as follows:

[0077] ;

[0078] Constraints for brain modalities and fusion modalities were obtained using the same method. and constraints on speech modalities and brain modalities. ;

[0079] The final loss function is defined as:

[0080] .

[0081] Preferably, in step S2, the fully connected layer network maps the fused features to class probabilities, specifically as follows:

[0082] Fusion features Flatten the input layer and feed it into the fully connected network. Use the Sigmoid function to map the output to class probabilities. The cross-entropy loss is defined as:

[0083] ;

[0084] in, It is the sample size. These are real sample labels. This refers to the predicted probability of the model.

[0085] In another aspect, the present invention provides a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to execute the above-described multimodal mood disorder classification method based on fMRI and speech.

[0086] In another aspect, the present invention provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus, and the processor calls logical instructions in the memory to execute the above-mentioned multimodal mood disorder classification method based on fMRI and speech.

[0087] In another aspect, the present invention provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer performs the aforementioned multimodal mood disorder classification method based on fMRI and speech.

[0088] Beneficial Effects: This invention presents a multimodal mood disorder classification method based on fMRI and speech, combining behavioral data and biological markers. For each modality, a convenient and reliable data set (fMRI and speech) is selected for multimodal fusion, jointly improving the recognition accuracy of mood disorders from multiple levels. To preserve key connectivity structures in the brain network, this invention first removes noisy connections using a sparse graph structure learning method. This method uses the ALFF features of each brain region as input to dynamically construct graph data of the brain network, preserving key structural features while obtaining crucial graph feature representations. Furthermore, in the fusion stage, a joint interaction module based on brain and speech features is proposed. This module controls the contribution of each modality's information to the final classification representation, guides the network structure of each branch modality, and considers key information from each other to make judgments. Finally, comprehensive experimental validation was conducted on collected MDA-SF data, showing that the proposed MISF model has significant performance advantages compared to other state-of-the-art methods. In addition, this invention focuses on the alignment properties of the data to obtain more direct and interpretable biomarkers. Attached Figure Description

[0089] Figure 1 This is a schematic diagram of the fMRI and voice interaction fusion network framework constructed in this invention;

[0090] Figure 2 This is a schematic diagram of the feature interaction fusion network of the present invention. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, embodiments of this invention, and should not be construed as limiting the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. In the description of this invention, it should be understood that the terminology used is for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0092] The following is combined Figures 1-2 This invention describes a multimodal mood disorder classification method based on fMRI and speech.

[0093] Example 1: This example provides a multimodal mood disorder classification method based on fMRI and speech. It mainly constructs an interactive fusion framework of speech and fMRI for mood disorder diagnosis, effectively extracting highly discriminative brain network topology and speech features relevant to the mood disorder classification task from fMRI and speech data. Simultaneously, it controls the information interaction and fusion between the two modalities in an interpretable manner. Specifically:

[0094] (1) Design an adaptive Sparse Graph Structure Learning (SGL) module to dynamically generate brain networks in a supervised task-driven manner, resulting in a more refined brain network structure. SGL dynamically evaluates the activity intensity of each brain region based on the low-frequency fluctuation amplitude information of the signal in each brain region, and generates the functional connections of the brain network in sequence. This achieves interpretability at both the node strength and functional connection levels.

[0095] (2) To mine interpretable speech features of mood disorders, an interpretable speech diagnostic module (DisNet) was used to locate speech biomarker regions related to mood disorders. In addition, a segment attention mechanism was designed to explore the correlation of speech segments, enhance the speech features of representative segments, weaken the influence of useless segments, and obtain a comprehensive and complete feature representation.

[0096] (3) Design a Brain-Speech col-laborative Interaction module (BSI). First, the BSI module promotes feature extraction between modalities by guiding interaction between different branches, taking into account each other's information. In addition, the BSI module dynamically adjusts the contribution of each modality to the diagnostic task, thereby achieving efficient information complementarity and in-depth correlation analysis.

[0097] The multimodal mood disorder classification method based on fMRI and speech in this embodiment has the following specific steps:

[0098] S1. Acquire the subject's speech data and fMRI signals, and construct a dataset after preprocessing.

[0099] This embodiment collected a new dataset called Speech and fMRI datasets for Multimodal Depression Analysis (MDA-SF). The collection procedure included participant recruitment, clinical assessment, and speech and fMRI acquisition. The aim was to investigate and analyze specific differences between patients with mood disorders and normal controls, measured by clinical and scale-based assessment criteria.

[0100] This embodiment recruited patients with mood disorders from the inpatient and outpatient departments of a neurology hospital. All patients with depression were diagnosed by physicians with associate senior professional titles. A healthy control group without a history of mood disorders was recruited through internet advertising. Inclusion criteria for voice data collection: 1) Age > 18 years, regardless of gender; 2) Normal hearing; 3) Ability to complete a text reading task in Mandarin. Exclusion criteria: 1) Patients with organic mental illness or other mental disorders; 2) Patients with a medically diagnosed disease that may affect voice (such as influenza, vocal cord disease); 3) Mental disorders caused by psychoactive substances or non-addictive substances.

[0101] Exclusion criteria for fMRI acquisition included: 1) a prior diagnosis of bipolar disorder, schizophrenia, schizoaffective disorder, or other mental disorders associated with other diseases; 2) a history of significant physical illness, especially diseases that may cause changes in brain tissue, such as hypertension, diabetes, or metastatic tumors; 3) a history of neurological abnormalities, including head trauma (clear loss of consciousness ≥5 minutes), seizures, cerebrovascular diseases or brain tumors, and neurodegenerative diseases; 4) conventional MRI exclusion criteria included the presence of metallic implants, claustrophobia, and other contraindications to MRI scanning; 5) pregnant or lactating women; and 6) a history of alcohol, drug abuse, or dependence. All participants were right-handed.

[0102] Voice recording: The voice recording process requires all participants to be in a relatively quiet and enclosed environment. Unauthorized personnel are not allowed to enter the room during recording. Participants are required to remain as natural and comfortable as possible, maintain a distance of approximately 20 cm from the recording pen, and avoid making sounds unrelated to the experiment. Audio will be recorded using a standardized recording pen (Newsmy). The audio content will be a fixed text: "Life is Like a Summer Flower." Each audio segment will be 1-5 minutes long. All audio files will be sampled at 44.1 kHz, with 16-bit PCM, and saved in .wav format.

[0103] The MDA-SF dataset contains 140 normal control samples and 63 samples with mood disorders.

[0104] Based on the constructed MDA-SF dataset, the first... The voice samples of the subjects were ,in , The total number of subjects; Divide into several segments, represented as , Indicates the first The first subject Segment speech data, Represents the total number of speech segments; defines the first... The fMRI signal of the subjects was ,in It is the number of vertices. The signal length for each vertex; Divided into multiple time windows, represented as , Representing the The first subject Data for each time window, Represents the total number of time windows; defines the first... The labels of the subjects were .

[0105] S2. Construct a network framework for fMRI and voice interaction fusion (MISF).

[0106] The goal of MISF is to extract and integrate task-relevant features from input information to form a unified representation, and then use this representation to accurately predict the ground truth reflecting the disease state. Specifically, the multimodal depression diagnosis task mainly consists of three stages: 1) First, the feature extraction stage: realizing the features and 1) Mapping; 2) Secondly, the feature fusion stage forms a mapping relationship. The fusion phase aims to build a function To incorporate multimodal features and Transformed into a comprehensive representation containing key information about each task's orientation. Finally, a fully connected layer network is used to process the labels. To perform a diagnosis, the parameters of all the above network models... and All optimizations were achieved under the drive of supervised classification tasks:

[0107]

[0108] in is the loss function, and D represents the dataset.

[0109] Based on the tasks in the above three stages, the fMRI and voice interaction fusion network framework constructed in this embodiment also consists of three network structures, such as... Figure 1 As shown, they are: a dual-branch feature extraction network for extracting features from speech data and fMRI signals, a feature interaction fusion network for converting multimodal features into fusion features containing key information of each modality, and a fully connected layer network for mapping fusion features to class probabilities.

[0110] (1) Dual-branch feature extraction network

[0111] The dual-branch feature extraction network includes the Speech Interpretive Feature Extraction module (SIE) and the Dynamic Brain Network Feature Extraction module (DBF).

[0112] (1-1) Dynamic brain network feature extraction network

[0113] fMRI provides a profound perspective for the neurobiological analysis of mood disorders, revealing abnormalities at the functional level of the brain. However, the high complexity and noise of its data pose challenges to identifying interpretable brain network functional connectivity features. Therefore, this embodiment designs a dynamic brain network feature extraction network (DBF) focused on analyzing and extracting biomarkers characterizing brain activity and network connectivity. The dynamic brain network feature extraction network includes a sparse graph structure learning module, a graph feature extraction module, and a brain network feature fusion module.

[0114] (1-1-1) Sparse Graph Structure Learning Module

[0115] The sparse graph structure learning module is used to learn sparse graph structures based on the first... fMRI signals of the subjects This generates interpretable brain network structures; specifically:

[0116] Brain map data definition: For an undirected brain network ,in This represents the set of vertices, corresponding to the regions of interest (ROIs) defined in the brain template. Represents the set of edges in the brain connectome. Let be the adjacency matrix of the graph, which describes the connection strength between ROIs.

[0117] This embodiment constructs graph data from two perspectives: 1) node feature extraction; 2) functional connection learning.

[0118] First, because temporal signals have limitations in intuitively representing the intensity of brain activity, they cannot be directly used to describe the intensity of brain activity in a specific region. Therefore, this embodiment equally segments the brain network signal into... Data from several time windows were used to extract the amplitude of low-frequency fluctuations (ALFF) for each time segment as a representative feature of brain region activity, with spontaneous low-frequency fluctuations limited to 0.01–0.08 Hz. For the first time window... 10 subjects Data for each time window The low-frequency fluctuation amplitude of each time window was extracted as a representative feature of brain region activity. The feature representation constructed from the subjects is as follows: , where 𝜁 represents the dimension of the ALFF feature.

[0119] To overcome the limitations of methods relying on prior knowledge such as thresholding and Pearson correlation, or artificially constructed graph adjacency matrices, and to minimize the construction of noisy edges, this embodiment shifts to dynamically learning the graph structure of the brain network, proposing an adaptive sparse graph structure learning (SGL) module. Specifically, a learning function is defined. Learning based on input feature matrix nodes and Functional connections between them, where 𝑚,𝑛∈{1,2, ,}, and Represent node characteristics; A learnable function obtained through neural network fitting, with learnable weights. By measuring the different importance of ALFF features, SGL can further acquire more representative low-frequency amplitude features. The learned graph structure... , represented as:

[0120] ;

[0121] Among them, weight By minimizing the loss function Update, represented as:

[0122] ;

[0123] The constraint means that when and Euclidean distance between The larger, The smaller the value, the better.

[0124] After initially learning the adjacency matrix A, to avoid redundant and complex functional connections in the brain network, the final learned graph structure representation must maintain effective functional connections while minimizing the sparsity of the learned adjacency matrix. That is: 1) The graph structure must retain the most representative functional connections. 2) The graph structure should be as sparse as possible.

[0125] Therefore, this embodiment assigns an adjacency matrix... Norm constraints are added to promote the sparsity of the adjacency matrix. From an optimizability perspective, in order to learn a realistic graph structure, this embodiment adopts a different approach that maintains a trade-off between graph diversity and sparsity, specifically:

[0126] Assume there exists a sparse matrix Through this sparse matrix For adjacency matrix Perform sparsification iterations, adjacency matrix The next state is represented as sparsified. This is equivalent to learning a globally shared edge mask matrix. Then it is applied to all brain networks in the dataset.

[0127] The sparsification iterative process is represented as a... Optimization issues:

[0128] ;

[0129] in, This represents element-wise multiplication (Hadamard product). is a sparsity parameter used to control the number of non-zero elements; the goal is to learn the mask matrix. To ensure the next state More sparse, while ensuring The application will not affect Key functional connections.

[0130] Will be The optimization problem is relaxed into an unconstrained convex optimization problem (Least Absolute Shrinkage and Selection Operator, LASSO) to be solved, denoted as:

[0131] ;

[0132] in, This represents the loss constraint term;

[0133] Furthermore, to ensure that the sparse matrix functions as a positive mask matrix, for Add a non-constraint , represented as:

[0134] ;

[0135] Then, this embodiment performs incremental optimization by performing an Iterative Shrinkage-Thresholding Algo-rithmetic (ISA) step, resulting in the following optimization formula:

[0136] ;

[0137] in, This represents the step size; a sparse matrix can be obtained through hierarchical sparse iteration. It acts as an interpretable mask during the iterative learning of the adjacency matrix, preserving useful connections and removing useless, noisy connections. Combining these steps, the Sparse Graph Structure Learning Module (SGL) is able to generate interpretable brain network structures and node features in mood disorder diagnosis tasks.

[0138] (1-1-2) Graph Feature Extraction Module

[0139] The graph feature extraction module is used to extract graph features from each brain network structure. For each fMRI time-series signal segment to be identified, the adaptive sparse graph structure learning module dynamically provides features for all nodes. and an adjacency matrix , forming graph data Used for subsequent graph convolution calculations.

[0140] This embodiment employs the Chebyshev expansion method of the graph Laplacian to control convolution locality and the range of information propagation, establishing... The Chebyshev polynomial is expressed as:

[0141] ;

[0142] in, The node feature matrix, For learnable parameters, It is the order of the polynomial. Represents the Chebyshev polynomial. Represents the Laplace matrix, It is a degree matrix; , It is the largest eigenvalue of the Laplace matrix. It is the identity matrix; Let represent the Chebyshev polynomial recursive term, where , ;

[0143] By approximating the expansion of Chebyshev polynomials, extracting the nodes centered on each node. Information about adjacent nodes.

[0144] (1-1-3) Brain Network Feature Fusion Module

[0145] Finally, each brain network feature obtained The data is input into the brain network feature fusion module to fit the dynamics of brain network features and to mine the correlations between time segments. The data update status is as follows:

[0146] Forgotten Gate: ,

[0147] Input Gate: ,

[0148] Candidate cell status: ,

[0149] Cell status update: ,

[0150] Output gate: ,

[0151] Hidden status update: ;

[0152] in, , , and For learnable weights, Indicates the bias term. It is the Sigmoid function;

[0153] This embodiment selects the last time step. Hidden state As a comprehensive feature of the subject's brain network .

[0154] (1-2) Interpretable speech feature extraction network

[0155] The interpretable speech feature extraction network is responsible for automatically extracting interpretable acoustic features from a patient's speech signal. By identifying differences in features across different speech segments, it reflects the patient's emotional state and provides a behavioral explanation for subsequent analysis.

[0156] Speech, as a direct medium for expressing an individual's emotions and cognitive state, offers a non-invasive monitoring method. This behavioral data, specifically speech, has been proven to be closely related to mood disorders; changes in its characteristics can reflect emotional and cognitive abnormalities in patients, providing a new perspective for the diagnosis and monitoring of mood disorders. Based on this, this embodiment first uses the DisNet model as the backbone network to extract key acoustic features related to mood disorders from speech recordings. Specifically, the subject's speech segments are input into the speech backbone network. In the middle, from the first Voice samples from 10 subjects Extract the CLS token feature from all speech segments ,in Dimensions representing speech features These are the learnable parameters of the speech backbone network; then, a segment attention mechanism is used to capture and automatically extract the dynamic features of speech segments and select the most representative regions as the comprehensive features of the speech modality. The DisNet method enables MISF to effectively capture the unique pronunciation attributes of each speech segment in a high-dimensional space.

[0157] Unlike the voting mechanism that determines the final label of a subject, in the comprehensive assessment of acquiring speech representations, the differences in the content contained in each segment and the dynamic changes in pronunciation can all have varying degrees of impact on the final diagnosis of mood disorders. Therefore, not all segments are equally important, and each segment has a different predictive impact on mood disorders. In order to accurately capture and automatically extract the dynamic features of speech segments and select the most representative regions, this embodiment defines a segment attention mechanism, expressed as follows:

[0158] ;

[0159] ;

[0160] in, , , , Represents learnable parameters; Indicates the current input. This represents the fragment correlation matrix, which is dynamically calculated from the input of the current layer. Representing fragments and fragments The specific relationship between them;

[0161] By exploring the overall relationships between each speech segment, each segment is assigned a weight to preserve the features of important segments and reduce the influence of noisy segments. The mean of all speech segments is then obtained as the comprehensive feature of the speech modality. .

[0162] (2) Feature Interaction Fusion Network (BSI)

[0163] Given that the two modalities convey information at different levels, another core task of this embodiment is to explore how to effectively capture their complementary and collaborative relationships. Therefore, to enhance the understanding of the connections between the two modalities, this embodiment designs a Feature Interaction Fusion Network (BSI) that integrates brain and speech features. By deeply modeling the interaction relationships between different modalities, BSI enables the branching model to more accurately analyze the correlation between information from one modality and information from another. In short, the introduction of BSI allows the branching model to apply knowledge learned from one modality to guide information processing in another modality, effectively integrating and optimizing cross-modal information and reducing heterogeneity between the two modalities. Furthermore, BSI can also measure the relationship between the two modalities, significantly improving the interpretability of the method.

[0164] like Figure 2 As shown, the comprehensive features of speech modalities and the comprehensive characteristics of brain networks The features of the two modalities are uniformly divided into feature subspaces of fixed size, which are represented as follows: and , Indicates the number of characteristic subspaces. , These are the feature dimensions of each subspace. Next, the corresponding feature subspaces are concatenated to obtain a joint representation of speech and brain features. ,in This represents the total number of dimensions of the spliced ​​features.

[0165] In the fusion of brain and speech, the fusion features must include the most critical feature information for the diagnostic task from each modality. Therefore, calculating speech features is crucial. With joint characterization joint correlation matrix This is used to measure the subtle relationships between modalities.

[0166] For speech modalities, the subspace correlation matrix of the speech modality is defined. and practical feature space matrix ,in , Given a learnable weight matrix, calculate speech features. With joint characterization joint correlation matrix The calculation formula is:

[0167] ;

[0168] in, This embodiment represents the correlation between fused features and individual speech modalities. To preserve the positive correlation between features, only the positive correlation weights of the two modalities are obtained. Joint correlation matrix. It not only represents the semantic measure of correlation across modalities, but also provides the semantic measure of correlation within the same modality. Joint correlation matrix A high score indicates that the corresponding samples are strongly correlated within the same modality and other modalities. Therefore, the proposed method can effectively utilize the complementarity and intramodal correlation between brain and speech modalities.

[0169] Then, effective modal information in the practical feature space is obtained by using the correlation matrix, and a learnable weight matrix is ​​used. The most relevant feature information for convergence and speech modality is represented as follows:

[0170] ;

[0171] in, This is the final representation of the speech modality.

[0172] Similar to the methods described above, the final representation of brain modalities is obtained using the same approach. Finally, the final representation of the speech modality is... and the final representation of brain modalities By splicing the data, a fused feature containing key information from each modality is obtained. .

[0173] For modality fusion tasks, the fusion process should adhere to two principles: intermodal correlation, meaning the features extracted from both the brain and speech branches should be closely related to the task and contain common representational information from both. The degree of this common information reflects the dependency between features from different modalities. Intramodal invariance, the fusion result must capture invariant cues within each modality to ensure the effectiveness of multimodal fusion.

[0174] Accordingly, this embodiment introduces unsupervised contrastive predictive coding (CPC) into the BSI module to control the extracted brain modality features and speech modality features to be interdependent and have the highest correlation with the fused modality features. Specifically, this embodiment uses a scoring function acting on the predicted vector and the true vector to measure the correlation between modalities. Specifically:

[0175] The correlation between modalities is measured using a scoring function that acts on both the predicted and true vectors, expressed as:

[0176] ;

[0177] in, It is the fitting function of the neural network, which will As input, to generate pairs The prediction;

[0178] Will It is incorporated into the Noise-Contrastive Estimation framework (NCE).

[0179] Taking speech modality as an example again, for fusion modality features, the speech modality is selected first. A set of random negative samples and a positive sample Then use To predict the h-th future step of the speech modality The constraints for speech modalities and fusion modalities are defined as follows:

[0180] ;

[0181] Capable of constraining fusion results Inversely predict cross-modal representations to pass on more mode-invariant information. Furthermore, by aligning the predictions with each modality, the model is able to determine how much information it should receive from each modality. Loss function Higher scores indicate a greater degree of dependency between features; conversely, lower scores indicate greater independence between features. Based on this, through refined cross-modal prediction, features with the same semantics across different modalities can be extracted from each other. In particular, Constraining the information flow of each modality to the fusion network controls the amount of information in the fusion result. By evaluating the relationships between modalities, the quality of feature combinations can be measured.

[0182] Constraints for brain modalities and fusion modalities were obtained using the same method. and constraints on speech modalities and brain modalities. ;

[0183] The final loss function is defined as:

[0184] .

[0185] Feature Interaction Fusion Network (BSI) transforms multimodal features into fused features containing key information from each modality, aiming to enhance the correlation and complementarity between features. BSI allows the MISF model to dynamically adjust the degree of attention given to each modality feature, achieving joint optimization and correlation measurement of different modality features.

[0186] (3) Fully connected layer network (classification stage)

[0187] Fusion features Flatten the input layer and feed it into the fully connected network. Use the Sigmoid function to map the output to class probabilities. The cross-entropy loss is defined as:

[0188] ;

[0189] in, It is the sample size. These are real sample labels. This refers to the predicted probability of the model.

[0190] S3. Use the constructed dataset to train the fMRI and voice interaction fusion network framework to obtain the trained multimodal mood disorder classification model;

[0191] S4. Acquire the subject's voice data and fMRI signal, and use the multimodal mood disorder classification model to predict the subject's mood disorder classification result.

[0192] The performance of the multimodal mood disorder classification method based on fMRI and speech, as well as the MISF model, of this invention was validated and analyzed.

[0193] The hyperparameters of the model are designed as follows:

[0194] 1) For the DBF module, the fMRI fragment length is 95, and the ALFF feature length is 18. Brain template selection: AAL90 The Chebyshev convolution has a kernel size of 12 and a polynomial order of 3. A two-layer LSTM is used, with a hidden state dimension of 384.

[0195] 2) The parameters for the DisNet module are set as follows: For the LFB module, the speech segment length is set to 6.46k, and the parameters are... , The STFT uses a 25µm window and a 10µm jump length. and For the HRE module, the patch block size is 8×8, and the multi-head attention count is 6. , and .

[0196] 3) In the BSI module, the subspace parameter is: 𝑠=4; the Adam optimizer is used, the batch size is 4, and the initial learning rate is 5e-5. Mel-spectral features are used. All were processed at a sampling rate of 16kHz, a window length of 25k, and a jump length of 10k.

[0197] To verify the effectiveness of the multimodal fusion model MISF, this embodiment compares it with several other methods on the MDA-SF dataset. This includes performance comparisons of each branch model in the single-modal case and performance comparisons in the fusion modality. All methods underwent stratified five-fold cross-validation, and the mean and standard deviation of F1-score, precision, and recall are reported. The comparison results are shown in Table 1.

[0198] In the speech modality, traditional machine learning methods were selected, including Support Vector Machine (SVM) and Random Forest (RF), and the openSMILE toolkit was used to extract low-level descriptor features composed of MFCC, COMPARE, and eGeMAPS. For deep learning methods, the DisNet used in this embodiment was compared with several models that use speech features as input to highlight the performance differences between branches. These included (i) CNN-based methods: DepAudioNet, voice-CNNs; (ii) RNN-based methods: vlad-GRU, ConvbiLSTM, ABAFnet; and (iii) Transformer-based models: Wav2Vec2.0.

[0199] In the brain modality, traditional machine learning methods, including Support Vector Machine (SVM) and Random Forest (RF), were selected, with Pearson correlation coefficient (PCC) chosen as the feature input. Specifically, the lower triangular portion of the symmetric matrix was extracted and flattened. Deep learning methods were used, including CNN-based methods such as ASD_DiagNet and BrainNetCnn, and GCN-based methods such as SHA-GCN, BrainGNN, GroupINN, and BrainUSL. The comparison results are shown in Table 1.

[0200] Table 1 Comparison of MISF with other methods

[0201]

[0202] The results show that: 1. Both branch models used in the MISF model exhibit superior performance. First, DisNet's classification performance on the speech modality is similar to that of the fine-tuned large speech model Wav2Vec2.0, indicating that the speech branch model has strong key acoustic feature extraction capabilities and does not affect the subsequent multimodal fusion process. Similarly, DBF also outperforms existing brain network classification models on the MDA-SF dataset; for example, its F1 score is 1.6% higher than that of the BrainNetCnn method.

[0203] 2. On the MDA-SF dataset, the fusion model MISF significantly outperforms other single-modal models in F1 score, further demonstrating MISF's advantages. This performance improvement is mainly attributed to the synergistic effect between modalities. Through the effective guidance and constraints of the BSI module, the model is able to extract more multi-level features related to mood disorders from high-dimensional signals. After weighing the relationship between the two, the model can make more accurate predictions.

[0204] 3. Notably, while maintaining the same dataset partitioning, each branch model was trained independently. Subsequently, after loading pre-trained parameters (PretrainedMISF), these models were fine-tuned. Observing the results at this point, a significant improvement in the F1 score can be seen. This improvement is mainly attributed to the smaller sample size of the dataset. By loading pre-trained parameters, the features of each branch modality are more refined, which not only helps the BSI module to perform modality fusion and interaction judgment more effectively, but also significantly improves the accuracy of the classification task.

[0205] Ablation experiments were conducted to verify whether the brain network structure constructed by SGL in this embodiment is more effective than other methods; and whether the BSI fusion module effectively guides the fusion of multimodal features.

[0206] (1) Effectiveness of Sparse Graph Structure Learning Component (SGL)

[0207] SGL aims to extract key connectivity differences and activity patterns between different regions of the brain network in a supervised learning manner to identify functional connectivity markers associated with mood disorders. Therefore, this embodiment focuses on designing several variations of the model to demonstrate the role of the SGL component.

[0208] (i) This indicates that SGL is not used to construct edges in the brain functional network, but rather PCC is used to construct the key structural connections of the brain network.

[0209] (ii) This indicates that instead of using SGL for learning sparse adjacency matrices, the key hypergraph structures of the brain network are aggregated through multi-graph clustering.

[0210] The experimental results are shown in Table 2. The SGL module achieves the best brain network classification performance. This is mainly attributed to the SGL module's ability to dynamically construct the sparse adjacency matrix of the brain network under the guidance of supervised loss by optimizing learnable parameters, thus effectively improving the performance of the subsequent classification model. Furthermore, the results also verify that this task-oriented sparse adjacency matrix construction method can more effectively capture key functional connections compared to edge-based clustering methods.

[0211] Table 2. Influence of different graph structure generation methods

[0212]

[0213] Table 2 also shows the impact of positive and negative correlational connections on classification results in various adjacency matrix construction methods. The results indicate that considering only either positive or negative correlational connections leads to the loss of useful functional connectivity information in mood disorder classification.

[0214] (2) Effectiveness of the Feature Fusion Module BSI

[0215] The BSI module aims to manage the fusion of speech and brain modality features, ensuring accurate measurement of relationships between modalities while preserving task-related information within each modality, and selecting features most closely related to mood disorders. Therefore, this embodiment selected the following fusion methods to demonstrate the positive effects of the BSI module. The results are shown in Table 3.

[0216] (i) Additive fusion: By adding features from different modalities element by element, the balanced contribution of each modal feature is emphasized.

[0217] (ii) Concatenation and fusion: The feature vectors of different modalities are concatenated end-to-end to preserve the independence of each modal information.

[0218] (iii) Self-attention fusion: The features of each modality can self-regulate their contribution through a weight allocation mechanism.

[0219] (iv) Cross-attention fusion: Through the cross-attention mechanism, features of one modality are allowed to query features of another modality, realizing dynamic interaction and fusion of information between modalities.

[0220] Table 3. Impact of non-modal fusion methods

[0221]

[0222] As shown in Table 3, different fusion methods have a significant impact on model performance. First, additive fusion and self-attention fusion performed relatively poorly across all metrics, with F1 scores of 0.753 and 0.754, respectively. Concatenation fusion and cross-attention fusion also did not show superior performance in terms of F1 scores. These results demonstrate that the compared fusion methods may lead to excessive dominance of single-modal information. Even with self-attention mechanisms, the complementary nature of different modalities may be lost during the weighting process, failing to achieve the complementary goal of multimodal fusion.

[0223] The BSI fusion method performed exceptionally well, outperforming other fusion strategies across all evaluation metrics. This demonstrates the efficiency of BSI fusion in capturing and utilizing complementary information between different modalities, thereby effectively improving the model's classification capabilities. This effect likely stems from its ability to integrate multimodal information at a deeper level and dynamically control the relevance of each modality and its impact on the final result. These findings highlight the importance of selecting an appropriate fusion strategy to improve the overall performance of the model when performing complex classification tasks.

[0224] Example 2: This example provides a non-transitory computer-readable storage medium storing computer instructions that cause a computer to execute a multimodal mood disorder classification method based on fMRI and speech. The method includes the following steps:

[0225] S1. Acquire the subject's speech data and fMRI signals, and construct a dataset after preprocessing;

[0226] S2. Construct an fMRI and speech interaction fusion network framework. This fMRI and speech interaction fusion network framework consists of a three-stage network structure, namely: a two-branch feature extraction network for extracting features from speech data and fMRI signals, a feature interaction fusion network for converting multimodal features into fusion features containing key information of each modality, and a fully connected layer network for mapping fusion features to class probabilities.

[0227] S3. Use the constructed dataset to train the fMRI and voice interaction fusion network framework to obtain the trained multimodal mood disorder classification model;

[0228] S4. Acquire the subject's voice data and fMRI signal, and use the multimodal mood disorder classification model to predict the subject's mood disorder classification result.

[0229] Example 3: This example provides an electronic device that may include a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor can call logical instructions from the memory to execute a multimodal mood disorder classification method based on fMRI and speech. This method includes the following steps:

[0230] S1. Acquire the subject's speech data and fMRI signals, and construct a dataset after preprocessing;

[0231] S2. Construct an fMRI and speech interaction fusion network framework. This fMRI and speech interaction fusion network framework consists of a three-stage network structure, namely: a two-branch feature extraction network for extracting features from speech data and fMRI signals, a feature interaction fusion network for converting multimodal features into fusion features containing key information of each modality, and a fully connected layer network for mapping fusion features to class probabilities.

[0232] S3. Use the constructed dataset to train the fMRI and voice interaction fusion network framework to obtain the trained multimodal mood disorder classification model;

[0233] S4. Acquire the subject's voice data and fMRI signal, and use the multimodal mood disorder classification model to predict the subject's mood disorder classification result.

[0234] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0235] Example 4: This example provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform a multimodal mood disorder classification method based on fMRI and speech. The method includes the following steps:

[0236] S1. Acquire the subject's speech data and fMRI signals, and construct a dataset after preprocessing;

[0237] S2. Construct an fMRI and speech interaction fusion network framework. This fMRI and speech interaction fusion network framework consists of a three-stage network structure, namely: a two-branch feature extraction network for extracting features from speech data and fMRI signals, a feature interaction fusion network for converting multimodal features into fusion features containing key information of each modality, and a fully connected layer network for mapping fusion features to class probabilities.

[0238] S3. Use the constructed dataset to train the fMRI and voice interaction fusion network framework to obtain the trained multimodal mood disorder classification model;

[0239] S4. Acquire the subject's voice data and fMRI signal, and use the multimodal mood disorder classification model to predict the subject's mood disorder classification result.

[0240] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0241] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal mood disorder classification method based on fMRI and speech, characterized in that, Includes the following steps: S1. Acquire the subject's speech data and fMRI signals, and construct a dataset after preprocessing; S2. Construct an fMRI and speech interaction fusion network framework. This fMRI and speech interaction fusion network framework consists of a three-stage network structure, namely: a two-branch feature extraction network for extracting features from speech data and fMRI signals, a feature interaction fusion network for converting multimodal features into fusion features containing key information of each modality, and a fully connected layer network for mapping fusion features to class probabilities. S3. Use the constructed dataset to train the fMRI and voice interaction fusion network framework to obtain the trained multimodal mood disorder classification model; S4. Acquire the subject's voice data and fMRI signal, and use the multimodal mood disorder classification model to predict the subject's mood disorder classification result.

2. The multimodal mood disorder classification method based on fMRI and speech according to claim 1, characterized in that, Step S1 involves acquiring the subject's speech data and fMRI signal, specifically as follows: S11. Recruit subjects, including patients with mood disorders and healthy controls; S12. Collect speech samples and fMRI signals from all subjects. S13, Definition of the The voice samples of the subjects were ,in , The total number of subjects; Divided into several segments, represented as , Indicates the first The first subject Segment speech data, Represents the total number of speech segments; S14, Definition of the The fMRI signal of the subjects was ,in It is the number of vertices. The signal length for each vertex; Divided into multiple time windows, represented as , Representing the The first subject Data for each time window, Indicates the total number of time windows; S15, Definition of the The labels of the subjects were .

3. The multimodal mood disorder classification method based on fMRI and speech according to claim 2, characterized in that, The dual-branch feature extraction network includes an interpretable speech feature extraction network and a dynamic brain network feature extraction network; in step S2, the dual-branch feature extraction network is used to extract features from the speech data and fMRI signal, specifically as follows: S21. Based on the interpretable speech feature extraction network, firstly through the speech backbone network... From the first Voice samples from 10 subjects Extract the CLS token feature from all speech segments ,in Dimensions representing speech features These are the learnable parameters of the speech backbone network; then, a segment attention mechanism is used to capture and automatically extract the dynamic features of speech segments and select the most representative regions as the comprehensive features of the speech modality. ; S22. Based on the dynamic brain network feature extraction network, firstly according to the... fMRI signals of 1 subject The process involves generating interpretable brain network structures, extracting graph features from each network structure, and finally fusing the extracted features to obtain a comprehensive feature profile of the subject's brain network. .

4. The multimodal mood disorder classification method based on fMRI and speech according to claim 3, characterized in that, The dynamic brain network feature extraction network mentioned in step S22 includes a sparse graph structure learning module, a graph feature extraction module, and a brain network feature fusion module; The sparse graph structure learning module is used to learn from the first... fMRI signals of 1 subject This generates interpretable brain network structures; specifically: S221. First, define an undirected brain network. ,in This represents the set of vertices, corresponding to the region of interest defined in the brain template. Represents the set of edges in the brain connectome. The adjacency matrix of the graph describes the connection strength between different regions; S222, for the first 10 subjects Data for each time window The low-frequency fluctuation amplitude of each time window was extracted as a representative feature of brain region activity. The feature representation constructed from the subjects is as follows: , where 𝜁 represents the dimension of the feature; S223. Define a learning function. Learning based on input feature matrix nodes and Functional connections between them, where 𝑚,𝑛∈{1,2, ,}, and Represent node characteristics; A learnable function obtained through neural network fitting, with learnable weights. ; Learned graph structure , represented as: ； Among them, weight By minimizing the loss function Update, represented as: ； The constraint means that when and Euclidean distance between The larger, The smaller the value; S224. Assume that a sparse matrix exists. Through this sparse matrix For adjacency matrix Perform sparsification iterations, adjacency matrix The next state is represented as sparsified. The sparsification iterative process is represented as a... Optimization issues: ； in, Indicates element-wise multiplication. It is a sparsity parameter used to control the number of non-zero elements; Will be The optimization problem is relaxed into an unconstrained convex optimization problem, denoted as: ； in, This represents the loss constraint term; for Add a non-constraint , represented as: ； By performing an approximate gradient descent method Incremental optimization yields the following optimization formula: ； in, Indicates the step size; The adaptive sparse graph structure learning module dynamically provides features for all nodes. and an adjacency matrix , forming graph data ; S225. The graph feature extraction module is used to extract graph features from each brain network structure. It employs the Chebyshev expansion method of the graph Laplacian to control convolutional locality and the range of information propagation, establishing... The Chebyshev polynomial is expressed as: ； in, The node feature matrix, For learnable parameters, It is the order of the polynomial. Represents the Chebyshev polynomial. Represents the Laplace matrix, It is a degree matrix; , It is the largest eigenvalue of the Laplace matrix. It is the identity matrix; Let represent the Chebyshev polynomial recursive term, where , ; By approximating the expansion of Chebyshev polynomials, extracting the nodes centered on each node. Information about adjacent nodes; S226. Obtain each brain network feature The data is input into the brain network feature fusion module to fit the dynamics of brain network features and to mine the correlations between time segments. The data update status is as follows: Forgotten Gate: , Input Gate: , Candidate cell status: , Cell status update: , Output gate: , Hidden status update: ; in, , , and For learnable weights, Indicates the bias term. It is the Sigmoid function; Select the last time step Hidden state Comprehensive characteristics of the subject's brain network .

5. The multimodal mood disorder classification method based on fMRI and speech according to claim 3, characterized in that, In step S21, a segment attention mechanism is used to capture and automatically extract the dynamic features of speech segments, and the most representative region is selected as the comprehensive feature of the speech modality. Specifically: A segment attention mechanism is defined, expressed as: ； ； in, , , , Represents learnable parameters; Indicates the current input. This represents the fragment correlation matrix, which is dynamically calculated from the input of the current layer. Representing fragments and fragments The specific relationship between them; By exploring the overall relationships between each speech segment, each segment is assigned a weight, and then the mean of all speech segments is obtained as the comprehensive feature of the speech modality. .

6. The multimodal mood disorder classification method based on fMRI and speech according to claim 3, characterized in that, In step S2, the feature interaction fusion network transforms the multimodal features into fused features containing key information from each modality's data, specifically as follows: S231. Comprehensive features of speech modalities and the comprehensive characteristics of brain networks The features of the two modalities are uniformly divided into feature subspaces of fixed size, which are represented as follows: and , Indicates the number of characteristic subspaces. , These are the feature dimensions of each subspace; S232. Concatenate the corresponding feature subspaces to obtain a joint representation of speech and brain features. ,in This represents the total number of dimensions of the spliced ​​features; S233, Define the subspace correlation matrix of speech modalities. and practical feature space matrix ,in , Given a learnable weight matrix, calculate speech features. With joint characterization joint correlation matrix The calculation formula is: ； Effective modal information in the practical feature space is obtained by using a correlation matrix, and a learnable weight matrix is ​​used. The most relevant feature information for convergence and speech modality is represented as follows: ； in, This is the final representation of the speech modality; S234. Obtain the final representation of the brain modality using the same method as in step S233. ; S235, The final representation of the speech modality and the final representation of brain modalities By splicing the data, a fused feature containing key information from each modality is obtained. .

7. The multimodal mood disorder classification method based on fMRI and speech according to claim 6, characterized in that, Unsupervised contrastive predictive coding is introduced into the feature interaction fusion network to control the interdependence of extracted brain modality features and speech modality features, ensuring maximum correlation with the fused features; specifically: The correlation between modalities is measured using a scoring function that acts on both the predicted and true vectors, expressed as: ； in, It is the fitting function of the neural network, which will As input, to generate pairs The prediction; Will Incorporating it into the noise contrast estimation framework for speech modality selection A set of random negative samples and a positive sample ,use To predict the h-th future step of the speech modality Then the constraints of the speech modality and the fusion modality are defined as follows: ； Constraints for brain modalities and fusion modalities were obtained using the same method. and constraints on speech modalities and brain modalities. ; The final loss function is defined as: 。 8. The multimodal mood disorder classification method based on fMRI and speech according to claim 6, characterized in that, In step S2, the fully connected layer network maps the fused features to class probabilities, specifically as follows: Fusion features Flatten the input layer and feed it into the fully connected network. Use the Sigmoid function to map the output to class probabilities. The cross-entropy loss is defined as: ； in, It is the sample size. These are real sample labels. This refers to the predicted probability of the model.

9. A non-transitory computer-readable storage medium, characterized in that, It stores computer instructions that cause the computer to execute the multimodal mood disorder classification method based on fMRI and speech as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor calls logical instructions from the memory to execute the multimodal mood disorder classification method based on fMRI and speech as described in any one of claims 1-7.

Images