Mental disease classification method fusing prior knowledge and multi-mode self-attention mechanism

Abstract

The invention discloses a mental disease classification method fusing prior knowledge and a multi-mode self-attention mechanism, and belongs to the technical field of medical image processing. In order to solve the problems that in the current mental disease classification process, biomarker information is dispersed, and reproducibility is insufficient, multi-mode nerve image data, field priori knowledge and the intelligent analysis ability of a large language model are fully fused, and existing mental disease biomarkers are integrated, optimized and utilized. And related medical knowledge is efficiently extracted and integrated by using a large language model, so that efficient and accurate classification of mental diseases is realized. The method provides an innovative technical path and theoretical support for precise diagnosis and mechanism research of mental disorders.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a method for classifying mental illnesses that integrates prior knowledge and multimodal self-attention mechanisms. Background Technology

[0002] Extensive research has been conducted on the classification of mental illnesses and the discovery of biomarkers. Using multimodal data such as neuroimaging, researchers attempt to reveal the biological basis of mental disorders and screen for potential biomarkers associated with these diseases. Theoretically, these biomarkers could help improve the accuracy of disease identification, risk prediction, and personalized treatment. However, existing research often remains at the preliminary discovery stage, failing to apply discovered biomarkers to new data. Particularly in multicenter, large-sample data environments, insufficient consistency between different studies and low reproducibility of biomarkers mean that their clinical value has not yet been fully explored.

[0003] Meanwhile, the rapid development of artificial intelligence technology, especially Large Language Models (LLMs), has brought new opportunities for the analysis and knowledge discovery of medical data. Language models possess powerful text understanding and generation capabilities, enabling them to efficiently process and integrate large amounts of medical literature and multi-source heterogeneous data, extracting and mining hidden patterns and correlations. However, exploration into how to deeply integrate the capabilities of large language models with the mining and integration of biomarkers for mental illness remains relatively limited. The unique advantages of large language models in knowledge summarization, information extraction, and text mining have not yet been fully utilized to promote the systematic review and optimized utilization of biomarkers for mental illness.

[0004] Therefore, there is an urgent need to develop new technical solutions. On the one hand, it is crucial to fully leverage the advantages of large language models in knowledge extraction and data integration, systematically integrating and efficiently utilizing existing information on mental illness biomarkers scattered across different studies, thereby enhancing the applicability of biomarkers to new data. On the other hand, it is essential to construct a framework capable of integrating and efficiently utilizing existing information on mental illness biomarkers scattered across different studies, achieving collaborative analysis of cross-modal features through multimodal data fusion, thus improving the applicability of biomarkers to new data and different populations. This has significant theoretical and practical implications for promoting biological research and translational medicine in mental illness, and reducing the subjectivity and uncertainty of traditional classification methods. Summary of the Invention

[0005] This invention addresses practical problems in the classification of mental illnesses and the discovery of biomarkers by proposing a method for classifying mental illnesses that integrates prior knowledge with multimodal self-attention mechanisms. This method fully leverages the intelligent analysis capabilities of multimodal neuroimaging data and large language models, achieving efficient and objective classification of mental illnesses through the systematic integration and optimization of existing biomarker information. This invention provides an innovative technical path and theoretical basis for precision medicine and mechanistic research in the field of neuropsychiatric disorders.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A method for predicting mental illness that integrates prior knowledge and multimodal self-attention mechanisms includes the following steps:

[0008] Step 1: Collect brain imaging data from a multi-center dataset, including resting-state functional magnetic resonance imaging (fMRI) data and structural magnetic resonance imaging (SMRI) data. Preprocess the subjects' brain imaging data to obtain standard functional and structural brain images of the subjects. Use brain imaging analysis methods to estimate brain imaging measures from the brain imaging data and construct a brain imaging magnetic resonance imaging dataset.

[0009] Step 1.1, preprocessing includes inter-layer correction, head movement correction, spatial normalization, and spatial smoothing;

[0010] Step 1.2: Based on the preprocessed resting-state functional magnetic resonance imaging, obtain the decomposition-based or region-of-interest-based methods. A functional connectivity matrix is ​​constructed by calculating the Pearson correlation coefficient between each functional region of interest and its corresponding time series. ;

[0011] Step 1.3: Based on the preprocessed structural magnetic resonance images, the whole brain is divided into sections using a brain structural atlas. For each structural region of interest, the gray matter volume within that region is extracted. A structural connectivity matrix is ​​constructed by calculating the KL divergence between the gray matter volumes of different regions of interest. .

[0012] Step 2: Prior knowledge of four biomarkers—functional connectivity, functional region of interest, structural connectivity, and structural region of interest—is obtained from the prior knowledge described in the published literature through a large language model. This prior knowledge is then processed by a pre-trained bidirectional encoder to construct a prior knowledge vector.

[0013] Step 2.1: Use Gemini 2.0 to analyze the descriptions of biomarkers in published literature on the corresponding mental illnesses, obtaining the functional modalities. Each function connection and There are 1 functional region of interest, where the functional connection is represented as "[region of interest 1]-[region of interest 2]" and the functional region of interest is represented as "[region of interest]";

[0014] Using Gemini 2.0, we analyzed the descriptions of biomarkers in published literature on corresponding mental illnesses, and obtained the structural modalities. Each structural connection and There are structural regions of interest, where structural connections are represented as "[region of interest 1]-[region of interest 2]", and four biomarkers are used as prior knowledge of this mental illness, with the structural regions of interest represented as "[region of interest]";

[0015] Step 2.2 involves processing this prior knowledge using the pre-trained BioBERT natural language representation base model to construct a prior knowledge vector. The specific steps are as follows:

[0016] According to statistics from Gemini 2.0 Of the functionally connected biomarkers, for the first The text description of the function connection is "[Region of Interest 1]-[Region of Interest 2]". This is broken down into two regions of interest, "[Region of Interest 1]" and "[Region of Interest 2]". Each region of interest is then decomposed into words and mapped using a pre-trained BioBERT natural language representation model. The word vector corresponding to each word in each region of interest is the prior vector. The prior vectors for the two regions of interest are represented as follows: and ,in and Indicates the number of lexical elements. Indicates feature dimension;

[0017] According to statistics from Gemini 2.0 Among the biomarkers of functional regions of interest, for the first The text description of a region of interest is "[Region of Interest]". This region of interest is decomposed into words and mapped using a pre-trained BioBERT natural language representation model. The word vector corresponding to each word in the region of interest is the prior vector. The prior vector of this region of interest is represented as follows: ,in Indicates the number of lexical elements. Indicates feature dimension;

[0018] According to statistics from Gemini 2.0 Among the biomarkers of structural connections, for the first The structured text is described as "[Region of Interest 1] - [Region of Interest 2]". It is then split into two Regions of Interest (ROIs): "[Region of Interest 1]" and "[Region of Interest 2]". These ROI names are decomposed into words using a pre-trained BioBERT natural language representation model and mapped. The word vector corresponding to each word in each ROI is the prior vector. The prior vectors for these two ROIs are represented as follows: and ,in and Indicates the number of lexical elements. Indicates feature dimension;

[0019] According to statistics from Gemini 2.0 Among the biomarkers of the structural region of interest, for the first... The text description of a structural region of interest is "[Region of Interest]". This region of interest is decomposed into words and mapped using a pre-trained BioBERT natural language representation model. The word vector corresponding to each word in the region of interest is the prior vector. The prior vector of this region of interest is represented as follows: ,in Indicates the number of lexical elements. Indicates feature dimension;

[0020] For whole-brain segmentation using decomposition-based or region-of-interest-based methods The region of interest for the function will be the first The names of regions of interest (ROIs) are input into the pre-trained BioBERT natural language representation base model, decomposed into words, and mapped. The word vector corresponding to each word in the ROI is the prior vector. The prior vector of this ROI is represented as follows: And among them, D represents the number of lexical units and the feature dimension.

[0021] For the use of brain structure atlases to divide the whole brain The region of interest for the structure will be the first The names of the regions of interest (ROIs) are input into the pre-trained BioBERT natural language representation base model, decomposed into words, and mapped. The word vector corresponding to each word in the ROI is the prior vector. The prior vector of this ROI is represented as follows: And among them, D represents the number of lexical units and the feature dimension.

[0022] Step 2.3 calculates the similarity between the names of biomarkers described in published literature and the names of regions of interest. The specific steps are as follows:

[0023] For functional connectivity priors, calculate and Similarity matrix ,use express The first in Line number Column elements:

[0024] ;

[0025] in, express The first in OK, express The first in Rows; Matrix Quantified and The similarity strength of corresponding word elements is indicated by a higher value, which means the two are more similar, and a lower value means the greater the difference.

[0026] Next, calculate the F1 score for "[Region of Interest 1]". :

[0027] ;

[0028] ;

[0029] ;

[0030] The F1 score matrix is ​​obtained by comparing the prior vector of "[Region of Interest 1]" in the functional connectivity of all biomarkers with the prior vector of the region of interest name based on decomposition or region of interest method partitioning. ;

[0031] Similarly, calculation and The F1 score matrix can be obtained by comparing the prior vector of "[Region of Interest 2]" in the functional connectivity of all biomarkers with the prior vectors of the names of all regions of interest based on decomposition or region of interest methods. ;

[0032] Next, construct a weighted connection similarity matrix. For the first Functional links:

[0033] ;

[0034] in, Represent the outer product; then calculate the maximum value under each function connection matrix:

[0035] ;

[0036] therefore, The elements in the file are updated as follows:

[0037] ;

[0038] in, Indicates the first The frequency of occurrence of each functional connection;

[0039] Ultimately, the matrix contains prior information about functional connectivity. Represented as:

[0040] ;

[0041] in, This means normalizing the elements in the matrix to the range [0, 1]. A matrix representing prior information of functional connections Elements in;

[0042] For the prior of the region of functional interest, calculate and Similarity matrix ,use express The Middle Line number Column elements:

[0043] ;

[0044] in, express The first in OK, express The first in Rows; Matrix Quantified and The similarity strength of corresponding word elements is indicated by a higher value, which means the two are more similar, and a lower value means the greater the difference.

[0045] Next, calculate the F1 score. :

[0046] ;

[0047] ;

[0048] ;

[0049] Obtain the functional region of interest prior vectors for all biomarkers and the F1 score matrix of the prior vectors of the names of all regions of interest partitioned based on decomposition or region of interest methods. ;

[0050] Then, calculate the maximum value for each vector:

[0051] ;

[0052] therefore, The elements in the file are updated as follows:

[0053] ;

[0054] in, Indicates the first Frequency of occurrence of each functional region of interest;

[0055] Ultimately, the vector contains prior information about the functional region of interest. Represented as:

[0056] ;

[0057] in, This means normalizing the elements in the vector to the range [0, 1]. Vector representing prior information of the functional region of interest Elements in;

[0058] For structural connection priors, calculation and Similarity matrix ,use express The first in Line number Column elements:

[0059] ;

[0060] in, express The first in OK, express The first in Rows; Matrix Quantified and The similarity strength of corresponding word elements is indicated by a higher value, which means the two are more similar, and a lower value means the greater the difference.

[0061] Next, calculate the F1 score for "[Region of Interest 1]". :

[0062] ;

[0063] ;

[0064] ;

[0065] The F1 score matrix is ​​obtained by comparing the prior vector of "[Region of Interest 1]" in the structural connectivity of all biomarkers with the prior vector of the region of interest name in all brain structural maps. ;

[0066] Similarly, calculation and This yields the F1 score matrix of the prior vectors for "[Region of Interest 2]" in the structural connectivity of all biomarkers and the prior vectors for the names of regions of interest in all brain structural maps. ;

[0067] Next, construct a weighted connection similarity matrix. For the first Each structural connection:

[0068] ;

[0069] in, Represent the outer product; then calculate the maximum value under each structural link matrix:

[0070] ;

[0071] therefore, The elements in the file are updated as follows:

[0072] ;

[0073] in, Indicates the first The frequency of structural connections appearing in prior knowledge;

[0074] Ultimately, the matrix contains prior information about structural connections. Represented as:

[0075] ;

[0076] in, This means normalizing the elements in the matrix to the range [0, 1]. A matrix representing prior information about structural connections Elements in;

[0077] For the prior of the structural region of interest, calculate and Similarity matrix ,use express The Middle Line number Column elements:

[0078] ;

[0079] in, express The first in OK, express The first in Rows; Matrix Quantified and The similarity strength of corresponding word elements is indicated by a higher value, which means the two are more similar, and a lower value means the greater the difference.

[0080] Next, calculate the F1 score. :

[0081] ;

[0082] ;

[0083] ;

[0084] The F1 score matrix is ​​obtained by combining the prior vectors of the structural regions of interest for all biomarkers with the prior vectors of the region of interest names for all brain structure atlases. ;

[0085] Then, calculate the maximum value for each vector:

[0086] ;

[0087] therefore, The elements in the file are updated as follows:

[0088] ;

[0089] in, Indicates the first The frequency of occurrence of each structural region of interest;

[0090] Finally, the vector containing prior information about the structural region of interest. Represented as:

[0091] ;

[0092] in, This means normalizing the elements in the vector to the range [0, 1]. Vector representing prior information of the region of interest in the structure The elements in.

[0093] Step 3: Train a mental illness classification model guided by prior knowledge. This model consists of three parts: a multi-channel self-attention transformation and fusion module for the functional connectivity matrix, a multi-channel self-attention transformation and fusion module for the structural connectivity matrix, and a classification module.

[0094] Step 3.1: Generate the functional connectivity matrix of all subjects in the training dataset. The input function connects to the multi-channel self-attention transformation and fusion module, i.e. hour ; the structural connectivity matrix of all subjects in the training dataset The multi-channel self-attention transformation and fusion module of the input structure connection matrix, i.e. hour .

[0095] Step 3.2, the specific content of the multi-channel self-attention transformation and fusion module of the functional connectivity matrix is ​​as follows: First, calculate the input of the functional connectivity matrix at the... The Q, K, and V matrices of the layer:

[0096] ;

[0097] ;

[0098] ;

[0099] in, This indicates the total number of layers in the functional connectivity model. It is the output of the functional connection matrix in the previous layer; , and For linear layer weights;

[0100] Then, the outputs of the three channels are calculated separately for the three self-attention mechanisms:

[0101] ;

[0102] ;

[0103] ;

[0104] in, , , , for The dimension; Represents the Hadamard product;

[0105] The outputs from the three channels are fused using a channel attention mechanism:

[0106] ;

[0107] ;

[0108] ;

[0109] in, , , ; and For linear layer weights;

[0110] The output obtained after the feedforward network layer is represented as follows:

[0111] ;

[0112] ;

[0113] ;

[0114] in, , , ; and For linear layer weights,

[0115] Step 3.3, the specific steps of the multi-channel self-attention transformation and fusion module of the structure connection matrix are: calculating the structure connection matrix input at the... The Q, K, and V matrices of the layer:

[0116] ;

[0117] ;

[0118] ;

[0119] in, This indicates the total number of layers in the structural connection model. It is the output of the structure connection matrix in the previous layer; , and For linear layer weights;

[0120] Then, the outputs of the three channels are calculated separately for the three self-attention mechanisms:

[0121] ;

[0122] ;

[0123] ;

[0124] in, , , , for The dimension; Represents the Hadamard product;

[0125] The outputs from the three channels are fused using a channel attention mechanism:

[0126] ;

[0127] ;

[0128] ;

[0129] in, , , ; and For linear layer weights;

[0130] The output obtained after the feedforward network layer is represented as follows:

[0131] ;

[0132] ;

[0133] ;

[0134] in, , , ; and For linear layer weights,

[0135] Step 3.4, the specific content of the output classification result is as follows: For and The output classification probability is:

[0136] ;

[0137] in, and Represents the classification layer; the loss function is calculated using cross-entropy:

[0138] ;

[0139] in, Indicates the number of samples. Indicates the number of categories; Indicates label index, Represents classification probability The index.

[0140] Step 4: Preprocess and extract brain imaging features from subjects with unknown diagnostic labels; then input them into a pre-trained mental illness classification model guided by prior knowledge for analysis, and output the classification result of whether they are mental illnesses.

[0141] Compared with the prior art, the present invention has the following advantages:

[0142] 1) This invention systematically integrates and optimizes the knowledge of mental illness-related biomarkers scattered across different studies and literature by introducing a large language model and domain prior knowledge. Utilizing the knowledge extraction, semantic understanding, and information fusion capabilities of the large language model, it enhances the consistency and reproducibility of biomarker information, overcoming the problems of inconsistent biomarker results and difficulty in reusing them in existing methods.

[0143] 2) This invention can efficiently integrate multimodal neuroimaging data and prior knowledge, thereby significantly improving the accuracy and stability of the classification of mental illnesses and healthy populations, and providing strong technical support and objective indicators for the study of the mechanisms of mental disorders. Attached Figure Description

[0144] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0145] Figure 1 This is a schematic diagram of the mental illness classification method that integrates prior knowledge and multimodal self-attention mechanisms according to the present invention.

[0146] Figure 2 A schematic diagram of biomarker results in schizophrenia data;

[0147] Figure 3 A schematic diagram of biomarker results from autism spectrum disorder data. Detailed Implementation

[0148] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.

[0149] Example 1

[0150] A classification method for mental illnesses that integrates prior knowledge and multimodal self-attention mechanisms includes the following steps:

[0151] Step 1: Collect brain imaging data from a multi-center dataset, including resting-state functional magnetic resonance imaging (fMRI) data and structural magnetic resonance imaging (SMRI) data. Preprocess the subjects' brain imaging data to obtain standard functional and structural brain images of the subjects. Use brain imaging analysis methods to estimate brain imaging measures from the brain imaging data and construct a brain imaging magnetic resonance imaging dataset.

[0152] Step 2: Prior knowledge of four biomarkers—functional connectivity, functional region of interest, structural connectivity, and structural region of interest—is obtained from the prior knowledge described in the published literature through a large language model. This prior knowledge is then processed by a pre-trained bidirectional encoder to construct a prior knowledge vector.

[0153] Step 3: Train a mental illness classification model guided by prior knowledge. This model consists of three parts: a multi-channel self-attention transformation and fusion module for the functional connectivity matrix, a multi-channel self-attention transformation and fusion module for the structural connectivity matrix, and a classification module.

[0154] Step 4: Preprocess and extract brain imaging features from subjects with unknown diagnostic labels; then input them into a pre-trained mental illness classification model guided by prior knowledge for analysis, and output the classification result of whether they are mental illnesses.

[0155] This invention's mental illness classification method, which integrates prior knowledge and multimodal self-attention mechanisms, was applied to the classification of mental disorders, distinguishing between patients and healthy controls (HC) in multiple datasets. For schizophrenia (SZ), we combined data from the bipolar and schizophrenia network on intermediate phenotypes (BSNIP), the center for biomedical research excellence (COBRE), the functional biomedical informatics research network (FBIRN), and the Maryland psychiatric research center (MPRC), totaling 957 participants (544 HC and 413 SZ). , The hyperparameters were optimized using five-fold nested cross-validation. Table 1 shows the results of the hyperparameter optimization. As can be seen from the results in Table 2, the present invention outperforms other methods under all four evaluation metrics. This confirms that the mental illness classification method integrating prior knowledge and multimodal self-attention mechanisms of the present invention performs better than other methods. The input of the SHAP analysis model was used. and The importance of each element in the classification accuracy.

[0156] Figure 2 This presents the biomarker results for the schizophrenia dataset. Figure 2 (A) represents the prior functional connection calculated from prior knowledge; Figure 2 (B) represents the functional connections that have a high impact on classification accuracy, obtained from SHAP analysis of the model. Figure 2 (C) represents the common connections between the prior functional connections and the functional connections obtained from SHAP analysis. Figure 2 (D) represents the prior structure connection calculated from prior knowledge; Figure 2 (E) represents the structural connections that have a high impact on classification accuracy, obtained from SHAP analysis of the model. Figure 2 (F) represents the common connections between the prior structural connections and the structural connections obtained from SHAP analysis.

[0157] Table 1 Hyperparameter Optimization Table

[0158]

[0159] Table 2 Comparison of different classification methods

[0160]

[0161] Example 2

[0162] A classification method for mental illnesses that integrates prior knowledge and multimodal self-attention mechanisms includes the following steps:

[0163] Step 1: Collect brain imaging data from a multi-center dataset, including resting-state functional magnetic resonance imaging (fMRI) data and structural magnetic resonance imaging (SMRI) data. Preprocess the subjects' brain imaging data to obtain standard functional and structural brain images of the subjects. Use brain imaging analysis methods to estimate brain imaging measures from the brain imaging data and construct a brain imaging magnetic resonance imaging dataset.

[0164] Step 2: Prior knowledge of four biomarkers—functional connectivity, functional region of interest, structural connectivity, and structural region of interest—is obtained from the prior knowledge described in the published literature through a large language model. This prior knowledge is then processed by a pre-trained bidirectional encoder to construct a prior knowledge vector.

[0165] Step 3: Train a mental illness classification model guided by prior knowledge. This model consists of three parts: a multi-channel self-attention transformation and fusion module for the functional connectivity matrix, a multi-channel self-attention transformation and fusion module for the structural connectivity matrix, and a classification module.

[0166] Step 4: Preprocess and extract brain imaging features from subjects with unknown diagnostic labels; then input them into a pre-trained mental illness classification model guided by prior knowledge for analysis, and output the classification result of whether they are mental illnesses.

[0167] The mental illness classification method of this invention, which integrates prior knowledge and multimodal self-attention mechanisms, was applied to the classification of mental disorders, distinguishing between patients and HCs in multiple datasets. For autism spectrum disorder (ASD), we used the Autism Brain Imaging Data Exchange (ABIDE) I and II datasets, which included 1528 subjects (829 HCs and 699 ASDs). , The hyperparameters were optimized using five-fold nested cross-validation. Table 3 shows the results of the hyperparameter optimization. As can be seen from the results in Table 4, the present invention outperforms other methods under all four evaluation metrics. This confirms that the mental illness classification method of the present invention, which integrates prior knowledge and multimodal self-attention mechanisms, performs better than other methods. The input of the SHAP analysis model was used. and The importance of each element in the classification accuracy.

[0168] Figure 3 This presents the biomarker results for the autism spectrum disorder dataset. Figure 3 (A) represents the prior functional connection calculated from prior knowledge; Figure 3 (B) represents the functional connections that have a high impact on classification accuracy, obtained from SHAP analysis of the model. Figure 3 (C) represents the common connections between the prior functional connections and the functional connections obtained from SHAP analysis. Figure 3 (D) represents the prior structure connection calculated from prior knowledge; Figure 3 (E) represents the structural connections that have a high impact on classification accuracy, obtained from SHAP analysis of the model. Figure 3 (F) represents the common connections between the prior structural connections and the structural connections obtained from SHAP analysis.

[0169] Table 3 Hyperparameter Optimization Table

[0170]

[0171] Table 4 Comparison of Different Classification Methods

[0172]

[0173] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A classification method for mental illnesses that integrates prior knowledge and multimodal self-attention mechanisms, characterized in that... Includes the following steps: Step 1: Collect brain imaging data from a multi-center dataset, including resting-state functional magnetic resonance imaging (fMRI) data and structural magnetic resonance imaging (SMRI) data. Preprocess the subjects' brain imaging data to obtain standard functional and structural brain images of the subjects. Use brain imaging analysis methods to estimate brain imaging measures from the brain imaging data and construct a brain imaging magnetic resonance imaging dataset. Step 2: Prior knowledge of four biomarkers—functional connectivity, functional region of interest, structural connectivity, and structural region of interest—is obtained from the prior knowledge described in the published literature through a large language model. This prior knowledge is then processed by a pre-trained bidirectional encoder to construct a prior knowledge vector. Step 3: Train a mental illness classification model guided by prior knowledge. This model consists of three parts: a multi-channel self-attention transformation and fusion module for the functional connectivity matrix, a multi-channel self-attention transformation and fusion module for the structural connectivity matrix, and a classification module. Step 4: Preprocess and extract brain imaging features from subjects with unknown diagnostic labels; then input them into a pre-trained mental illness classification model guided by prior knowledge for analysis, and output the classification result of whether they are mental illnesses.

2. The mental illness classification method that integrates prior knowledge and multimodal self-attention mechanisms according to claim 1, characterized in that: The specific steps in step 1 of constructing the brain imaging magnetic resonance imaging dataset are as follows: Step 1.1, preprocessing includes inter-layer correction, head movement correction, spatial normalization, and spatial smoothing; Step 1.2: Based on the preprocessed resting-state functional magnetic resonance imaging, obtain the decomposition-based or region-of-interest-based methods. A functional connectivity matrix is ​​constructed by calculating the Pearson correlation coefficient between each functional region of interest and its corresponding time series. ; Step 1.3: Based on the preprocessed structural magnetic resonance images, the whole brain is divided into sections using a brain structural atlas. For each structural region of interest, the gray matter volume within that region is extracted. A structural connectivity matrix is ​​constructed by calculating the KL divergence between the gray matter volumes of different regions of interest. .

3. The mental illness classification method that integrates prior knowledge and multimodal self-attention mechanisms according to claim 1, characterized in that: The construction of the prior knowledge vector in step 2 specifically involves: Step 2.1: Use Gemini 2.0 to analyze the descriptions of biomarkers in published literature on the corresponding mental illnesses, obtaining the functional modalities. Each function connection and There are 1 functional region of interest, where the functional connection is represented as "[region of interest 1]-[region of interest 2]" and the functional region of interest is represented as "[region of interest]"; Using Gemini 2.0, we analyzed the descriptions of biomarkers in published literature on corresponding mental illnesses, and obtained the structural modalities. Each structural connection and A structural region of interest is defined, where structural connections are represented as "[region of interest 1]-[region of interest 2]". Four biomarkers are considered as prior knowledge of this mental illness, and the structural region of interest is represented as "[region of interest]". Step 2.2 involves processing this prior knowledge using the pre-trained BioBERT natural language representation base model to construct a prior knowledge vector. The specific steps are as follows: According to statistics from Gemini 2.0, Among the biomarkers of functional connectivity, for the first The text description of the function connection is "[Region of Interest 1]-[Region of Interest 2]". This is broken down into two regions of interest, "[Region of Interest 1]" and "[Region of Interest 2]". Each region of interest is then decomposed into words and mapped using a pre-trained BioBERT natural language representation model. The word vector corresponding to each word in each region of interest is the prior vector. The prior vectors for the two regions of interest are represented as follows: and ,in and Indicates the number of lexical elements. Indicates the feature dimension; According to statistics from Gemini 2.0, Among the biomarkers of functional regions of interest, for the first The text description of a region of interest is "[Region of Interest]". This region of interest is decomposed into words and mapped using a pre-trained BioBERT natural language representation model. The word vector corresponding to each word in the region of interest is the prior vector. The prior vector of this region of interest is represented as follows: ,in Indicates the number of lexical elements. Indicates the feature dimension; According to statistics from Gemini 2.0, Among the biomarkers of structural connections, for the first The structured text description is "[Region of Interest 1]-[Region of Interest 2]". This is split into two regions of interest, "[Region of Interest 1]" and "[Region of Interest 2]", for further processing. The names of these two regions of interest are decomposed into words and mapped using a pre-trained BioBERT natural language representation model. The word vector corresponding to each word in each region of interest is the prior vector. The prior vectors for these two regions of interest are represented as follows: and ,in and Indicates the number of lexical elements. Indicates the feature dimension; According to statistics from Gemini 2.0, Among the biomarkers of the structural region of interest, for the first... The text description of a structural region of interest is "[Region of Interest]". This region of interest is decomposed into words and mapped using a pre-trained BioBERT natural language representation model. The word vector corresponding to each word in the region of interest is the prior vector. The prior vector of this region of interest is represented as follows: ,in Indicates the number of lexical elements. Indicates the feature dimension; For whole-brain segmentation using decomposition-based or region-of-interest-based methods The region of interest for the function will be the first The names of regions of interest (ROIs) are input into the pre-trained BioBERT natural language representation base model, decomposed into words, and mapped. The word vector corresponding to each word in the ROI is the prior vector. The prior vector of this ROI is represented as follows: And among them, D represents the number of lexical units and the feature dimension. For the use of brain structure atlases to divide the whole brain The region of interest for the structure will be the first The names of the regions of interest (ROIs) are input into the pre-trained BioBERT natural language representation base model, decomposed into words, and mapped. The word vector corresponding to each word in the ROI is the prior vector. The prior vector of this ROI is represented as follows: And among them, D represents the number of lexical units and the feature dimension. Step 2.3 calculates the similarity between the names of biomarkers described in published literature and the names of regions of interest. The specific steps are as follows: For functional connectivity priors, calculate and Similarity matrix ,use express The first in Line number Column elements: ； in, express The first in OK, express The first in Rows; Matrix Quantified and The similarity strength of corresponding word elements is indicated by a higher value, which means the two are more similar, and a lower value means the greater the difference. Next, calculate the F1 score for "[Region of Interest 1]". : ； ； ； The F1 score matrix is ​​obtained by comparing the prior vector of "[Region of Interest 1]" in the functional connectivity of all biomarkers with the prior vector of the region of interest name based on decomposition or region of interest method partitioning. ; Similarly, calculation and The F1 score matrix can be obtained by comparing the prior vector of "[Region of Interest 2]" in the functional connectivity of all biomarkers with the prior vectors of the names of all regions of interest based on decomposition or region of interest methods. ; Next, construct a weighted connection similarity matrix. For the first Functional links: ； in, Represent the outer product; then calculate the maximum value under each function connection matrix: ； therefore, The elements in the file are updated as follows: ； in, Indicates the first The frequency of occurrence of each functional connection; Ultimately, the matrix contains prior information about functional connectivity. Represented as: ； in, This means normalizing the elements in the matrix to the range [0, 1]. A matrix representing prior information of functional connections Elements in; For the prior of the region of functional interest, calculate and Similarity matrix ,use express The Middle Line number Column elements: ； in, express The first in OK, express The first in Rows; Matrix Quantified and The similarity strength of corresponding word elements is indicated by a higher value, which means the two are more similar, and a lower value means the greater the difference. Next, calculate the F1 score. : ； ； ； Obtain the functional region of interest prior vectors for all biomarkers and the F1 score matrix of the prior vectors of the names of all regions of interest partitioned based on decomposition or region of interest methods. ; Then, calculate the maximum value for each vector: ； therefore, The elements in the file are updated as follows: ； in, Indicates the first Frequency of occurrence of each functional region of interest; Ultimately, the vector contains prior information about the functional region of interest. Represented as: ； in, This means normalizing the elements in the vector to the range [0, 1]. Vector representing prior information of the functional region of interest Elements in; For structural connection priors, calculation and Similarity matrix ,use express The first in Line number Column elements: ； in, express The first in OK, express The first in Rows; Matrix Quantified and The similarity strength of corresponding word elements is indicated by a higher value, which means the two are more similar, and a lower value means the greater the difference. Next, calculate the F1 score for "[Region of Interest 1]". : ； ； ； The F1 score matrix is ​​obtained by comparing the prior vector of "[Region of Interest 1]" in the structural connectivity of all biomarkers with the prior vector of the region of interest name in all brain structural maps. ; Similarly, calculation and This yields the F1 score matrix of the prior vectors for "[Region of Interest 2]" in the structural connections of all biomarkers and the prior vectors for the names of regions of interest in all brain structural maps. ; Next, construct a weighted connection similarity matrix. For the first Each structural connection: ； in, Represent the outer product; then calculate the maximum value under each structural link matrix: ； therefore, The elements in the file are updated as follows: ； in, Indicates the first The frequency of structural connections appearing in prior knowledge; Ultimately, the matrix contains prior information about structural connections. Represented as: ； in, This means normalizing the elements in the matrix to the range [0, 1]. A matrix representing prior information about structural connections Elements in; For the prior of the structural region of interest, calculate and Similarity matrix ,use express The Middle Line number Column elements: ； in, express The first in OK, express The first in Rows; Matrix Quantified and The similarity strength of corresponding word elements is indicated by a higher value, which means the two are more similar, and a lower value means the greater the difference. Next, calculate the F1 score. : ； ； ； The F1 score matrix is ​​obtained by combining the prior vectors of the structural regions of interest for all biomarkers with the prior vectors of the region of interest names for all brain structure atlases. ; Then, calculate the maximum value for each vector: ； therefore, The elements in the file are updated as follows: ； in, Indicates the first The frequency of occurrence of each structural region of interest; Finally, the vector containing prior information about the structural region of interest. Represented as: ； in, This means normalizing the elements in the vector to the range [0, 1]. Vector representing prior information of the region of interest in the structure The elements in.

4. The mental illness classification method that integrates prior knowledge and multimodal self-attention mechanisms according to claim 1, characterized in that: The training of the prior knowledge-guided mental illness classification model in step 3 specifically involves: Step 3.1: Generate the functional connectivity matrix of all subjects in the training dataset. The input function connects to the multi-channel self-attention transformation and fusion module, i.e. hour ; the structural connectivity matrix of all subjects in the training dataset The multi-channel self-attention transformation and fusion module of the input structure connection matrix, i.e. hour ; Step 3.2, the specific content of the multi-channel self-attention transformation and fusion module of the functional connectivity matrix is ​​as follows: First, calculate the function connection matrix input at the [number]th [node]. The Q, K, and V matrices of the layer: ； ； ； in, This indicates the total number of layers in the functional connectivity model. It is the output of the functional connection matrix in the previous layer; , and For linear layer weights; Then, the outputs of the three channels are calculated separately for the three self-attention mechanisms: ； ； ； in, , , , for The dimension; Represents the Hadamard product; The outputs from the three channels are fused using a channel attention mechanism: ； ； ； in, , , ; and For linear layer weights; The output obtained after the feedforward network layer is represented as follows: ； ； ； in, , , ; and For linear layer weights, Step 3.3, the specific steps of the multi-channel self-attention transformation and fusion module of the structure connection matrix are: calculating the structure connection matrix input at the... The Q, K, and V matrices of the layer: ； ； ； in, This indicates the total number of layers in the structural connection model. It is the output of the structure connection matrix in the previous layer; , and For linear layer weights; Then, the outputs of the three channels are calculated separately for the three self-attention mechanisms: ； ； ； in, , , , for The dimension; Represents the Hadamard product; The outputs from the three channels are fused using a channel attention mechanism: ； ； ； in, , , ; and For linear layer weights; The output obtained after the feedforward network layer is represented as follows: ； ； ； in, , , ; and For linear layer weights, Step 3.4, the specific content of the output classification result is as follows: For and The output classification probability is: ； in, and Represents the classification layer; the loss function is calculated using cross-entropy: ； in, Indicates the number of samples. Indicates the number of categories; Indicates label index, Represents classification probability The index.

Images