Stress injury diagnosis method based on multi-modal brain image data and deep learning neural network

Abstract

The invention discloses a stress injury diagnosis method based on multi-modal brain image data and a deep learning neural network, and relates to the technical field of imaging medicine, and the method comprises the steps: S1, integrating data from different modals and different dimensions; s2, improving a multi-modal feature fusion and classification method, automatically performing feature selection on different modals by using position coding and an attention mechanism, and fusing and matching a brain structure and two types of neural connection features; and S3, explaining the abnormal neural connection. According to the stress injury diagnosis method based on the multi-modal brain image data and the deep learning neural network, in an interpretable experiment, it can be found that a model concentratedly pays attention to brain regions related to language, movement and somatosensory, such as central anterior return, triangular subfrontal return and central posterior return; meanwhile, functional connectivity values associated with the left and right cortices and fibrous structure connectivity values associated with the temporal lobe are more concerned by the model, indicating that diseases may affect these brain structures and nerve connections.

Description

Technical Field

[0001] The present invention relates to the field of imaging medical technology, and specifically to a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network. Background Art

[0002] Stress-induced injury refers to physical or psychological damage caused by external pressure or stress factors, which seriously affects the patient's life. The complexity and high intensity of modern military tasks make soldiers face great psychological and physiological stress. Long-term high-intensity training, complex combat environment and continuous psychological pressure will not only affect the physical health and psychological state of soldiers, but also may affect the execution effect of tasks and the overall combat effectiveness of the team. Therefore, accurate diagnosis and assessment of the stress state of soldiers is conducive to timely warning and provision of countermeasures, which is very critical for coping with stress injuries. In recent years, magnetic resonance imaging technology has become the main means of stress injury diagnosis research, such as structural MRI, functional MRI (fMRI), diffusion tensor imaging (DTI), etc., which are widely used in clinical detection, diagnosis and treatment of various brain diseases. At the same time, with the rapid development of deep learning technology, the stress injury neuroimaging diagnosis model based on deep learning has excellent performance. Researchers use deep learning models to fully learn the rich features and hidden representations of brain magnetic resonance imaging data, significantly improving the efficiency and reliability of early diagnosis of stress injuries. There is currently one Chinese patent related to stress diagnosis based on brain imaging: "A diagnostic evaluation system for post-traumatic stress disorder" applied by Sichuan University (application number: 201511028623.0). The patent discloses a diagnostic evaluation system for post-traumatic stress disorder, including: a functional imaging module for obtaining resting-state functional imaging data from the brain of the detected individual; an attribute extraction module for extracting resting-state functional imaging attribute data from the resting-state functional imaging data obtained by the functional imaging module; a post-traumatic stress disorder feature sample library module for storing historical resting-state functional imaging attribute data of different post-traumatic stress disorder levels; a post-traumatic stress disorder assessment module for comparing resting-state functional imaging attribute data with historical resting-state functional imaging attribute data to obtain the post-traumatic stress disorder level to which the resting-state functional imaging attribute data belongs. Quantitative indicators can be obtained to objectively and accurately assess the post-traumatic stress disorder level of the individual being tested, so as to guide medical staff to diagnose, manage and treat patients. At present, there is a lack of stress injury diagnosis methods that combine multimodal imaging data. Therefore, this study proposed an interpretable multimodal CNN-Transformer model based on brain connectivity based on structural magnetic resonance imaging, functional connectivity, and diffusion tensor imaging (DTI) fiber structure connectivity. The model consists of three DenseNets for feature extraction and a Transformer Encoder for feature selection. In order to determine the brain features that are closely related to the disease in each mode category, we also used Grad-CAM (gradient weighted class activation mapping) technology to generate heat maps, combined with brain maps mapped to specific brain regions and neural connection features.

[0003] The current multimodal diagnostic model lacks the fusion of key features of neural connectivity, and most of the modal fusion methods are simple convolution or fully connected layer structures, which can only learn limited information. Therefore, this patent integrates data from different modalities and dimensions, not only using 3D brain structure imaging data, but also integrating two types of brain connection data-2D fMRI functional connection network and 2D DTI fiber structure network, using multimodal features to improve model performance. Introducing position encoding and attention mechanism, combining Transformer Encoder and CNN to build a classifier: Improve multimodal feature fusion and classification methods, use position encoding and attention mechanism to automatically select features for different modalities, fuse and match brain structure with two types of neural connection features, and improve model sensitivity and accuracy. Secondly, most of the current methods ignore model interpretability, and it is very necessary to find key brain features related to the disease. Therefore, this patent uses deep learning interpretability technology Grad-CAM, combined with brain maps and statistical methods, to find key brain area features associated with the disease and explain abnormal neural connections.

[0004] Therefore, there is an urgent need to improve this shortcoming. The present invention studies and improves the existing technology and its shortcomings, and provides a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network. Summary of the invention

[0005] The purpose of the present invention is to provide a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network, comprising the following steps: S1. Integrate data from different modalities and dimensions, not only using 3D brain structure imaging data, but also fusing two types of brain connection data - 2D fMRI functional connection network and 2D DTI fiber structure network; S2. Improve the multimodal feature fusion and classification method, use position encoding and attention mechanism to automatically select features from different modalities, and fuse and match brain structure with two types of neural connection features; S3. Use deep learning interpretability technology Grad-CAM, combined with brain maps and statistical methods, to find key brain region features and explain abnormal neural connections.

[0007] Furthermore, in step S1, the data integration includes data preprocessing, that is, preprocessing the data in the subject package. Each subject contains three types of imaging data: T1 MRI, DTI and fMRI, and corresponding preprocessing operations are performed on the three types of modality data.

[0008] Furthermore, the specific preprocessing operations of the three types of modal data are as follows: Preprocessing of T1 MRI data: use FSL to perform linear registration, format conversion and perform z-score voxel normalization, clip out the intensity outliers and background removal. After preprocessing, the size of each subject's T1 MRI data is 181x217x181. Preprocessing of fMRI data: The CONN toolbox was used to preprocess the fMRI data. The data preprocessing pipeline used in the experiment included functional realignment and unwarp, slice-timing correction, outlier identification, and MNI-space normalization. The functional connectivity matrix was extracted from the preprocessed fMRI data using the Python nilearn library. The main process was to use the Harvard-Oxford cortical and subcortical structural atlas to extract the time series of each ROI, and then calculate the Pearson correlation between them to construct the functional connectivity matrix. After preprocessing, the dimension of the FC network of each subject was 118x118. Preprocessing of DTI data: DTI data was preprocessed through the pipeline toolbox PANDA. Specifically, Deterministic Fiber Tracking was performed through the PANDA toolbox to track the white matter fibers of the whole brain and calculate the structural network. In the parameter setting, the FACT (Fiber Assignment by Continuous Tracking) method was selected; during fiber tracking, the Angle Threshold was set to 35°; the FA Threshold was set from 0.2 to 1, and finally the fiber number network was obtained - a brain network constructed using the number of fibers between each partition of the AAL90 brain atlas, with a data size of 90x90.

[0009] Furthermore, in step S2, multimodal feature extraction and fusion construct feature extractors of different modalities based on the Densenet169 model. Specifically, the feature extractor of T1 MRI data is a 3D structure, while the fMRI functional network and DTI structural network data are 2D structures, and after passing through multiple dense blocks and transition layers, the number of channels and size of the feature map are changed.

[0010] Furthermore, the number of channels of the feature maps obtained by the last Dense Block of the feature extractor for the three types of modal data, T1 MRI, DTI and fMRI, are all 1664, that is, after global average pooling and flatten operations, feature vectors with a length of 1664 are obtained.

[0011] Furthermore, the subsequent processing flow of the feature vector is as follows: The feature vectors learned by the network from T1 MRI, DTI and fMRI modality data Perform dimension conversion and fusion respectively; Will Divide it into 13 small vectors of size 1x128 in sequence, and get , After segmentation, the A small vector, The value range of is 1 to 13. Concatenate into a matrix of size 13×128 according to dimension 0 to obtain the concatenated features Similarly, Perform the same segmentation and splicing operations to obtain the spliced ​​features ; In the direction of dim=0 Concatenated into a multimodal feature vector, expressed as , size is 39x128; For multimodal feature vectors Position encoding is performed to help the model capture the position information in the input features. It is generated by sine and cosine functions. For position p and dimension i, the encoding definition formula is as follows: ; ; Where d is the embedding dimension of the model, which is 128; i ranges from 0 to , p ranges from 1 to 39; the position encoding matrix PE is added to the eigenvector are combined, as shown in the following formula: .

[0012] Furthermore, in step S2, the self-attention mechanism allows the model to dynamically focus on different parts when processing input data, and calculates weights based on the relevance of different parts, while introducing position encoding and Transformer Encoder into the classification structure.

[0013] Furthermore, the multimodal features are processed by the Transformer encoder, and then go through two convolution, ReLU, and pooling operations, and finally the diagnosis of the model is obtained through the fully connected layer and the softmax function.

[0014] Furthermore, in step S3, the process is explained as follows: the grad-cam heat map of the subjects under three types of modalities is extracted, and then the range of brain areas that the model pays high attention to, abnormal neural functional connection values, and fiber structure connection values ​​are analyzed at the individual level and the group level in combination with brain maps and statistical methods.

[0015] Furthermore, the three types of modal data are analyzed using the following two processes: For the Grad-CAM 3D heat map of F1 MRI data, combined with the AAL3 brain atlas, the heat average of all voxels in each brain region was extracted for each subject; specific operations: at the individual level, the brain regions of each subject in the experimental group were arranged in ascending order according to the heat value, and the top few brain regions with the highest values ​​were found; at the group level, the values ​​of the same brain region of each subject were taken as a group of data, and the data of the experimental group and the control group were subjected to a two-sample independent t-test, and FDR correction was performed; In fMRI or DTI data, each position represents the functional connection value (number of fibers) between two brain regions. At the individual level, the values ​​at the same position in the heat map of all subjects in the experimental group are extracted as a group of data. After the average value of each group of data is calculated, they are sorted from large to small. At the group level, combined with the original input data, the main process is to extract values ​​from the same position of the fMRI (DTI) connection network of the subjects in the experimental group (control group) to form the current brain region connection value sample data. Subsequently, the data of the experimental group and the control group are subjected to a two-sample independent t-test and FDR correction. The final result combines the individual level and group level analysis.

[0016] The present invention provides a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network, which has the following beneficial effects: 1. The method proposed in this invention integrates data from different modalities and dimensions. It not only uses 3D brain structure imaging data, but also fuses two types of brain connection data - 2D fMRI functional connection network and 2D DTI fiber structure network, integrating multimodal features to improve model performance.

[0017] 2. The improved multimodal feature fusion and classification method proposed in the present invention uses position encoding and attention mechanism to automatically select features from different modalities, fuses and matches brain structure with two types of neural connection features, and improves model sensitivity and accuracy.

[0018] 3. The present invention proposes to use the deep learning interpretability technology Grad-CAM, combined with brain maps and statistical methods, to find the characteristics of key brain areas and explain abnormal neural connections.

[0019] Compared with the prior art, the experimental results of the present invention show that the average accuracy of the experimental group vs. the control group is 95.19%, which is 1% to 3% higher than the advanced model; in addition, in the interpretability experiment, it can be found that the model focuses on the precentral gyrus, inferior frontal gyrus of the triangular division, and postcentral gyrus, which are brain areas related to language, movement, and somatic sensation; at the same time, the functional connectivity values ​​associated with the left and right Cingulate Gyrus (posterior division) cortices, and the fiber structure connectivity values ​​associated with the temporal lobe are more concerned by the model, indicating that the disease will affect these brain structures and neural connections. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a general framework diagram of a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network of the present invention; Figure 2A schematic diagram of the network structure of a Densenet169 feature extraction module of a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network of the present invention; Figure 3 A schematic diagram of the classification structure of a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to the present invention; Figure 4 This is a flow chart of the model interpretability module of a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network in the present invention. DETAILED DESCRIPTION

[0021] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0022] like Figure 1-Figure 4 As shown, a stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network includes the following steps: First, an interpretable multimodal CNN-Transformer model based on brain structure and brain connectivity for diagnosing stress injury neuroimaging is proposed, and the overall framework is shown in Figure 1. The classification network learns all processes in an overall framework, while utilizing three different types of images and fully integrating multimodal and multidimensional data. It mainly includes data preprocessing module, feature extraction and fusion module, classification module, and model interpretability module.

[0023] S1. Integrate data from different modalities and dimensions, not only using 3D brain structure imaging data, but also fusing two types of brain connection data - 2D fMRI functional connection network and 2D DTI fiber structure network; Data preprocessing: Preprocess the data in the subject package. Each subject contains three types of imaging data: T1MRI, DTI and fMRI. Perform corresponding preprocessing operations on the three types of modality data; In this embodiment, the specific preprocessing operations of the three types of modal data are as follows: Preprocessing of T1 MRI data: use FSL for linear registration, format conversion and perform z-score voxel normalization, clip out the intensity outliers and background removal. After preprocessing, the size of each subject’s T1 MRI data is 181x217x181.

[0024] Preprocessing of fMRI data: The CONN toolbox was used to preprocess the fMRI data. The data preprocessing pipeline used in the experiment included functional realignment and unwarp, slice-timing correction, outlier identification, and MNI-space normalization. The python nilearn library was used to extract the functional connectivity matrix from the preprocessed fMRI data. The main process was to use the Harvard-Oxford cortical and subcortical structural atlas to extract the time series of each ROI, and then calculate the Pearson correlation between them to construct the functional connectivity matrix. After preprocessing, the dimension of the FC network of each subject was 118x118.

[0025] Preprocessing of DTI data: DTI data was preprocessed through the pipeline toolbox PANDA. Specifically, Deterministic Fiber Tracking was performed through the PANDA toolbox to track the white matter fibers of the whole brain and calculate the structural network. In the parameter setting, the FACT (Fiber Assignment by Continuous Tracking) method was selected; during fiber tracking, the Angle Threshold was set to 35°; the FA Threshold was set from 0.2 to 1, and finally the fiber number network was obtained - a brain network constructed using the number of fibers between each partition of the AAL90 brain atlas, with a data size of 90x90.

[0026] S2. Improve the multimodal feature fusion and classification method, use position encoding and attention mechanism to automatically select features from different modalities, and fuse and match brain structure with two types of neural connection features; Multimodal feature extraction and fusion: In the proposed method, feature extractors of different modalities are constructed based on the Densenet169 model. The network structure is as follows: Figure 2 shown.

[0027] In this embodiment, the feature extractor of T1 MRI data is a 3D structure, while the fMRI functional network and DTI structural network data are 2D structures, and after multiple dense blocks and transition layers, the number of channels and size of the feature map are changed. Among them, the number of channels of the feature map obtained by the last dense block of the feature extractor for the three types of modal data is 1664, so after global average pooling and flatten operations, feature vectors with a length of 1664 are obtained.

[0028] Next, the feature vectors learned by the network from T1 MRI, DTI, and fMRI modality data are Dimension conversion and fusion are performed separately. This part of the operation simply splits and coaxially splices the three types of modal vectors, and then uses position encoding and Transformer encoder to automatically capture the key information of the features and the correlation between the features.

[0029] Will Divide it into 13 small vectors of size 1x128 in sequence, which can be expressed by formula 1, where After segmentation, the A small vector, The value range of is 1 to 13. Indicates that from The element is taken to elements; --Formula 1.

[0030] Then will Concatenate into a matrix of size 13×128 at dimension 0, as shown in Formula 2, where Represented as concatenated features; --Formula 2.

[0031] Similarly, yes The same splitting and splicing operations are performed, which are expressed by formulas 3, 4, 5, and 6 respectively, where The features after splicing is completed; --Formula 3; --Formula 4; --Formula 5; --Formula 6.

[0032] Next, in the direction of dim=0, Concatenated into a multimodal feature vector, expressed as , size is 39x128, as shown in formula 7; --Formula 7.

[0033] Finally, for the multimodal feature vector Position encoding is performed to help the model capture the position information in the input features. It is generated by sine and cosine functions. For position p and dimension i, the encoding definition is shown in formulas 8 and 9. d is the embedding dimension of the model, which is 128. The range of i is from 0 to , p ranges from 1 to 39. The position encoding matrix PE is added to the eigenvector The fusion is shown in formula 10; --Formula 8; --Formula 9; --Formula 10.

[0034] Self-Attention: The self-attention mechanism plays a key role in Transformer. It allows the model to dynamically focus on different parts when processing input data and calculate weights based on their relevance. Transformer usually uses multi-head self-attention, that is, using multiple sets of query, key and value at the same time, which allows the model to capture the relationship between different features and better learn the importance of features. Inspired by Transformer, this example introduces position encoding and Transformer Encoder in the classification structure. The network structure is as follows: Figure 3 shown.

[0035] After being processed by the Transformer encoder, the multimodal features undergo two more convolutions, ReLU, and pooling operations. The process can be expressed by formulas 11 and 12, where is the input, It is the feature obtained after the first convolution, ReLU activation and pooling. It is the final feature representation obtained after two operations.

[0036] --Formula 11; --Formula 12; Finally, the model diagnosis is obtained through the fully connected layer and the softmax function, see formula 13; --Formula 13.

[0037] S3. Use deep learning interpretability technology Grad-CAM, combined with brain maps and statistical methods, to find key brain region features and explain abnormal neural connections; Model interpretability: The overall framework of the model interpretability module is as follows Figure 4 As shown in the figure, it includes (a) the analysis process for F1 MRI data and (b) the analysis process for fMRI (DTI) data. In the whole method, the grad-cam heat map of the three types of subjects is extracted, and then the brain map and statistical methods are combined to analyze the range of brain regions that the model pays high attention to, abnormal neural functional connectivity values, and fiber structure connectivity values ​​at the individual level and group level. For the Grad-CAM3D heat map of F1 MRI data, combined with the AAL3 brain map, the heat average of all voxels in each brain region is extracted for each subject. At the individual level, the brain regions of each subject in the experimental group are arranged in ascending order according to the heat value, and the top few brain regions with the highest values ​​are found. At the group level, the values ​​of the same brain region of each subject are taken as a group of data, and the data of the experimental group and the control group are subjected to a two-sample independent t-test, and FDR correction is performed.

[0038] In fMRI (DTI) data, each position represents the functional connection value (number of fibers) between two brain regions. At the individual level, the values ​​at the same position in the heat map of all subjects in the experimental group are extracted as a group of data. After the average value of each group of data is calculated, they are sorted from large to small. At the group level, combined with the original input data, the main process is to extract values ​​from the same position of the fMRI (DTI) connection network of the subjects in the experimental group (control group) to form the current brain region connection value sample data. Subsequently, the data of the experimental group and the control group were subjected to a two-sample independent t-test and FDR correction. The final results are a comprehensive analysis of the individual level and the group level.

[0039] The experimental results of this example show that the average accuracy of the experimental group vs. the control group is 95.19%, which is 1% to 3% higher than the advanced model. In addition, in the interpretability experiment, it can be found that the model focuses on the precentral gyrus, triangular inferior frontal gyrus, and postcentral gyrus, which are brain areas related to language, movement, and somatic sensation; at the same time, the functional connectivity values ​​associated with the left and right Cingulate Gyrus (posterior division) cortices and the fiber structure connectivity values ​​associated with the temporal lobe are more concerned by the model, indicating that the disease affects these brain structures and neural connections. The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network, characterized in that: The following steps are involved: S1. Integrate data from different modalities and dimensions, not only using 3D brain structure imaging data, but also fusing two types of brain connection data - 2D fMRI functional connection network and 2D DTI fiber structure network; S2. Improve the multimodal feature fusion and classification method, use position encoding and attention mechanism to automatically select features from different modalities, and fuse and match brain structure with two types of neural connection features; S3. Use deep learning interpretability technology Grad-CAM, combined with brain maps and statistical methods, to find key brain region features and explain abnormal neural connections.

2. The stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 1, characterized in that: In step S1, integrating data includes preprocessing the data in the subject package. Each subject contains three types of imaging data: T1 MRI, DTI and fMRI. Corresponding preprocessing operations are performed on the three types of modality data.

3. The stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 2, characterized in that: The specific preprocessing operations of the three types of modal data are as follows: Preprocessing of T1 MRI data: FSL was used for linear registration, format conversion, z-score voxel normalization, trimming of intensity outliers, and background removal. After preprocessing, the size of each subject’s T1 MRI data was 181x217x181. Preprocessing of fMRI data: The CONN toolbox was used to preprocess the fMRI data. The data preprocessing process included functional image alignment and correction, slice time correction, outlier identification, and MNI space normalization. The functional connectivity matrix was extracted from the preprocessed fMRI data using the python nilearn library. After preprocessing, the dimension of the FC network of each subject was 118x118. Preprocessing of DTI data: DTI data were preprocessed through the pipeline toolbox PANDA. Specifically, Deterministic Fiber Tracking was performed through the PANDA toolbox to track the white matter fibers of the whole brain and calculate the structural network. In the parameter setting, the FACT method was selected. When tracking fibers, the Angle Threshold was set to 35°. The FAThreshold was set from 0.2 to 1, and finally the fiber number network was obtained, which was a brain network constructed using the number of fibers between each partition of the AAL90 brain atlas. The data size was 90x90.

4. The stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 1, characterized in that: In step S2, multimodal feature extraction and fusion construct feature extractors of different modalities based on the Densenet169 model. Specifically, the feature extractor of T1 MRI data is a 3D structure, while the feature extractor of fMRI functional network and DTI structural network data is a 2D structure.

5. A stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 4, characterized in that: The number of channels of the feature maps obtained by the three types of modal data, T1 MRI, DTI and fMRI, after the last Dense Block of the feature extractor is 1664, that is, after global average pooling and flatten operations, feature vectors with a length of 1664 are obtained.

6. The stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 5, characterized in that: The subsequent processing flow of the feature vector is as follows: The feature vectors learned by the network from T1 MRI, DTI and fMRI modality data Perform dimension conversion and fusion respectively; Will Divide it into 13 small vectors of size 1x128 in sequence, and get , After segmentation, the A small vector, The value range of is 1 to 13. Concatenate into a matrix of size 13×128 according to dimension 0 to obtain the concatenated features Similarly, Perform the same segmentation and splicing operations to obtain the spliced ​​features ; In the direction of dim=0 Concatenated into a multimodal feature vector, expressed as , size is 39x128; For multimodal feature vectors Position encoding is performed, which is specifically generated by sine and cosine functions. For position p and dimension i, the encoding definition formula is as follows: ； ； Where d is the embedding dimension of the model, which is 128; i ranges from 0 to , p ranges from 1 to 39; the position encoding matrix PE is added to the eigenvector are combined, as shown in the following formula: 。 7. The stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 1, characterized in that: In step S2, the self-attention mechanism allows the model to dynamically focus on different parts when processing input data, and calculates weights based on the relevance of different parts, while introducing position encoding and Transformer Encoder into the classification structure.

8. The stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 6, characterized in that: The multimodal features are processed by the Transformer encoder, and then go through two convolution, ReLU, and pooling operations, and finally the diagnosis of the model is obtained through the fully connected layer and the softmax function.

9. The stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 1, characterized in that: In step S3, the explanation process is as follows: extracting the grad-cam heat map of the subject under three types of modalities, and then combining brain maps and statistical methods to analyze the range of brain areas that the model pays high attention to, abnormal neural functional connection values, and fiber structure connection values ​​at the individual level and group level.

10. A stress injury diagnosis method based on multimodal brain imaging data and deep learning neural network according to claim 9, characterized in that: The three types of modal data are analyzed using the following two analysis processes: For the Grad-CAM 3D heat map of F1 MRI data, combined with the AAL3 brain atlas, the heat average of all voxels in each brain region was extracted for each subject; specific operations: at the individual level, the brain regions of each subject in the experimental group were arranged in ascending order according to the heat value, and the top few brain regions with the highest values ​​were found; at the group level, the values ​​of the same brain region of each subject were taken as a group of data, and the data of the experimental group and the control group were subjected to a two-sample independent t-test, and FDR correction was performed; In fMRI or DTI data, each position represents the functional connection value between two brain regions; at the individual level, the values ​​at the same position in the heat map of all subjects in the experimental group are extracted as a group of data. After the average value of each group of data is calculated, they are sorted from large to small; at the group level, combined with the original input data, the main process is to extract values ​​from the same position of the fMRI or DTI connection network of the subjects in the experimental group / control group to form the current brain region connection value sample data, and then a two-sample independent t-test is performed on the data of the experimental group and the control group, and FDR correction is performed. The final result combines the individual level and group level analysis.

Images