Image processing method, device and equipment based on multi-modal nerve image fusion

Abstract

The invention discloses an image processing method, device and equipment based on multi-modal neural image fusion, and relates to the technical field of image recognition. The method comprises the following steps: calculating a functional connection matrix among regions in brain nerves based on functional magnetic resonance imaging data; extracting a first feature of the structural magnetic resonance imaging data and a second feature of the functional link matrix; obtaining a first feature vector and a second feature vector corresponding to the first feature and the second feature, and adding the first feature vector and the second feature vector to obtain a common feature vector; and performing feature enhancement on the first feature vector and the second feature vector through the common feature vector, splicing the enhanced first feature vector and the enhanced second feature vector to obtain a spliced feature vector, and inputting the spliced feature vector into a pre-constructed feature classification network to obtain a classification result. According to the scheme, complementarity among different modal data can be enhanced, so that diagnosis of depression is better assisted.

Description

Technical Field

[0001] The present invention relates to the technical field of image data feature recognition, and in particular to an image processing method, device and equipment based on multimodal neural image fusion. Background Art

[0002] Major Depressive Disorder (MDD) is a common mental illness, characterized by long-term, persistent low and depressed mood, and even severe inferiority complex and loss of confidence in life. In recent years, with the launch of the Global Brain Project, relevant departments have proposed a "one body and two wings" architecture, among which "the development of effective methods for early diagnosis / intervention of brain diseases" as the core content of the "two wings" has attracted the attention of many researchers. In particular, the traditional method of diagnosing depression mainly relies on subjective interviews based on scales, which are easily affected by environmental and individual differences, resulting in a low clinical recognition rate of depression. This means that how to develop an objective quantitative auxiliary diagnosis method for depression has become a must for computer-aided diagnosis (CAD) in the diagnosis of depression.

[0003] Neuroimaging technology provides a new means for objective quantification of mental illness. Magnetic Resonance Imaging (MRI) achieves the exploration of brain structure and functional changes in a non-invasive way. Structural MRI (sMRI) uses the interaction between magnetic fields and radio waves to present the macroscopic structural changes of the brain caused by brain development and disease. Functional MRI (fMRI) tracks the functional activities of the brain by recording the fluctuations of blood oxygen level-dependent (BOLD) signals in the brain. Diffusion tensor imaging (DTI) studies the microscopic structural changes of the brain by visualizing the movement trajectory of white matter fiber bundles.

[0004] In recent years, with the development of various neuroimaging technologies, how to utilize their imaging characteristics, explore the information complementarity of multimodal neuroimaging, and thus improve the diagnostic reliability of depression has become a hot topic of concern for many researchers. Many studies have attempted to use multimodal neuroimaging data to diagnose major depressive disorder. These methods mainly focus on the two stages of feature extraction and feature fusion. Research on auxiliary diagnosis of depression based on multi-source imaging continues to emerge. According to the feature extraction and fusion methods, the mainstream methods can be divided into traditional machine learning algorithms and deep learning-based methods. Sun et al. extracted four features from multimodal MRI data from two centers as research objects, and used support vector machine (SVM) as a diagnostic model to explore its role in the diagnosis of MDD. Li et al. improved the diagnostic performance of MDD by fusing common features of sMRI and fMRI, using random forest (RF) and SVM as classifiers. However, traditional machine learning methods often rely on manually designed feature engineering, which is prone to dimensionality disaster when processing high-dimensional data, affecting the generalization performance of the model. This situation may become more significant when processing complex tasks and large-scale data. In terms of auxiliary diagnosis of depression based on multi-source imaging based on deep learning algorithms, Zheng et al. designed a brain function-structure fusion module to capture the interaction between deep features extracted from different MRI modalities, aiming to improve the fusion ability of multimodal imaging. Qin et al. proposed a graph convolutional neural network (GCN) for identifying MDD and obtaining robust diagnostic performance.

[0005] However, due to the significant modal differences between multimodal data (such as differences in data distribution and feature expression), existing deep learning-based multimodal fusion methods have not fully exploited the complementarity of multimodal features. Summary of the invention

[0006] Based on this, it is necessary to provide image processing methods, devices and equipment based on multimodal neural image fusion to address the technical problem that the existing technology fails to fully explore the complementarity of multimodal features.

[0007] The present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides an image processing method based on multimodal neural image fusion, the method comprising:

[0009] Acquiring structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculating the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data;

[0010] Extracting the features of the structural magnetic resonance imaging data by a pre-constructed three-dimensional convolutional neural network to obtain a first feature; extracting the features of the functional link matrix by a pre-constructed two-dimensional convolutional neural network to obtain a second feature;

[0011] Acquire a first feature vector and a second feature vector corresponding to the first feature and the second feature respectively, and add the first feature vector and the second feature vector to obtain a common feature vector of the brain nerves, and perform feature enhancement on the first feature vector and the second feature vector respectively through the common feature vector to obtain an enhanced first feature vector and an enhanced second feature vector;

[0012] The common feature vector is concatenated with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, and the concatenated feature vector is input into a pre-constructed feature classification network to obtain a classification result.

[0013] Furthermore, the functional connection matrix between various regions in the brain nerves is calculated based on the functional magnetic resonance imaging data, and the calculation expression is:

[0014]

[0015] where r i and r j is the blood oxygen level dependence signal of different brain regions, cov(r i ,r j ) is r i and r j The covariance between ri and σ rj r i and r j The standard deviation of Corr(r i , r j ) is r i With r j The correlation between .

[0016] Further, the first feature vector and the second feature vector are added to obtain the common feature vector of the brain nerves, specifically including:

[0017] The first eigenvector and the second eigenvector are compressed to the same dimension through a flattening operation, and the compressed first eigenvector and the compressed second eigenvector are added to obtain a common eigenvector of the brain nerves.

[0018] Furthermore, the first feature vector and the second feature vector are respectively enhanced by the common feature vector, and the expression is:

[0019]

[0020] Among them, F s represents the enhanced feature information, softmax represents the softmax function, Q c is the query vector of the information hub, is the transpose of the K vector of feature information that needs to be enhanced, V s is the V vector of the target information, d k is the dimension of the K vector.

[0021] Further, the common feature vector is concatenated with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, which specifically includes:

[0022] The elements in corresponding channels of the first eigenvector, the second eigenvector and the third eigenvector are added respectively to obtain the concatenated eigenvector.

[0023] In a second aspect, the present invention provides an image processing method based on multimodal neural image fusion and an image processing device based on multimodal neural image fusion, characterized in that it includes:

[0024] A multimodal feature extraction module, used to obtain structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculate the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; and also used to extract the features of the structural magnetic resonance imaging data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; and extract the features of the functional connection matrix through a pre-constructed two-dimensional convolutional neural network to obtain a second feature;

[0025] A progressive multimodal information enhancement module, used to obtain a first feature vector and a second feature vector corresponding to the first feature and the second feature respectively, and add the first feature vector and the second feature vector to obtain a common feature vector of the brain nerves, and perform feature enhancement on the first feature vector and the second feature vector respectively through the common feature vector to obtain an enhanced first feature vector and an enhanced second feature vector;

[0026] The multimodal feature classification module is used to concatenate the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, and input the concatenated feature vector into a pre-constructed feature classification network to obtain a classification result.

[0027] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image processing method based on multimodal neural image fusion is implemented when the processor executes the program.

[0028] The at least one technical solution adopted by the present invention can achieve the following beneficial effects: the present invention obtains structural magnetic resonance imaging data and functional magnetic resonance imaging data, and calculates the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; extracts the features of the structural magnetic resonance imaging data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; extracts the features of the functional link matrix through a pre-constructed two-dimensional convolutional neural network to obtain a second feature; obtains a first feature vector and a second feature vector corresponding to the first feature and the second feature, respectively, and adds the first feature vector to the second feature vector to obtain a common feature vector of the brain nerves, and performs feature enhancement on the first feature vector and the second feature vector respectively through the common feature vector to obtain an enhanced first feature vector and an enhanced second feature vector; splices the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a spliced ​​feature vector, which can enhance the features of each modality data, combine the structural changes of the brain presented by the structural magnetic resonance imaging with the abnormalities of the brain function presented by the functional magnetic resonance imaging, thereby realizing cross-modal adaptive interaction, enhancing the complementarity between different modality data, and thus better assisting the diagnosis of depression. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 A schematic diagram of the flow of an image processing method based on multimodal neural image fusion provided by the present invention;

[0031] Figure 2 A schematic diagram of an image processing device based on multimodal neural image fusion provided by the present invention;

[0032] Figure 3 A schematic diagram of a multimodal information interaction unit workflow provided by the present invention;

[0033] Figure 4 A schematic diagram of a computer device for implementing an image processing method based on multimodal neural image fusion provided by the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] The server mentioned in the present invention can be a server set on a business platform, or a device such as a desktop computer, a laptop computer, etc. that can execute the solution of the present invention. For the convenience of description, the following description is only based on the server as the execution subject. The following is a detailed description of the technical solutions provided by various embodiments of the present invention in conjunction with the accompanying drawings.

[0036] refer to Figure 1 , is an image processing method based on multimodal neural image fusion in the present invention, which specifically includes the following steps:

[0037] S10: Acquire structural magnetic resonance imaging data and functional magnetic resonance imaging data, and calculate the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data.

[0038] In this embodiment, the structural magnetic resonance imaging data includes but is not limited to the gray matter volume data of the brain nerves, and the structural magnetic resonance imaging data is preprocessed to obtain the gray matter volume data of the brain nerves. Functional magnetic resonance imaging includes but is not limited to the blood oxygen level dependent signal data of the brain nerves.

[0039] Optionally, the public dataset REST-meta-MDD is used for validation. The inclusion and exclusion criteria are:

[0040] ①The fMRI scanning time points are no less than 170 time points.

[0041] ② The subjects did not appear repeatedly. Finally, a total of 1179 patients with major depression and 1008 healthy control participants from 20 different cohorts were included in the analysis.

[0042] Exemplary, reference Figure 2 For sMRI, the gray matter volume provided by REST-meta-MDD was used as the input of sMRI. For fMRI, the Blood Oxygen Level-Dependent Signal (BOLD) provided by REST-meta-MDD was used to obtain the BOLD signals of 116 brain regions using the Automated Anatomical Labeling (AAL) atlas.

[0043] Among them, the Functional Connectivity Matrix (FCM) is a matrix that represents the correlation of functional activities between different brain regions. In this matrix, the rows and columns represent different brain regions, and the elements in the matrix represent the strength of functional connections between these regions.

[0044] Specifically, the functional connection matrix between various regions in the brain nerves is calculated based on the blood oxygen level dependent signal data, and the calculation expression is:

[0045]

[0046] where r i and r j is the blood oxygen level dependence signal of different brain regions, cov(r i ,r j ) is r i and r j The covariance between ri and σ rj r i and r j The standard deviation of Corr(r i , r j ) is r i With r j The correlation between .

[0047] Specifically, the correlation refers to the degree of correlation between the blood oxygen level dependent signal of any brain region and the blood oxygen level dependent signals of other brain regions. For example, the brain is divided into 90 brain regions using the AAL template, and the correlation includes the correlation between the blood oxygen level dependent signal of the first brain region and the blood oxygen level dependent signals of the other 89 brain regions, and the correlation between the blood oxygen level dependent signal of the second brain region and the blood oxygen level dependent signals of the other 89 brain regions.

[0048] S20: extracting features of gray matter volume data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; extracting features of a functional link matrix through a pre-constructed two-dimensional convolutional neural network to obtain a second feature.

[0049] In this embodiment, considering the limited size of the data, a stacked convolutional neural network model (Convolutional Neural Network, CNN) is constructed as a feature encoder to realize the extraction of multimodal features. Specifically, for gray matter volume data, the present invention constructs a three-dimensional CNN to realize the feature extraction of sMRI. The three-dimensional CNN mainly includes 3 trainable weight layers and 3×3×3 convolution layers, and the number of convolution kernels is 64, 128, and 512 respectively. For the functional connection matrix, a traditional two-dimensional CNN is used as a feature extractor for feature extraction, which mainly includes 3×3 convolution layers, and the number of convolution kernels is 64, 128, and 512 respectively.

[0050] S30: Obtain the first feature vector and the second feature vector corresponding to the first feature and the second feature respectively, and add the first feature vector and the second feature vector to obtain a common feature vector of brain nerves, and perform feature enhancement on the first feature vector and the second feature vector respectively through the common feature vector to obtain an enhanced first feature vector and an enhanced second feature vector.

[0051] In this embodiment, the first eigenvector and the second eigenvector are added to obtain a common eigenvector of brain nerves, which specifically includes:

[0052] The first eigenvector and the second eigenvector are compressed to the same dimension through a flattening operation, and the compressed first eigenvector and the compressed second eigenvector are added to obtain a common eigenvector of brain nerves.

[0053] S40: concatenating the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, and inputting the concatenated feature vector into a pre-constructed feature classification network to obtain a classification result.

[0054] In this embodiment, the elements in the corresponding channels of the first feature vector, the second feature vector and the third feature vector are added respectively to obtain a concatenated feature vector. The feature classification network is constructed based on a multi-layer perceptron.

[0055] In the specific process of assisting depression diagnosis, the method of assisting brain nerve analysis based on splicing feature vectors is as follows:

[0056] A set of fully connected layers is constructed based on the multi-layer perceptron for feature classification. The number of neurons in the first layer of the fully connected layer is 128, and the number of neurons in the second layer is 2. Finally, the analysis results of the brain nerves are determined as severe depression or normal people.

[0057] Optionally, the number of neurons in the second layer of the fully connected layer can be modified according to the number of analysis results. For example, when the analysis results are four results: severe depression, moderate depression, mild depression, and normal, the number of neurons in the second layer is set to 4.

[0058] based on Figure 1 The image processing method based on multimodal neural image fusion shown in the present invention obtains structural magnetic resonance imaging data and functional magnetic resonance imaging data, and calculates the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; extracts the characteristics of the structural magnetic resonance imaging data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; extracts the characteristics of the functional link matrix through a pre-constructed two-dimensional convolutional neural network to obtain a second feature; obtains a first feature vector and a second feature vector corresponding to the first feature and the second feature, respectively, and adds the first feature vector and the second feature vector to obtain a common feature vector of the brain nerves, and enhances the first feature vector and the second feature vector respectively through the common feature vector to obtain an enhanced first feature vector and an enhanced second feature vector; splices the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a spliced ​​feature vector, and inputs the spliced ​​feature vector into a pre-constructed feature classification network to obtain a classification result. Through the above scheme, the characteristics of each modality data can be enhanced, and the structural changes of the brain presented by structural MRI can be combined with the abnormalities of brain function presented by functional MRI, so as to achieve cross-modal adaptive interaction and enhance the complementarity between different modality data, so as to better assist in the diagnosis of depression.

[0059] When applying the image processing method based on multimodal neural image fusion provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0060] In addition, in one or more embodiments of the present invention, the first feature vector and the second feature vector are respectively enhanced by using a common feature vector, and the expression is:

[0061]

[0062] Among them, F s represents the enhanced feature information, softmax represents the softmax function, Q c is the query vector of the information hub, is the transpose of the K vector of feature information that needs to be enhanced, V s is the V vector of the target information, d k is the dimension of the K vector.

[0063] This embodiment uses the common eigenvector obtained by adding the first eigenvector and the second eigenvector to enhance the first eigenvector and the second eigenvector respectively, which can further highlight the changes in brain structure contained in the structural magnetic resonance imaging and the lesions in brain nerve function contained in the functional magnetic resonance imaging.

[0064] The above is an image processing method based on multimodal neural image fusion provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding image processing device based on multimodal neural image fusion, such as Figure 2 As shown, including:

[0065] The multimodal feature extraction module is used to obtain structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculate the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; it is also used to extract the features of the structural magnetic resonance imaging data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; and to extract the features of the functional connection matrix through a pre-constructed two-dimensional convolutional neural network to obtain a second feature.

[0066] The progressive multimodal information enhancement module is used to obtain the first feature vector and the second feature vector corresponding to the first feature and the second feature respectively, and add the first feature vector and the second feature vector to obtain the common feature vector of the brain nerves, and perform feature enhancement on the first feature vector and the second feature vector respectively through the common feature vector to obtain the enhanced first feature vector and the enhanced second feature vector.

[0067] The multimodal feature classification module is used to concatenate the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, and input the concatenated feature vector into a pre-constructed feature classification network to obtain a classification result.

[0068] Specifically, refer to Figure 2 The multimodal feature extraction module includes a three-dimensional convolutional neural network (3D Convolutional Neural Network, 3DCNN) and a two-dimensional convolutional neural network (2D Convolutional Neural Network, 2DCNN); the progressive multimodal information enhancement module includes an information hub, a feature addition unit and two multimodal information interaction units; the multimodal feature classification module includes a feature splicing unit and a feature classifier.

[0069] Among them, 3DCNN is specifically used to extract feature information of gray matter volume, 2DCNN is specifically used to extract feature information of functional connection matrix, feature addition unit is used to add feature information extracted by 3DCNN and 2DCNN to obtain common information, information hub is used to receive common information output by feature addition unit, and send common information to multimodal information interaction unit. Feature concatenation unit is specifically used to concatenate common information with feature information output by multimodal information interaction unit, and feature classifier is used to classify concatenated feature vector output by feature concatenation unit.

[0070] Specifically, refer to Figure 3 ,The multimodal information interaction unit first enhances the feature information and the public information through the cross attention mechanism, and then ,splices the enhanced feature information through the feature splicing unit, and finally classifies the ,splice features through the multi-layer perceptron.

[0071] For the specific definition of the image processing device based on multimodal neural image fusion, please refer to the definition of the image processing method based on multimodal neural image fusion mentioned above, which will not be repeated here. Each module in the image processing device based on multimodal neural image fusion can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0072] The present invention also provides Figure 4 The structural diagram of the computer device shown in FIG. Figure 4 As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the Figure 1 Provided is an image processing method based on multimodal neural image fusion.

[0073] Those skilled in the art can understand that all or part of the processes in the embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0074] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. An image processing method based on multimodal neural image fusion, characterized in that: include: Acquiring structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculating the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; Extracting features of the structural magnetic resonance imaging data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; Extracting features of the functional link matrix through a pre-built two-dimensional convolutional neural network to obtain a second feature; Acquire a first feature vector and a second feature vector corresponding to the first feature and the second feature respectively, and add the first feature vector and the second feature vector to obtain a common feature vector of the brain nerves, and perform feature enhancement on the first feature vector and the second feature vector respectively through the common feature vector to obtain an enhanced first feature vector and an enhanced second feature vector; The common feature vector is concatenated with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, and the concatenated feature vector is input into a pre-constructed feature classification network to obtain a classification result.

2. The image processing method based on multimodal neural image fusion according to claim 1, characterized in that: The functional connection matrix between various regions in the brain nerves is calculated based on the functional magnetic resonance imaging data, and the calculation expression is: where r i and r j is the blood oxygen level dependence signal of different brain regions, cov(r i ,r j ) is r i and r j The covariance between ri and σ rj r i and r j The standard deviation of Corr(r i , r j ) is r i With r j The correlation between .

3. The image processing method based on multimodal neural image fusion according to claim 1, characterized in that: Adding the first eigenvector and the second eigenvector to obtain the common eigenvector of the brain nerves specifically includes: The first eigenvector and the second eigenvector are compressed to the same dimension through a flattening operation, and the compressed first eigenvector and the compressed second eigenvector are added to obtain a common eigenvector of the brain nerves.

4. The image processing method based on multimodal neural image fusion according to claim 1, characterized in that: The first feature vector and the second feature vector are enhanced respectively by using the common feature vector, and the expression is: Among them, F s represents the enhanced feature information, softmax represents the softmax function, Q c is the query vector of the information hub, is the transpose of the K vector of feature information that needs to be enhanced, V s is the V vector of the target information, d k is the dimension of the K vector.

5. The image processing method based on multimodal neural image fusion according to claim 1, characterized in that: The common feature vector is concatenated with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, specifically comprising: The elements in corresponding channels of the first eigenvector, the second eigenvector and the third eigenvector are added respectively to obtain the concatenated eigenvector.

6. An image processing device based on multimodal neural image fusion based on the image processing method based on multimodal neural image fusion according to claim 1, characterized in that: include: A multimodal feature extraction module, used to obtain structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculate the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; and also used to extract features of the structural magnetic resonance imaging data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; Extracting features of the functional link matrix through a pre-built two-dimensional convolutional neural network to obtain a second feature; A progressive multimodal information enhancement module, used to obtain a first feature vector and a second feature vector corresponding to the first feature and the second feature respectively, and add the first feature vector and the second feature vector to obtain a common feature vector of the brain nerves, and perform feature enhancement on the first feature vector and the second feature vector respectively through the common feature vector to obtain an enhanced first feature vector and an enhanced second feature vector; The multimodal feature classification module is used to concatenate the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, and input the concatenated feature vector into a pre-constructed feature classification network to obtain a classification result.

7. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method based on multimodal neural image fusion as claimed in any one of claims 1 to 5 is implemented.

Images