Image processing method, device and equipment based on multimodal neural image fusion

Abstract

The present invention discloses an image processing method, apparatus, and device based on multimodal neural imaging fusion, relating to the field of image recognition technology. The method comprises: calculating a functional connectivity matrix between various brain regions based on functional magnetic resonance imaging data; extracting a first feature from the structural magnetic resonance imaging data and a second feature from the functional connectivity matrix; obtaining a first eigenvector and a second eigenvector corresponding to the first and second features, adding the first and second eigenvectors to obtain a common eigenvector; enhancing the first and second eigenvectors using the common eigenvector, concatenating the enhanced first and second eigenvectors to obtain a concatenated eigenvector, and inputting the concatenated eigenvector into a pre-constructed feature classification network to obtain a classification result. This approach enhances the complementarity between data from different modalities, thereby better assisting in the diagnosis of depression.

Description

Technical Field

[0001] The present invention relates to the technical field of image data feature recognition, and in particular to an image processing method, apparatus and device based on multimodal neural image fusion. Background Art

[0002] Major Depressive Disorder (MDD) is a common mental illness characterized by long-term, persistent low and depressed moods, and even severe feelings of inferiority and loss of confidence in life. In recent years, with the launch of the Global Brain Project, relevant departments have proposed a "one body, two wings" architecture, of which "the development of effective methods for early diagnosis and intervention of brain diseases" as the core content of the "two wings" has attracted the attention of many researchers. In particular, traditional depression diagnosis methods mainly rely on subjective interviews based on scales, which are easily affected by environmental and individual differences, resulting in a low clinical recognition rate of depression. This means that how to develop an objective quantitative auxiliary diagnostic method for depression has become a must for computer-aided diagnosis (CAD) in the diagnosis of depression.

[0003] Neuroimaging technology provides a new means for objectively quantifying psychiatric disorders. Magnetic resonance imaging (MRI) non-invasively explores changes in brain structure and function. Structural MRI (sMRI) uses the interaction between magnetic fields and radio waves to visualize macrostructural changes in the brain during development and disease. Functional MRI (fMRI) tracks brain function by recording fluctuations in the blood oxygen level-dependent (BOLD) signal. Diffusion tensor imaging (DTI) studies microstructural changes in the brain by visualizing the movement of white matter fiber tracts.

[0004] In recent years, with the development of various neuroimaging technologies, leveraging their imaging characteristics, exploring the complementary information of multimodal neuroimaging, and thereby improving the diagnostic reliability of depression has become a hot topic for many researchers. Numerous studies have attempted to utilize multimodal neuroimaging data for the diagnosis of major depressive disorder. These approaches primarily focus on feature extraction and feature fusion. Research on multi-source imaging-based auxiliary diagnosis of depression continues to emerge. Based on the feature extraction and fusion methods, mainstream approaches can be divided into those based on traditional machine learning algorithms and those based on deep learning. Sun et al. extracted four features from multimodal MRI data from two centers and used a support vector machine (SVM) as a diagnostic model to investigate their role in the diagnosis of major depressive disorder (MDD). Li et al. improved the diagnostic performance of MDD by fusing common features from sMRI and fMRI using a random forest (RF) and SVM classifier. However, traditional machine learning methods often rely on manually designed feature engineering, which can easily lead to the curse of dimensionality when processing high-dimensional data, affecting the model's generalization performance. This problem can be particularly pronounced when dealing with complex tasks and large-scale data. Regarding the use of deep learning algorithms in multi-source imaging-assisted depression diagnosis, Zheng et al. designed a brain function-structure fusion module to capture the interactions between deep features extracted from different MRI modalities, aiming to improve the fusion capabilities of multimodal imaging. Qin et al. proposed a graph convolutional neural network (GCN) for identifying MDD and achieving robust diagnostic performance.

[0005] However, due to the significant modal differences between multimodal data (such as differences in data distribution and feature expression), existing deep learning-based multimodal fusion methods have not fully exploited the complementarity of multimodal features. Summary of the Invention

[0006] Based on this, it is necessary to provide image processing methods, devices and equipment based on multimodal neural image fusion to address the technical problem that existing technologies fail to fully explore the complementarity of multimodal features.

[0007] The present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides an image processing method based on multimodal neural image fusion, the method comprising:

[0009] Acquiring structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculating a functional connectivity matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data;

[0010] Extracting features of the structural magnetic resonance imaging data using a pre-constructed three-dimensional convolutional neural network to obtain a first feature; extracting features of the functional link matrix using a pre-constructed two-dimensional convolutional neural network to obtain a second feature;

[0011] Obtaining a first eigenvector and a second eigenvector corresponding to the first feature and the second feature, respectively, and adding the first eigenvector and the second eigenvector to obtain a common eigenvector of the brain nerves, and performing feature enhancement on the first eigenvector and the second eigenvector using the common eigenvector to obtain an enhanced first eigenvector and an enhanced second eigenvector;

[0012] The common feature vector is spliced ​​with the enhanced first feature vector and the enhanced second feature vector to obtain a spliced ​​feature vector, and the spliced ​​feature vector is input into a pre-constructed feature classification network to obtain a classification result.

[0013] Furthermore, the functional connectivity matrix between the various regions in the brain nerves is calculated based on the functional magnetic resonance imaging data, and the calculation expression is:

[0014]

[0015] where r i and r j is the blood oxygen level dependent signal in different brain regions, cov(r i ,r j ) is r i and r j The covariance between ri and σ rj r i and r j The standard deviation of Corr(r i , r j ) is r i With r j The correlation between .

[0016] Furthermore, the first eigenvector and the second eigenvector are added to obtain the common eigenvector of the brain nerves, specifically including:

[0017] The first eigenvector and the second eigenvector are compressed to the same dimension through a flattening operation, and the compressed first eigenvector and the compressed second eigenvector are added to obtain a common eigenvector of the brain nerves.

[0018] Furthermore, feature enhancement is performed on the first feature vector and the second feature vector respectively using the common feature vector, and the expression is:

[0019]

[0020] Among them, F s represents the enhanced feature information, softmax represents the softmax function, Q c is the query vector of the information hub, is the transpose of the K vector of feature information that needs to be enhanced, V s is the V vector of target information, d k is the dimension of the K vector.

[0021] Furthermore, the common feature vector is concatenated with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, which specifically includes:

[0022] The elements in corresponding channels of the first eigenvector, the second eigenvector, and the third eigenvector are added together to obtain the concatenated eigenvector.

[0023] In a second aspect, the present invention provides an image processing method based on multimodal neural image fusion and an image processing device based on multimodal neural image fusion, characterized in that it includes:

[0024] a multimodal feature extraction module for acquiring structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculating a functional connectivity matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; and for extracting features of the structural magnetic resonance imaging data using a pre-constructed three-dimensional convolutional neural network to obtain a first feature; and extracting features of the functional connectivity matrix using a pre-constructed two-dimensional convolutional neural network to obtain a second feature;

[0025] a progressive multimodal information enhancement module, configured to obtain a first eigenvector and a second eigenvector corresponding to the first feature and the second feature, respectively, and add the first eigenvector and the second eigenvector to obtain a common eigenvector of the brain nerves, and perform feature enhancement on the first eigenvector and the second eigenvector using the common eigenvector to obtain an enhanced first eigenvector and an enhanced second eigenvector;

[0026] The multimodal feature classification module is used to splice the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a spliced ​​feature vector, and input the spliced ​​feature vector into a pre-built feature classification network to obtain a classification result.

[0027] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image processing method based on multimodal neural image fusion is implemented.

[0028] The at least one technical solution adopted by the present invention can achieve the following beneficial effects: the present invention obtains structural magnetic resonance imaging data and functional magnetic resonance imaging data, and calculates the functional connectivity matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; extracts the features of the structural magnetic resonance imaging data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; extracts the features of the functional link matrix through a pre-constructed two-dimensional convolutional neural network to obtain a second feature; obtains a first eigenvector and a second eigenvector corresponding to the first feature and the second feature, respectively, and adds the first eigenvector and the second eigenvector to obtain a common eigenvector of the brain nerves, and performs feature enhancement on the first eigenvector and the second eigenvector respectively using the common eigenvector to obtain an enhanced first eigenvector and an enhanced second eigenvector; splices the common eigenvector with the enhanced first eigenvector and the enhanced second eigenvector to obtain a spliced ​​eigenvector, which can enhance the features of each modality data, combine the structural changes of the brain presented by the structural magnetic resonance imaging with the abnormalities of brain function presented by the functional magnetic resonance imaging, thereby realizing cross-modal adaptive interaction and enhancing the complementarity between different modality data, thereby better assisting in the diagnosis of depression. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 A schematic flow chart of the image processing method based on multimodal neural image fusion provided by the present invention;

[0031] Figure 2 Schematic diagram of an image processing device based on multimodal neural image fusion provided by the present invention;

[0032] Figure 3 A schematic diagram of the workflow of a multimodal information interaction unit provided by the present invention;

[0033] Figure 4 A schematic diagram of a computer device for implementing an image processing method based on multimodal neural image fusion provided by the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] The server mentioned in the present invention can be a server installed on a business platform, or a device such as a desktop computer or laptop computer capable of executing the solution of the present invention. For ease of explanation, the following description will only use the server as the execution entity. The following, combined with the accompanying drawings, details the technical solutions provided by various embodiments of the present invention.

[0036] refer to Figure 1 , is an image processing method based on multimodal neural image fusion in the present invention, which specifically includes the following steps:

[0037] S10: Acquire structural magnetic resonance imaging data and functional magnetic resonance imaging data, and calculate the functional connectivity matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data.

[0038] In this embodiment, the structural MRI data includes, but is not limited to, gray matter volume data of brain nerves, which is obtained by preprocessing the structural MRI data. The functional MRI includes, but is not limited to, blood oxygen level-dependent signal data of brain nerves.

[0039] Optionally, the public dataset REST-meta-MDD is used for validation. The inclusion and exclusion criteria are:

[0040] ①The fMRI scanning time points are no less than 170 time points.

[0041] ② Subjects were not repeated. Finally, a total of 1,179 patients with major depression and 1,008 healthy control participants from 20 different cohorts were included in the analysis.

[0042] Exemplary, reference Figure 2 For sMRI, the gray matter volume provided by REST-meta-MDD was used as the input for sMRI. For fMRI, the BOLD signal of 116 brain regions was acquired using the Automated Anatomical Labeling (AAL) atlas provided by REST-meta-MDD.

[0043] The functional connectivity matrix (FCM) is a matrix that represents the correlation between functional activities in different brain regions. In this matrix, rows and columns represent different brain regions, and the elements in the matrix represent the strength of the functional connections between these regions.

[0044] Specifically, the functional connectivity matrix between various regions in the brain nerves is calculated based on the blood oxygen level dependent signal data. The calculation expression is:

[0045]

[0046] where r i and r j is the blood oxygen level dependent signal in different brain regions, cov(r i ,r j ) is r i and r j The covariance between ri and σ rj r i and r j The standard deviation of Corr(r i , r j ) is r i With r j The correlation between .

[0047] Specifically, correlation refers to the degree of correlation between the blood oxygen level-dependent signal of any brain region and the blood oxygen level-dependent signals of other brain regions. For example, if the brain is divided into 90 regions using the AAL template, the correlation includes the correlation between the blood oxygen level-dependent signal of the first brain region and the blood oxygen level-dependent signals of the other 89 brain regions, and the correlation between the blood oxygen level-dependent signal of the second brain region and the blood oxygen level-dependent signals of the other 89 brain regions.

[0048] S20: extracting features of the gray matter volume data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; extracting features of the functional link matrix through a pre-constructed two-dimensional convolutional neural network to obtain a second feature.

[0049] In this embodiment, considering the limited size of the data, a stacked convolutional neural network model (CNN) is constructed as a feature encoder to realize the extraction of multimodal features. Specifically, for gray matter volume data, the present invention constructs a three-dimensional CNN to realize the feature extraction of sMRI. The three-dimensional CNN mainly includes three trainable weight layers and 3×3×3 convolution layers, and the number of convolution kernels is 64, 128, and 512 respectively. For the functional connectivity matrix, a traditional two-dimensional CNN is used as a feature extractor for feature extraction, which mainly includes a 3×3 convolution layer, and the number of convolution kernels is 64, 128, and 512 respectively.

[0050] S30: Obtain the first eigenvector and the second eigenvector corresponding to the first feature and the second feature respectively, and add the first eigenvector and the second eigenvector to obtain a common eigenvector of the brain nerves, and perform feature enhancement on the first eigenvector and the second eigenvector respectively through the common eigenvector to obtain an enhanced first eigenvector and an enhanced second eigenvector.

[0051] In this embodiment, the first eigenvector and the second eigenvector are added to obtain the common eigenvector of the brain nerves, which specifically includes:

[0052] The first eigenvector and the second eigenvector are compressed to the same dimension through a flattening operation, and the compressed first eigenvector and the compressed second eigenvector are added to obtain a common eigenvector of brain nerves.

[0053] S40: concatenating the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, and inputting the concatenated feature vector into a pre-built feature classification network to obtain a classification result.

[0054] In this embodiment, the elements in the corresponding channels of the first eigenvector, the second eigenvector, and the third eigenvector are added together to obtain a concatenated eigenvector. The feature classification network is constructed based on a multi-layer perceptron.

[0055] In the specific process of assisting depression diagnosis, the method of assisting brain nerve analysis based on splicing feature vectors is as follows:

[0056] A set of fully connected layers was constructed based on a multi-layer perceptron for feature classification. The number of neurons in the first layer of the fully connected layer was 128, and the number of neurons in the second layer was 2. The final analysis of the brain nerves determined whether the patient had severe depression or was normal.

[0057] Optionally, the number of neurons in the second layer of the fully connected layer can be modified based on the number of analysis results. For example, when the analysis results are severe depression, moderate depression, mild depression, and normal, the number of neurons in the second layer is set to 4.

[0058] based on Figure 1 The image processing method based on multimodal neural imaging fusion shown in the figure obtains structural magnetic resonance imaging data and functional magnetic resonance imaging data, and calculates the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; extracts the features of the structural magnetic resonance imaging data through a pre-constructed three-dimensional convolutional neural network to obtain a first feature; extracts the features of the functional connection matrix through a pre-constructed two-dimensional convolutional neural network to obtain a second feature; obtains the first eigenvector and the second eigenvector corresponding to the first feature and the second feature respectively, and adds the first eigenvector and the second eigenvector to obtain a common eigenvector of the brain nerves, and enhances the first eigenvector and the second eigenvector respectively through the common eigenvector to obtain an enhanced first eigenvector and an enhanced second eigenvector; splices the common eigenvector with the enhanced first eigenvector and the enhanced second eigenvector to obtain a spliced ​​eigenvector, and inputs the spliced ​​eigenvector into a pre-constructed feature classification network to obtain a classification result. Through the above scheme, the characteristics of each modality data can be enhanced, and the structural changes of the brain presented by structural magnetic resonance imaging can be combined with the abnormalities of brain function presented by functional magnetic resonance imaging, thereby realizing cross-modal adaptive interaction and enhancing the complementarity between different modality data, which can better assist in the diagnosis of depression.

[0059] When applying the image processing method based on multimodal neural image fusion provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0060] In addition, in one or more embodiments of the present invention, the first eigenvector and the second eigenvector are respectively enhanced by using a common eigenvector, and the expression thereof is:

[0061]

[0062] Among them, F s represents the enhanced feature information, softmax represents the softmax function, Q c is the query vector of the information hub, is the transpose of the K vector of feature information that needs to be enhanced, V s is the V vector of target information, d k is the dimension of the K vector.

[0063] This embodiment uses the common eigenvector obtained by adding the first eigenvector and the second eigenvector to enhance the first eigenvector and the second eigenvector respectively, which can further highlight the changes in brain structure contained in the structural magnetic resonance imaging and the lesions in brain nerve function contained in the functional magnetic resonance imaging.

[0064] The above is an image processing method based on multimodal neural image fusion provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding image processing device based on multimodal neural image fusion, such as Figure 2 Shown, including:

[0065] The multimodal feature extraction module is used to obtain structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculate the functional connection matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; it is also used to extract the features of the structural magnetic resonance imaging data through a pre-built three-dimensional convolutional neural network to obtain a first feature; and to extract the features of the functional connection matrix through a pre-built two-dimensional convolutional neural network to obtain a second feature.

[0066] The progressive multimodal information enhancement module is used to obtain the first eigenvector and the second eigenvector corresponding to the first feature and the second feature respectively, and add the first eigenvector and the second eigenvector to obtain the common eigenvector of the brain nerves, and perform feature enhancement on the first eigenvector and the second eigenvector respectively through the common eigenvector to obtain the enhanced first eigenvector and the enhanced second eigenvector.

[0067] The multimodal feature classification module is used to splice the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a spliced ​​feature vector, and input the spliced ​​feature vector into a pre-built feature classification network to obtain a classification result.

[0068] Specifically, refer to Figure 2 The multimodal feature extraction module includes a three-dimensional convolutional neural network (3DCNN) and a two-dimensional convolutional neural network (2DCNN); the progressive multimodal information enhancement module includes an information hub, a feature addition unit and two multimodal information interaction units; the multimodal feature classification module includes a feature splicing unit and a feature classifier.

[0069] Specifically, the 3DCNN is used to extract feature information of gray matter volume, the 2DCNN is used to extract feature information of the functional connectivity matrix, the feature addition unit is used to add the feature information extracted by the 3DCNN and 2DCNN to obtain common information, the information hub is used to receive the common information output by the feature addition unit and send the common information to the multimodal information interaction unit. The feature splicing unit is used to perform feature splicing on the common information and the feature information output by the multimodal information interaction unit, and the feature classifier is used to classify the spliced ​​feature vector output by the feature splicing unit.

[0070] Specifically, refer to Figure 3 The multimodal information interaction unit first enhances the enhanced feature information and the public information through the cross-attention mechanism, then splices the enhanced feature information through the feature splicing unit, and finally classifies the spliced ​​features through the multi-layer perceptron.

[0071] For the specific definition of the image processing device based on multimodal neural image fusion, please refer to the definition of the image processing method based on multimodal neural image fusion above, which will not be repeated here. The various modules in the image processing device based on multimodal neural image fusion can be implemented in whole or in part by software, hardware and a combination thereof. The modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0072] The present invention also provides Figure 4 The structural diagram of the computer equipment shown in FIG. Figure 4 As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the Figure 1 Provided is an image processing method based on multimodal neural imaging fusion.

[0073] Those skilled in the art will appreciate that all or part of the processes in the embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the various methods described. Among them, any reference to memory, storage, database or other media used in the various embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0074] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. An image processing method based on multimodal neural image fusion, characterized in that: include: Acquiring structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculating a functional connectivity matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; Extracting features of the structural magnetic resonance imaging data using a pre-built three-dimensional convolutional neural network to obtain a first feature; Extracting features of the functional connectivity matrix through a pre-built two-dimensional convolutional neural network to obtain a second feature; Obtaining a first eigenvector and a second eigenvector corresponding to the first feature and the second feature, respectively, and adding the first eigenvector and the second eigenvector to obtain a common eigenvector of the brain nerves, and performing feature enhancement on the first eigenvector and the second eigenvector using the common eigenvector to obtain an enhanced first eigenvector and an enhanced second eigenvector; Splicing the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a spliced ​​feature vector, and inputting the spliced ​​feature vector into a pre-built feature classification network to obtain a classification result; The functional connection matrix between the various regions in the brain nerves is calculated based on the functional magnetic resonance imaging data, and the calculation expression is: ； in r i and r j is the blood oxygen level dependent signal in different brain regions, cov( r i , r j )for r i and r j The covariance between σ ri and σ rj They are r i and r j The standard deviation of Corr ( r i , r j )for r i and r j The correlation between Adding the first eigenvector and the second eigenvector to obtain the common eigenvector of the brain nerves specifically includes: The first eigenvector and the second eigenvector are compressed to the same dimension through a flattening operation, and the compressed first eigenvector and the compressed second eigenvector are added to obtain a common eigenvector of the brain nerves.

2. The image processing method based on multimodal neural image fusion according to claim 1, characterized in that: The first feature vector and the second feature vector are enhanced respectively by using the common feature vector, and the expression is: ； in, F s Represents the enhanced feature information, softmax represents the softmax function, Q c is the query vector of the information hub, is the transpose of the K vector of feature information that needs to be enhanced, V s is the V vector of target information, d k is the dimension of the K vector.

3. The image processing method based on multimodal neural image fusion according to claim 1, wherein: The common feature vector is concatenated with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, specifically comprising: The elements in corresponding channels of the first eigenvector, the second eigenvector, and the third eigenvector are added together to obtain the concatenated eigenvector.

4. An image processing device based on multimodal neural image fusion based on the image processing method based on multimodal neural image fusion according to claim 1, characterized in that: include: a multimodal feature extraction module, configured to obtain structural magnetic resonance imaging data and functional magnetic resonance imaging data of brain nerves, and calculate a functional connectivity matrix between various regions in the brain nerves based on the functional magnetic resonance imaging data; and further configured to extract features from the structural magnetic resonance imaging data using a pre-built three-dimensional convolutional neural network to obtain a first feature; Extracting features of the functional connectivity matrix through a pre-built two-dimensional convolutional neural network to obtain a second feature; a progressive multimodal information enhancement module, configured to obtain a first eigenvector and a second eigenvector corresponding to the first feature and the second feature, respectively, and add the first eigenvector and the second eigenvector to obtain a common eigenvector of the brain nerves, and perform feature enhancement on the first eigenvector and the second eigenvector using the common eigenvector to obtain an enhanced first eigenvector and an enhanced second eigenvector; a multimodal feature classification module, configured to concatenate the common feature vector with the enhanced first feature vector and the enhanced second feature vector to obtain a concatenated feature vector, and input the concatenated feature vector into a pre-built feature classification network to obtain a classification result; The functional connection matrix between the various regions in the brain nerves is calculated based on the functional magnetic resonance imaging data, and the calculation expression is: ； in r i and r j is the blood oxygen level dependent signal in different brain regions, cov( r i , r j )for r i and r j The covariance between σ ri and σ rj They are r i and r j The standard deviation of Corr ( r i , r j )for r i and r j The correlation between Adding the first eigenvector and the second eigenvector to obtain the common eigenvector of the brain nerves specifically includes: The first eigenvector and the second eigenvector are compressed to the same dimension through a flattening operation, and the compressed first eigenvector and the compressed second eigenvector are added to obtain a common eigenvector of the brain nerves.

5. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method based on multimodal neural image fusion according to any one of claims 1 to 3 is implemented.

Images