An alzheimer's disease multi-modal medical image classification method based on spatial perception decoupling

Abstract

This invention belongs to the field of disease classification technology, specifically relating to a multimodal medical image classification method for Alzheimer's disease based on spatial perception decoupling. It includes inputting a high-dimensional spatial feature sequence into a decoupled bidirectional state space processing module. After injecting two-dimensional spatial location prior information into the high-dimensional spatial feature sequence, it independently extracts the forward long-range dependency information and backward sequence long-range dependency information corresponding to each orthogonal slice. This invention uses physically isolated independent forward processing units and independent backward processing units to construct the decoupled bidirectional state space processing module, blocking the cross-contamination problem of forward and backward hidden states in traditional bidirectional scanning mechanisms, effectively avoiding gradient anomalies during model training. Simultaneously, by introducing a learnable two-dimensional spatial location encoding matrix, two-dimensional spatial topological prior information is re-injected into the flattened one-dimensional feature sequence, compensating for the loss of spatial structure information caused by serialization.

Description

Technical Field

[0001] This invention belongs to the field of disease classification technology, specifically relating to a multimodal medical image classification method for Alzheimer's disease based on spatial perception decoupling. Background Technology

[0002] Early and accurate diagnosis of Alzheimer's Disease (AD), a neurodegenerative disease, is crucial for slowing disease progression. Multimodal medical imaging, particularly the combined analysis of structural magnetic resonance imaging (sMRI) and positron emission tomography (PET), can simultaneously provide detailed anatomical and functional metabolic information of the brain, offering a more comprehensive data foundation for computer-aided diagnosis of AD. Currently, deep learning-based multimodal fusion methods have become mainstream research, aiming to improve classification accuracy by integrating brain atrophy patterns revealed by sMRI with abnormal glucose metabolism patterns reflected by PET.

[0003] However, these existing methods still have significant shortcomings in achieving efficient and in-depth multimodal information fusion: existing fusion strategies often fail to fully and adaptively mine and utilize the complex and deep complementary and synergistic relationships between features of different modalities. Specifically, many methods employ relatively simple late-stage feature concatenation or decision-level fusion, and this loosely coupled approach fails to achieve the interaction and mutual enhancement of fine-grained features between modalities. Although some studies have introduced attention mechanisms to improve fusion results, their attention interactions are usually unidirectional (e.g., retrieving PET features using only sMRI features as a query) or limited to the superficial level, failing to model the naturally occurring, complex bidirectional correlations and mutual corroboration relationships between brain structure and function (e.g., structural atrophy in specific brain regions and metabolic reduction are mutually corroborating pathological markers). Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this invention provides a multimodal medical image classification method for Alzheimer's disease based on spatial perception decoupling, comprising: S1. Obtain structural magnetic resonance imaging modal medical images and positron emission tomography modal medical images of the subject, and extract several orthogonal section slices from the two modal medical images respectively; S2. Input each orthogonal slice into the image feature extraction backbone module to extract high-dimensional spatial feature sequences; S3. Input the high-dimensional spatial feature sequence into the decoupled bidirectional state space processing module. After injecting two-dimensional spatial location prior information into the high-dimensional spatial feature sequence, independently extract the forward long-range dependency information and backward sequence long-range dependency information corresponding to each orthogonal sectional slice. S4. Input the features containing long-range dependency information of forward and backward sequences into the cross-section interaction module. Within the same modality, integrate the spatial structure information and metabolic information from different orthogonal cross-section perspectives, and output the cross-section fusion features of the corresponding modality. S5. Input the cross-modal fusion features of each modality into the bidirectional cross-modal attention fusion module, perform bidirectional mutual retrieval and feature enhancement on the cross-modal fusion features of the two modalities, and output cross-modal enhanced features; S6. Input the cross-sectional fusion features from step S4 and the cross-modal enhancement features from step S5 into the dynamic gating and classification module, adaptively allocate the fusion weights of the cross-sectional fusion features and the cross-modal enhancement features, and output the Alzheimer's disease classification prediction results based on the final fusion features after weight allocation.

[0005] The beneficial effects of this invention are as follows: This invention employs physically isolated, independent forward and backward processing units to construct a decoupled bidirectional state-space processing module. This blocks the cross-contamination problem between forward and backward hidden states in traditional bidirectional scanning mechanisms, effectively avoiding gradient anomalies during model training. Simultaneously, by introducing a learnable two-dimensional spatial position encoding matrix, two-dimensional spatial topological prior information is re-injected into the flattened one-dimensional feature sequence, compensating for the loss of spatial structure information caused by serialization. This "physical isolation and decoupling followed by nonlinear fusion" architecture design enables the model to stably handle large-scale, highly heterogeneous patient data while maintaining the linear computational efficiency of the state-space model. The feature extraction process is more stable, effectively suppressing the risk of overfitting.

[0006] This invention makes key structural improvements to the terminal residual block group of deep residual networks, eliminating the terminal downsampling operation and setting the stride parameters of both the core convolutional layer and the residual bypass downsampling layer to (1,1). When the input size is 112×112 pixels, the output feature map size remains at 14×14 instead of the usual 4×4, increasing the length of the flattened high-dimensional spatial feature sequence from 16 to 196, a 12.25-fold improvement. This improvement overcomes the bottleneck of modeling long-sequence medical features without incurring additional computational pressure, not only preserving the microscopic details of lesions but also establishing macroscopic long-range dependencies across brain regions, laying a structural foundation for subsequent state-space models to fully extract deep microscopic lesion information.

[0007] The first and second cross-attention pathways constructed in this invention respectively realize the clinical diagnostic logic of "precisely focusing on high- or low-metabolic regions using structurally atrophied areas" and "inferring structural changes in specific brain regions from metabolic hotspots." This bidirectional query-key-value mechanism highly replicates the clinical thinking of medical experts who "cross-verify structure and metabolism" when reviewing images. Compared to traditional feature splicing or unidirectional attention mechanisms, it can capture the early Alzheimer's disease patients and their pathological characteristics, significantly improving the model's recall sensitivity to positive case features.

[0008] The content-aware dynamic gating network introduced in this invention can dynamically generate adaptive fusion weights based on the content distribution of the current input features, and intelligently allocate the retention ratio of original aggregated features and cross-modal enhanced features. This adaptive fusion mechanism based on feature content enables the model to flexibly cope with the heterogeneity of pathological features of different subjects in complex clinical auxiliary diagnostic scenarios, significantly reducing the misdiagnosis rate and improving the recall accuracy of real patient samples. It provides an efficient, robust, and clinically valuable underlying algorithm architecture for the accurate classification of neurodegenerative diseases. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention.

[0011] Figure 2 This is a schematic diagram of the execution flow of the method of the present invention.

[0012] Figure 3 This is a schematic diagram of the execution flow for decoupling the bidirectional state space processing module.

[0013] Figure 4 This is a schematic diagram of the execution flow of the bidirectional cross-modal cross-attention fusion module. Detailed Implementation

[0014] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the specific embodiments. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0016] Figure 1 This is a schematic flowchart illustrating a multimodal medical image classification method for Alzheimer's disease based on spatial perception decoupling, provided by the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0017] like Figure 1 As shown, the method includes: S1. Obtain structural magnetic resonance imaging modal medical images and positron emission tomography modal medical images of the subject, and extract several orthogonal section slices from the two modal medical images respectively; S2. Input each orthogonal slice into the image feature extraction backbone module to extract high-dimensional spatial feature sequences; S3. Input the high-dimensional spatial feature sequence into the decoupled bidirectional state space processing module. After injecting two-dimensional spatial location prior information into the high-dimensional spatial feature sequence, independently extract the forward long-range dependency information and backward sequence long-range dependency information corresponding to each orthogonal sectional slice. S4. Input the features containing long-range dependency information of forward and backward sequences into the cross-section interaction module. Within the same modality, integrate the spatial structure information and metabolic information from different orthogonal cross-section perspectives, and output the cross-section fusion features of the corresponding modality. S5. Input the cross-modal fusion features of each modality into the bidirectional cross-modal attention fusion module, perform bidirectional mutual retrieval and feature enhancement on the cross-modal fusion features of the two modalities, and output cross-modal enhanced features; S6. Input the cross-sectional fusion features from step S4 and the cross-modal enhancement features from step S5 into the dynamic gating and classification module, adaptively allocate the fusion weights of the cross-sectional fusion features and the cross-modal enhancement features, and output the Alzheimer's disease classification prediction results based on the final fusion features after weight allocation.

[0018] To facilitate understanding of the present invention, the following describes the principle of the Alzheimer's disease multimodal medical image classification method based on spatial perception decoupling, combined with the process of classifying Alzheimer's disease multimodal medical images based on spatial perception decoupling in the embodiments, and in conjunction with, for example... Figure 2The execution flow of the Alzheimer's disease multimodal medical image classification method based on spatial perception decoupling provided in the embodiment of the present invention is shown below, which further describes the Alzheimer's disease multimodal medical image classification method based on spatial perception decoupling provided by the present invention.

[0019] First, step S1 includes: S11. After converting the acquired original structural magnetic resonance imaging images and positron emission tomography images into different formats, spatial positioning and registration are performed based on the previous joint. S12. Perform tissue segmentation, image standardization, and cranial removal on the registered image; S13. Extract slices from each modality's 3D image at equal intervals along the sagittal, coronal, and axial planes, and select representative orthogonal slices from each direction based on the information entropy maximization criterion.

[0020] Specifically, the original structural magnetic resonance imaging (MRI) modal medical images and original positron emission tomography (PET) modal medical images of the subjects were obtained from the medical imaging database. The original images were in DICOM format. The original DICOM format structural MRI modal medical images and original PET modal medical images were converted into NIFTI format three-dimensional image files using image processing tools. The converted structural MRI three-dimensional images and PET three-dimensional images were spatially located, and the images were reconstructed based on the previous joint to ensure that the spatial coordinate systems of different modal images were consistent. The reconstructed structural MRI modal medical images were used as reference images to perform spatial registration and alignment processing on the PET modal medical images.

[0021] The registered and aligned structural magnetic resonance imaging (MRI) 3D images were segmented into images containing different tissue components, including gray matter, white matter, and cerebrospinal fluid. Both the MRI and positron emission tomography (PET) 3D images were normalized to ensure pixel values ​​conformed to a standard normal distribution. Gaussian smoothing filtering was then applied to the normalized PET images to reduce noise and improve image quality. Finally, the filtered MRI and PET images underwent skull and non-brain tissue removal. Non-brain tissue included the skull, scalp, eyeballs, and neck tissue to eliminate interference from non-brain tissue in subsequent feature extraction.

[0022] The three-dimensional images of the completed craniotomy are sliced. Two-dimensional slices are extracted from each modality of the three-dimensional images of each subject along the sagittal, coronal and axial planes at equal intervals. From several slices in each direction, the single slice containing the most feature information is selected based on the information entropy maximization criterion. Three orthogonal slices, namely sagittal slice, coronal slice and axial slice, are selected from each modality to form several orthogonal slice output results of step S1.

[0023] This embodiment selected data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database of subjects meeting the inclusion criteria. Inclusion criteria included: clinically diagnosed subjects with normal cognition (NC), mild cognitive impairment (MCI), and Alzheimer's disease (AD); aged between 55 and 90 years; possessing complete structural magnetic resonance imaging (MRI) T1-weighted images and fluorodeoxyglucose (FDG) positron emission tomography (PET) images; and image quality meeting visual assessment criteria, without severe motion artifacts, metallic artifacts, or other image quality issues affecting diagnosis. Exclusion criteria included: other neurological diseases such as Parkinson's disease, vascular dementia, Lewy body dementia, etc.; a history of severe mental illness or substance abuse; and severe quality defects in structural MRI or PET images.

[0024] The raw images are stored in a medical digital imaging and communication standard format, which contains pixel data and rich metadata information, including key imaging parameters such as subject information, scanning equipment parameters, scanning sequence parameters, slice thickness, slice spacing, field of view size, matrix size, repetition time, and echo time. This metadata information is of great guiding significance for parameter settings in subsequent image preprocessing steps.

[0025] This embodiment uses the Statistical Parametric Mapping Toolkit (SPM12 or SPM25 version) in the MATLAB software platform for format conversion. The Statistical Parametric Mapping Toolkit provides specialized functions for reading standard formats in medical digital imaging and communications and writing formats in the Neuroimaging Information Technology Initiative (NITI A) format. It can accurately parse pixel data and metadata information in standard formats in medical digital imaging and communications and convert them into NITI A format.

[0026] The specific operations of the conversion process include: first, using the spm_dicom_headers function to read the header information of the medical digital imaging and communication standard format file, and extracting key parameters such as image dimensions, voxel size, and spatial orientation; then, using the spm_dicom_convert function to convert the pixel data into a three-dimensional matrix form; and finally, using the niftiwrite function or the write function built into the statistical parametric mapping toolkit to save the three-dimensional matrix as a neuroimaging information technology initiative format file.

[0027] The Neuroimaging Information Technology Initiative (NITI Aayog) format adopts a unified coordinate system definition, setting the origin of the image at the lower left corner, with the x-axis pointing to the right, the y-axis pointing forward, and the z-axis pointing upward. This standardized coordinate definition ensures that images acquired from different modalities, scanning devices, and time points can be compared and analyzed within a unified spatial framework.

[0028] The anterior commissure is an important anatomical landmark in the brain, located at the junction of the anterior wall of the third ventricle and the lamina terminalis. On structural magnetic resonance imaging (MRI) T1-weighted images, it appears as a low-signal punctate structure, while on positron emission tomography (PET) images, it shows a region with relatively low metabolic activity. The anterior commissure was chosen as a reference point for spatial localization because this structure is well-identifiable in both modalities of imaging, its location is relatively fixed, and it is less affected by pathological changes.

[0029] The specific steps for spatial localization include: opening the converted Neuroimaging Information Technology Initiative (NITI A) format image in the Statistical Parametric Mapping Toolkit; using the display tool to browse the image in three orthogonal cross-sectional views; and manually locating the anterior commissure. Localization requires cross-validation in the sagittal, coronal, and axial views simultaneously to ensure accuracy. After localization, the spatial repositioning function of the Statistical Parametric Mapping Toolkit is used to set the anterior commissure's position as the image's origin coordinates (0,0,0), and the image is then translated accordingly, re-establishing the entire image's spatial coordinate system based on this anterior commissure position.

[0030] This spatial localization step ensures that the structural magnetic resonance imaging (MRI) 3D images and positron emission tomography (PET) 3D images of the same subject have a unified spatial coordinate origin, laying the foundation for subsequent image registration and alignment operations. Simultaneously, the establishment of a coordinate system based on standardized anatomical landmarks makes images from different subjects comparable, facilitating group-level statistical analysis.

[0031] Tissue segmentation is a crucial step in structural magnetic resonance imaging (MRI) image processing, aiming to separate brain tissue into different tissue types for targeted analysis. This embodiment employs the Unified Segmentation Algorithm from the Statistical Parametric Mapping Toolkit (SPMAP) for tissue segmentation. This algorithm, based on a Gaussian mixture model and the expectation-maximization algorithm, is capable of accurately classifying tissues in T1-weighted images.

[0032] The specific inputs to the segmentation process include: the registered and aligned structural magnetic resonance imaging (MRI) 3D image, a pre-defined tissue probability map (e.g., a tissue probability map or a probability map of a specific population), and algorithm parameter settings (e.g., the number of components in the Gaussian mixture model, the regularization parameters for bias field correction, etc.). The algorithm outputs include: gray matter probability map, white matter probability map, cerebrospinal fluid probability map, bias field corrected image, and deformation field.

[0033] In the gray matter probability map, each voxel value represents the probability that the location belongs to gray matter tissue, ranging from 0 to 1; the same logic applies to the white matter probability map and the cerebrospinal fluid probability map. These probability maps preserve tissue boundary information, avoiding information loss caused by hard segmentation. In subsequent analysis, the probability maps can be used as weights, or the probabilities can be... Figure 2 Value-based methods are used to obtain a mask for the tissue.

[0034] The purpose of tissue segmentation is threefold: First, to distinguish between brain tissue and non-brain tissue, providing a basis for subsequent skull dissection; second, to separate gray matter and white matter, as the pathological changes in Alzheimer's disease mainly occur in the gray matter region, especially gray matter atrophy in the hippocampus, entorhinal cortex, temporal lobe, and parietal lobe, which are important imaging markers for diagnosis; and third, the segmentation of cerebrospinal fluid helps to assess the degree of ventricular enlargement and brain atrophy.

[0035] Removing skull and non-brain tissue is a crucial step in medical image preprocessing. Its purpose is to limit the analysis to the brain tissue itself, eliminating interference from non-brain tissue in feature extraction and classification. Non-brain tissue includes the skull, scalp, muscles, fat, eyeballs, optic nerve, and neck tissue. These tissues may have high signal intensity in structural magnetic resonance imaging (MRI) and positron emission tomography (PET) images, but they are unrelated to the pathological changes of Alzheimer's disease. Failure to remove them would introduce noise and interference.

[0036] This embodiment employs a skull dissection method based on tissue probability maps. For structural magnetic resonance imaging (MRI) images, using the gray matter probability map, white matter probability map, and cerebrospinal fluid probability map obtained in step S15, voxels whose sum of the three tissue probabilities is greater than 0.5 are defined as brain tissue masks. For positron emission tomography (PET) images, since the tissue contrast in this mode is relatively poor, a method is adopted to directly apply the registered structural MRI brain tissue mask to the PET images.

[0037] After obtaining the brain tissue mask, the voxel pixel values ​​outside the mask are set to zero or background values ​​to obtain a 3D image after removing the skull and non-brain tissue. This operation significantly reduces the amount of image data, lowers the computational complexity of subsequent processing, and eliminates interference factors that may be introduced by non-brain tissue, allowing subsequent feature extraction to focus on the pathological changes in the brain tissue itself.

[0038] Slicing is the process of converting a three-dimensional volumetric image into a two-dimensional slice image, which serves as a bridge connecting three-dimensional medical images and two-dimensional deep learning networks. In this embodiment, slices are extracted from three orthogonal directions: sagittal plane (cut along the left-right direction to display the anterior-posterior and superior-inferior directions), coronal plane (cut along the anterior-posterior direction to display the left-right and superior-inferior directions), and axial plane (cut along the superior-inferior direction to display the left-right and anterior-posterior directions).

[0039] Slice extraction employs an equal-interval method, meaning that multiple candidate slices are extracted from one end of the image to the other in each direction at fixed intervals. For example, in the sagittal plane, if the image size in the x-axis is 181 voxels, and the extraction interval is set to 10 voxels, approximately 18 candidate slices can be extracted.

[0040] The candidate slices are selected based on the amount of feature information contained. In this embodiment, the information entropy maximization criterion is used for selection. Information entropy is a concept in information theory that measures the amount of information. For images, information entropy reflects the complexity of the image content and the uniformity of gray-level distribution. The formula for calculating information entropy is: ; Where H represents information entropy; i represents gray level (0 to 255); p(i) represents the probability of gray level i appearing in the image; This represents a logarithmic function with base 2. A higher information entropy indicates a more dispersed grayscale distribution in the image, containing richer information; a lower information entropy indicates a more concentrated grayscale distribution, potentially corresponding to background areas or single tissue regions.

[0041] For each candidate slice in each direction, its information entropy value is calculated, and the slice with the highest information entropy is selected as the representative slice for that direction. This selection criterion ensures that the selected slice contains the richest anatomical and pathological information, avoiding the selection of slices located at the image edge or mainly containing background areas.

[0042] Finally, three orthogonal slices (sagittal, coronal, and axial) are selected from each modality (structural magnetic resonance imaging and positron emission tomography), resulting in a total of six two-dimensional slice images across both modalities. These slice images serve as the output of several orthogonal slices from step S1. These slice images retain key anatomical structures and pathological features from the original three-dimensional images, while also having a uniform size and format, facilitating input into subsequent deep learning networks for processing.

[0043] Secondly, the image feature extraction backbone module in step S2 adopts an improved deep residual network architecture that includes residual block groups. The specific improvements include: canceling the downsampling operation of the terminal residual block group, and explicitly setting the stride parameters of the core convolutional layer and the downsampling layer in the residual bypass connection within the terminal residual block group to 1, so that the orthogonal slices output by the terminal residual block group maintain a high-resolution spatial feature map and are flattened along the spatial dimension into a one-dimensional high-dimensional spatial feature sequence.

[0044] Specifically, the image feature extraction backbone module in step S2 adopts an improved deep residual network architecture. The improved deep residual network architecture includes an initial convolutional layer, a batch normalization layer, a modified linear unit activation layer, a max pooling layer, and four residual block groups. The first, second, and third residual block groups maintain the downsampling configuration of the original residual network, while the fourth residual block group is improved as the final residual block group. In the fourth residual block group, the end downsampling operation is cancelled, and the stride parameter of the core convolutional layer in the fourth residual block group is explicitly set from the original (2,2) to (1,1). The stride parameter of the downsampling layer in the residual bypass connection in the fourth residual block group is also set to (1,1). The high-resolution spatial feature map output by the orthogonal slice through the fourth residual block group is flattened along the spatial dimension into a one-dimensional high-dimensional spatial feature sequence.

[0045] For example, when the input orthogonal slice size is 112×112 pixels, after successive downsampling by the first, second, and third residual block groups, the feature map spatial size is reduced to 14×14 pixels. Since the fourth residual block group cancels the downsampling operation, the final output high-resolution spatial feature map size remains 14×14 pixels, which is 12.25 times higher than the 4×4 pixel feature map output by the original residual network. The length L of the high-dimensional spatial feature sequence is determined by the spatial size of the feature map, and the calculation formula is: ; in, Indicates the height dimension of the output feature map; The width dimension of the output feature map is represented by L. For a feature map with a size of 14×14 pixels, the length of the high-dimensional feature sequence is L=14×14=196. The channel dimension D of the high-dimensional feature sequence is determined by the number of output channels of the fourth residual block group. The dimension of the high-dimensional feature sequence is represented as [Batch,L,D], where Batch represents the batch size, L represents the sequence length, and D represents the feature dimension. This high-dimensional feature sequence serves as the input to step S3, providing a long sequence data foundation containing rich fine-grained spatial information for the subsequent decoupled bidirectional state space processing module.

[0046] In the fourth residual block group, canceling the end-of-line downsampling operation further includes: The fourth residual block group contains multiple residual learning units. Each residual learning unit consists of a main path and a residual bypass path. The main path sequentially includes the first convolutional layer, the first batch normalization layer, the modified linear unit activation function, the second convolutional layer, and the second batch normalization layer. The residual bypass path is divided into an identity mapping path and a projection shortcut path depending on whether downsampling is required. In the original residual network configuration, when the spatial size needs to be reduced, the stride parameter of the second convolutional layer in the main path is set to (2,2), and the residual bypass path adopts the projection shortcut path, which contains a convolutional layer with a stride of (2,2) to match the change in spatial dimension. In this improved structure, the stride parameter s of the second convolutional layer in the main path is changed from 2 to 1. The stride parameter s controls the interval distance at which the convolutional kernel slides on the feature map. The calculation relationship of the stride parameter is as follows: ; ; in, and These represent the height and width of the input feature map, respectively. and These represent the height and width of the output feature map, respectively; p represents the number of pixels with zero padding; k represents the size of the convolution kernel; and s represents the stride parameter. This indicates the floor function; When the step size parameter s is set to 1, the spatial size of the output feature map is consistent with the spatial size of the input feature map, that is... and This achieves the preservation of spatial resolution; Simultaneously, the stride parameter of the convolutional layer in the projection shortcut path in the residual bypass path is also set to (1,1), so that the feature map spatial size output by the main path and the residual bypass path is completely consistent. No additional spatial dimension matching operation is required. The output of the main path and the output of the residual bypass path are directly added element by element to obtain the output features of the residual learning unit. Through the above structural improvements to the fourth residual block group, without increasing the network depth and the number of parameters, the length of the high-dimensional spatial feature sequence input to the subsequent decoupled bidirectional state space processing module is expanded from the original 16 to 196, which multiplies the ability of the state space model to model long-range dependencies, while preserving the fine-grained spatial structure information of the small lesion area.

[0047] like Figure 3 As shown, the decoupled bidirectional state-space processing module in step S3 includes a spatial position encoding unit, an independent forward processing unit, an independent backward processing unit, and a nonlinear fusion unit. Step S3 specifically includes: S31. The high-dimensional spatial feature sequence is fused with the generated two-dimensional spatial location coding matrix to obtain a location-aware feature sequence injected with topological prior information. S32. Input the position-aware feature sequence into the independent forward processing unit and the independent backward processing unit respectively; The independent forward processing unit performs forward causal scanning modeling on the position-aware feature sequence through the pre-built first state space model to obtain the long-range dependency information of the forward sequence; The independent backward processing unit flips the position-aware feature sequence in the time dimension, performs reverse causal scanning modeling through the pre-built second state space model, and flips it again to restore alignment to obtain the long-range dependency information of the backward sequence. S33. Input the long-range dependency information of the forward and backward sequences into the nonlinear fusion unit, splice them in the channel dimension, and perform fusion and dimensionality reduction through layer normalization, linear mapping and nonlinear activation processing in sequence. S34. Perform residual connection between the fused and dimensionality-reduced features and the input position-aware feature sequence to obtain the final output of the decoupled bidirectional state-space processing module.

[0048] Specifically, a two-dimensional spatial location coding matrix matching the dimension of the high-dimensional spatial feature sequence is generated through a spatial location coding unit; the high-dimensional spatial feature sequence obtained in step S2 is added element-wise to the two-dimensional spatial location coding matrix to inject two-dimensional spatial topological prior information into the high-dimensional spatial feature sequence, thereby obtaining a location-aware feature sequence. ; Where x represents the input one-dimensional high-dimensional spatial feature sequence; Represents a learnable two-dimensional spatial location encoding matrix; This represents the position-aware feature sequence after the injection of position information.

[0049] Location-aware feature sequences Separately input independent forward processing units and independent backward processing units: The independent forward processing unit contains a first state space model, which is a position-aware feature sequence of the input. Forward causal scanning modeling is performed, and features at each position are processed sequentially from t=1 to t=L according to the time step order to obtain the long-range dependency information of the forward sequence. Forward causal scanning ensures that each location can only access information from its current and previous locations; the independent backward processing unit contains a second state space model, which processes the input location-aware feature sequence. Perform a flipping operation in the time dimension to transform the location-aware feature sequence. The original sequence [1,2,…,L] is converted to the reverse order [L,L] [1,…,1], a reverse causal scan model is performed on the flipped sequence. The features at each position are processed sequentially from t=L to t=1 according to the reverse time step. After the scan is completed, the output sequence is flipped a second time to restore the original time step alignment, thus obtaining the long-range dependency information of the backward sequence. The first state-space model and the second state-space model have the same network structure but their parameters are independent of each other.

[0050] Long-range dependency information of the forward sequence and backward sequence long-range dependency information The input is fed into a nonlinear fusion unit, where it is concatenated along the channel dimension. Then, it undergoes layer normalization, fully connected layer mapping, and nonlinear activation functions for fusion and dimensionality reduction, yielding the final output features of the decoupled bidirectional state-space processing module. : ; in, This indicates a channel dimension concatenation operation, which concatenates the output features from two directions along dimension D, resulting in a 2D dimension. This represents the weight matrix of the fully connected layer, with dimensions [2D,D]. represents the bias vector of the fully connected layer, with dimension [D]; LayerNorm represents the layer normalization operation; GELU represents the Gaussian error linear unit activation function; This represents the output features after fusion, with the dimensions restored to [Batch, L, D].

[0051] The fused output features With enhanced feature sequences Perform residual connections, obtain the final output features by adding them element by element, and use the final output features as the input for step S4.

[0052] Furthermore, both the first and second state space models are built on a structured state space model architecture, which includes an input linear projection layer, a state space parameter generation layer, a selective scanning mechanism, and an output linear projection layer. The input linear projection layer maps the input features to an extended dimension representation, and the state space parameter generation layer dynamically generates the state transition matrix and the parameters of the input matrix based on the input content.

[0053] In the state-space parameter generation layer, an adaptive nonlinear mapping is performed on the time step parameter Δt of the control state transition process. The time step parameter Δt determines the dynamic transmission range and discretization step size of the feature information in the state space. The input feature z is mapped to an intermediate variable through a linear projection layer. The mathematical expression for linear projection is: ; Where z represents the feature vector input to the state space parameter generation layer; This represents the weight matrix of the time-step projection layer; This represents the bias vector of the projection layer at the time step. This represents an intermediate variable corresponding to the time step.

[0054] For intermediate variables The initial time step estimate is obtained by applying the softplus activation function for nonlinear transformation. The mathematical expression of the softplus activation function is as follows: ; Where z represents the input variable; exp represents the exponential function; ln represents the natural logarithm function; the softplus activation function ensures that the output value is always positive while maintaining smooth gradient characteristics.

[0055] A truncation operation is appended after the softplus activation function to relax the upper limit truncation threshold of the initial time step estimate to 10.0, resulting in the final time step parameter Δt. The mathematical expression for the truncation operation is as follows: ; Wherein, min represents the minimum value operation; 10.0 represents the preset truncation threshold, and the truncation operation ensures that the time step parameter Δt is in the range of (0, 10.0).

[0056] By relaxing the truncation threshold from the usual value of 1.0 or 2.0 to 10.0, the dynamic expression range of the time step parameter Δt is significantly expanded, enabling the first state space model and the second state space model to adapt to the large dynamic range differences in medical image features. At the same time, it effectively suppresses the training instability problem caused by gradient explosion when processing very deep sequences, and improves the model's ability to model long-range dependencies and the convergence stability of the training process.

[0057] Next, the cross-section interaction module in step S4 is constructed using a self-attention mechanism. Specifically, it includes: stacking the sagittal, coronal, and axial section features processed by the decoupled bidirectional state space processing module within the same modality into a view feature sequence in a preset order; the cross-section interaction module linearly maps the view feature sequence into a query matrix, a key matrix, and a value matrix, respectively; obtaining the attention weight distribution between each orthogonal section by calculating the similarity distribution between the query matrix and the key matrix; and performing weighted aggregation and multi-head feature concatenation on the value matrix based on the attention weight distribution, finally outputting a section fusion feature that integrates view information from different orthogonal sections.

[0058] Specifically, step S4 includes: S41. The output of step S3 contains long-range dependency information of the forward sequence. Long-range dependency information of backward sequence The features are grouped according to modal type. For structural magnetic resonance imaging (MRI) modality, the corresponding sagittal, coronal, and axial section features are grouped into the first modality section feature group. For positron emission tomography (PET) modality, the corresponding sagittal, coronal, and axial section features are grouped into the second modality section feature group. Each section feature is a high-dimensional feature vector enhanced by a decoupled bidirectional state-space processing module. S42. Project and flatten the three cross-sectional features within the same modality into feature vectors of uniform dimension. Stack the three feature vectors of the first modality cross-sectional feature group or the three feature vectors of the second modality cross-sectional feature group in a preset order to form a view feature sequence. The dimension of the view feature sequence is represented as follows: Where Batch represents the batch size; Indicates the number of orthogonal tangents and ; Indicates the feature dimension after projection; S43. Input the viewpoint feature sequence into the cross-section interaction module. The cross-section interaction module is constructed using a multi-head self-attention mechanism, which concatenates the output features of multiple attention heads along the channel dimension and performs a linear transformation to obtain the section fusion feature. The section fusion feature integrates spatial structure information and metabolic information from different orthogonal section viewpoints within the same modality, with an output dimension of... The cross-section fusion features are used as input for step S5.

[0059] Specifically, the cross-face interaction module includes a query linear projection layer, a key linear projection layer, a value linear projection layer, a scaled dot product attention calculation layer, and an output linear projection layer: S431. Map the view feature sequence into a query matrix by using the query linear projection layer, key linear projection layer, and value linear projection layer, respectively. Key matrix Sum matrix The mathematical expression for the mapping process is: ; in, This represents the input viewpoint feature sequence, with dimension . ; These represent the linear projection weight matrices for query, key, and value, respectively, each with dimension 1. ; The dimension representing each attention head; These represent the mapped query matrix, key matrix, and value matrix, respectively, all with dimension 1. ; S432. The attention weight distribution between different orthogonal facets is calculated through a scaled dot product attention layer, and the value matrix is ​​weighted and aggregated based on the attention weight distribution to obtain the feature representation after facet interaction. The mathematical expression for scaled dot product attention is: ; in, Represents the transpose of the key matrix; Indicates the scaling dimension of the key vector; The scaling factor is used to prevent the gradient from vanishing due to excessively large dot product results; Softmax represents the normalized exponential function used to convert the attention score into a probability distribution; S433. The output features of multiple attention heads are concatenated along the channel dimension and linearly transformed through an output linear projection layer to obtain the cross-sectional fusion feature. The cross-sectional fusion feature integrates spatial structure and metabolic information from different orthogonal cross-sectional perspectives within the same modality, with an output dimension of... The cross-section fusion features are used as input for step S5.

[0060] Furthermore, the method for calculating the attention weight distribution between different orthogonal tangents by scaling the dot product attention calculation layer in step S432 includes: S4321, query matrix Bond matrix Performing matrix multiplication yields the unscaled attention score matrix. The mathematical expression for matrix multiplication is: ; in, This represents the unscaled attention score matrix with dimension . Each element in the matrix This represents the original similarity score between the i-th and j-th facets; S4322, For the unscaled attention score matrix Perform scaling by dividing each element by the scaling factor. The scaled attention score matrix is ​​obtained. The mathematical expression for scaling is: ; Among them, scaling ensures that the numerical range of attention scores remains within a reasonable range, avoiding the problem of increased variance in dot product results due to increased dimensionality; S4323, Scaled attention score matrix Normalization is performed using a normalized exponential function to convert the attention scores of each slice to all other slices into a probability distribution, resulting in the attention weight matrix. The mathematical expression for the normalized exponential function is: ; in, Represents the scaled attention score matrix The element in the i-th row and j-th column; exp represents the exponential function. This means that for the i-th tangent, all Sum the exponential fractions of each cross section; Let the attention weight of the i-th face to the j-th face satisfy the following condition: Attention weight matrix The dimension is ; S4324, Adjust the attention weight matrix AND-value matrix Perform matrix multiplication to obtain the weighted aggregated aspect interaction features. The mathematical expression for matrix multiplication is: ; in, This represents the attention weight matrix, with dimension 1. ; This represents the aspect interaction features after weighted aggregation, with dimensions of [dimensionality]. This feature indicates that each cross section integrates relevant information from other cross sections, enabling deep interaction and dynamic aggregation of spatial structural and metabolic information from different orthogonal cross section perspectives.

[0061] Furthermore, the bidirectional cross-modal cross-attention fusion module in step S5 includes a first cross-attention path and a second cross-attention path, specifically including: S51. In the first cross-attention pathway, the cross-sectional fusion features of the structural magnetic resonance imaging modality are used as the query vector, and the cross-sectional fusion features of the positron emission tomography modality are used as the key vector and value vector. The extraction of metabolic features guided by structural information is achieved through cross-attention calculation. S52. In the second cross-attention pathway, the cross-sectional fusion features of the positron emission tomography mode are used as the query vector, and the cross-sectional fusion features of the structural magnetic resonance imaging mode are used as the bond vector and value vector. The structural feature localization guided by metabolic information is achieved through cross-attention calculation. S53. Combine the output features of the first cross-attention path and the second cross-attention path to obtain cross-modal enhancement features.

[0062] Specifically, the cross-sectional fusion features of the structural magnetic resonance imaging (SMRI) modality and the cross-sectional fusion features of the positron emission tomography (PET) modality output in step S4 are respectively input into the bidirectional cross-modal cross-attention fusion module. The bidirectional cross-modal cross-attention fusion module includes two parallel feature retrieval paths: a first cross-attention path and a second cross-attention path. The first cross-attention path is used to extract features of the PET modality guided by the SMRI modality, and the second cross-attention path is used to locate features of the SMRI modality guided by the PET modality. In the first cross-attention path, the cross-sectional fusion features of the SMRI modality are used as the query vector. The cross-sectional fusion features of positron emission tomography modes are used as key vectors. Sum value vector Metabolic feature extraction guided by structural information is achieved through multi-head cross-attention computation. The calculation formula for the first cross-attention pathway is as follows: ; in, The query vector representing the structural magnetic resonance imaging modality has a dimension of . ; The bond vector representing the positron emission tomography (PET) mode has a dimension of . ; This represents the transpose of the key vector, with dimension 1. ; The value vector representing the positron emission tomography (PET) mode has dimensions of 1. ; Represents the scaling dimension of the key vector and h represents the number of attention heads; Softmax represents the normalized exponential function; This represents the modal characteristics of positron emission tomography (PET) enhanced by structural magnetic resonance imaging (MRI), with dimensions of [dimension not specified]. .

[0063] In the second cross-attention pathway, the cross-sectional fusion features of the positron emission tomography (PET) mode are used as the query vector. The cross-sectional fusion features of structural magnetic resonance imaging modes are used as key vectors. Sum value vector Metabolic information-guided structural feature localization is achieved through multi-head cross-attention computation. The calculation formula for the second cross-attention pathway is as follows: ; in, The query vector represents the positron emission tomography (PET) mode, with dimension 1. ; The key vector representing the structural magnetic resonance imaging mode has a dimension of . ; This represents the transpose of the key vector, with dimension 1. ; The value vector representing the structural magnetic resonance imaging mode has dimension 1. ; Indicates the scaling dimension of the key vector; This represents the structural magnetic resonance imaging modal features enhanced by positron emission tomography (PET), with dimensions of [dimension missing]. .

[0064] The output of the first cross-attention pathway Output of the second cross attention pathway By combining these features, cross-modal enhancement features are obtained. These features include structural and functional features obtained through bidirectional mutual retrieval, which are used to characterize the deep complementary information between structural magnetic resonance imaging mode and positron emission tomography mode.

[0065] like Figure 4 As shown, the methods for constructing the first and second cross-attention pathways using a multi-head attention mechanism include: The input query vector, key vector, and value vector are split into multiple parts, and the dimension is... The original vector is projected onto h attention heads, each attention head processing a dimension of... For the i-th attention head, the input features are projected into a query vector using independent linear projection matrices. Key vector Sum value vector The mathematical expression for linear projection is: ; in, Represents the cross-sectional fusion features of structural magnetic resonance imaging modes; The cross-sectional fusion characteristics representing positron emission tomography (PET) modes; Let represent the query, key, and value linear projection weight matrices of the i-th attention head, respectively, with each matrix having dimension 1. ; Let them represent the query, key, and value vectors after the projection of the i-th attention head, respectively, with each dimension being [dimension 1]. ; Calculate scaled dot product attention within each attention head to obtain the output feature of the i-th attention head. The mathematical expression for scaled dot product attention is: ; in, This represents the query vector for the i-th attention head; This represents the transpose of the key vector of the i-th attention head; This represents the value vector of the i-th attention head; The output feature of the i-th attention head has a dimension of . ; The output features of h attention heads are concatenated along the channel dimension to obtain the concatenated multi-head attention output. The final attention output is obtained by performing a fusion transformation through the output linear projection layer. The mathematical expressions for splicing and projection are: ; ; in, This indicates a channel-level concatenation operation; This represents the concatenated features, with dimensions of [dimensionality]. ; This represents the weight matrix of the output linear projection layer, with dimension 1. ; This represents the final output feature of the first cross-attention pathway. The second cross-attention pathway is computed using the same multi-head attention mechanism structure.

[0066] The first and second cross-attention pathways constructed in this invention respectively realize the clinical diagnostic logic of "precisely focusing on high- or low-metabolic regions using structurally atrophied areas" and "inferring structural changes in specific brain regions from metabolic hotspots." This bidirectional query-key-value mechanism highly replicates the clinical thinking of medical experts who "cross-verify structure and metabolism" when reviewing images. Compared to traditional feature splicing or unidirectional attention mechanisms, it can capture the early Alzheimer's disease patients and their pathological characteristics, significantly improving the model's recall sensitivity to positive case features.

[0067] Finally, step S6 includes: constructing a content-aware dynamic gating network for each modality; concatenating the cross-modal fusion features and cross-modal enhancement features corresponding to the modality along the channel dimension and inputting them into the gating network, whereby the gating network extracts joint feature content and outputs an adaptive gating weight vector; based on the gating weight vector, performing weighted fusion on the cross-modal fusion features and cross-modal enhancement features to obtain the final fusion features for each modality; and finally concatenating the final fusion features of the two modalities and inputting them into the classifier network to output the predicted probabilities of each category and determine the final classification prediction result.

[0068] Specifically, step S6 includes: S61. The cross-sectional fusion features of the structural magnetic resonance imaging mode and the cross-sectional fusion features of the positron emission tomography mode output in step S4 are respectively used as the original aggregated features. and The cross-modal enhancement features of the structural magnetic resonance imaging mode and the cross-modal enhancement features of the positron emission tomography mode output in step S5 are respectively used as cross-modal enhancement features. and For each modality, a content-aware dynamic gating network is constructed separately; S62. For structural magnetic resonance imaging modalities, the original aggregated features will be... With cross-modal enhancement features By stitching the data along the channel dimension, a joint feature representation of the structural magnetic resonance imaging modality is formed. The joint feature representation is input into a content-aware dynamic gating network, which consists of a first fully connected layer, a modified linear unit activation function, a second fully connected layer, and a normalized exponential function layer. The joint feature content is extracted through the first and second fully connected layers, and a gating weight vector is output. The expression for calculating the gating weight vector is as follows: ; in, This represents the joint feature after the channel dimension concatenation operation; This represents the weight matrix of the first fully connected layer; This represents the bias vector of the first fully connected layer; This indicates a modified linear unit activation function; This represents the weight matrix of the second fully connected layer; Represents the bias vector of the second fully connected layer; Softmax represents the normalized exponential function; This represents the output gate weight vector; Dynamic weights assigned to the original aggregated features; For the dynamic weights assigned to cross-modal enhancement features, and satisfying ; S63. Based on the gated weight vector, the original aggregated features and cross-modal enhancement features are adaptively weighted and fused to obtain the final fused features of the structural magnetic resonance imaging modality. The mathematical expression for adaptive weighted fusion is: ; in, This represents the element-wise multiplication operation; and Each feature is expanded to the same dimension as its characteristics via a broadcast mechanism. This represents the final fusion characteristics of structural magnetic resonance imaging modalities; S64. Using the same content-aware dynamic gating network structure to process the positron emission tomography (PET) modes, the final fused features of the PET modes are obtained. The final fusion features of the two modalities are concatenated along the channel dimension and input into a multilayer perceptron classifier consisting of multiple fully connected layers. The predicted probabilities of each category are output through a normalized exponential function, and the category with the highest probability is taken as the Alzheimer's disease classification prediction result.

[0069] Furthermore, the predicted probabilities for each category are output through the normalized exponential function, including: The final fusion features of structural magnetic resonance imaging modalities Final fusion features of positron emission tomography (PET) modes The features are concatenated along the channel dimension to obtain the comprehensive feature vector. The dimension of the comprehensive feature vector is ,in The dimension representing the final fused features of a single modality; Combined feature vectors The input is a multilayer perceptron classifier, which consists of an input layer, hidden layers, and an output layer. The hidden layers use a modified linear unit activation function for nonlinear transformation. The number of neurons in the output layer is equal to the number of classification categories, C. A linear transformation maps the input features to the original predicted scores for each category. The mathematical expression for the linear transformation is: ; in, This represents the comprehensive feature vector of the input; This represents the weight matrix of the output layer, with dimension 1. ; The bias vector of the output layer is [C]; z represents the original predicted score vector for each category, with dimension [C]. ; This represents the original predicted score for the c-th category; For the original predicted score vector By applying the normalized exponential function, the original predicted scores are transformed into a probability distribution form, yielding the predicted probabilities for each category. The mathematical expression for the normalized exponential function is: ; in, represents the original predicted score for the c-th category; exp represents the exponential function; This represents the original predicted score for the j-th category. ; This represents the predicted probability that input sample x belongs to the c-th class, satisfying... ; Based on the predicted probabilities of each category, the final Alzheimer's disease classification prediction result is determined using the maximum probability decision criterion. The mathematical expression of the maximum probability decision criterion is: ; in, The parameter represents the parameter that makes the function reach its maximum value; c represents the category index; {1,2,…,C} represents the set of all possible categories; This indicates the final Alzheimer's disease classification prediction results, with classification categories including normal cognition, mild cognitive impairment, and Alzheimer's disease.

[0070] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A multimodal medical image classification method for Alzheimer's disease based on spatial perception decoupling, characterized in that, include: S1. Obtain structural magnetic resonance imaging modal medical images and positron emission tomography modal medical images of the subject, and extract several orthogonal section slices from the two modal medical images respectively; S2. Input each orthogonal slice into the image feature extraction backbone module to extract high-dimensional spatial feature sequences; S3. Input the high-dimensional spatial feature sequence into the decoupled bidirectional state space processing module. After injecting two-dimensional spatial location prior information into the high-dimensional spatial feature sequence, independently extract the forward long-range dependency information and backward sequence long-range dependency information corresponding to each orthogonal sectional slice. S4. Input the features containing long-range dependency information of forward and backward sequences into the cross-section interaction module. Within the same modality, integrate the spatial structure information and metabolic information from different orthogonal cross-section perspectives, and output the cross-section fusion features of the corresponding modality. S5. Input the cross-modal fusion features of each modality into the bidirectional cross-modal attention fusion module, perform bidirectional mutual retrieval and feature enhancement on the cross-modal fusion features of the two modalities, and output cross-modal enhanced features; S6. Input the cross-sectional fusion features from step S4 and the cross-modal enhancement features from step S5 into the dynamic gating and classification module, adaptively allocate the fusion weights of the cross-sectional fusion features and the cross-modal enhancement features, and output the Alzheimer's disease classification prediction results based on the final fusion features after weight allocation.

2. The method according to claim 1, characterized in that, Step S1 includes: S11. After converting the acquired original structural magnetic resonance imaging images and positron emission tomography images into different formats, spatial positioning and registration are performed based on the previous joint. S12. Perform tissue segmentation, image standardization, and cranial removal on the registered image; S13. Extract slices from each modality of 3D image at equal intervals along the sagittal, coronal and axial planes, and select representative orthogonal slices from each direction based on the information entropy maximization criterion.

3. The method according to claim 1, characterized in that, The image feature extraction backbone module in step S2 adopts an improved deep residual network architecture that includes residual block groups. The specific improvements include: canceling the downsampling operation of the terminal residual block group, and explicitly setting the stride parameters of the core convolutional layer and the downsampling layer in the residual bypass connection within the terminal residual block group to 1, so that the orthogonal slices output by the terminal residual block group maintain a high-resolution spatial feature map and are flattened along the spatial dimension into a one-dimensional high-dimensional spatial feature sequence.

4. The method according to claim 1, characterized in that, The decoupled bidirectional state-space processing module in step S3 includes a spatial position encoding unit, an independent forward processing unit, an independent backward processing unit, and a nonlinear fusion unit. Step S3 specifically includes: S31. The high-dimensional spatial feature sequence is fused with the generated two-dimensional spatial location coding matrix to obtain a location-aware feature sequence injected with topological prior information. S32. Input the position-aware feature sequence into the independent forward processing unit and the independent backward processing unit respectively; The independent forward processing unit performs forward causal scanning modeling on the position-aware feature sequence through the pre-built first state space model to obtain the long-range dependency information of the forward sequence; The independent backward processing unit flips the position-aware feature sequence in the time dimension, performs reverse causal scanning modeling through the pre-built second state space model, and flips it again to restore alignment to obtain the long-range dependency information of the backward sequence. S33. Input the long-range dependency information of the forward and backward sequences into the nonlinear fusion unit, splice them in the channel dimension, and perform fusion and dimensionality reduction through layer normalization, linear mapping and nonlinear activation processing in sequence. S34. Perform residual connection between the fused and dimensionality-reduced features and the input position-aware feature sequence to obtain the final output of the decoupled bidirectional state-space processing module.

5. The method according to claim 2, characterized in that, The cross-section interaction module in step S4 is constructed using a self-attention mechanism. Specifically, it includes: stacking the sagittal, coronal, and axial section features processed by the decoupled bidirectional state space processing module within the same modality into a view feature sequence in a preset order; the cross-section interaction module linearly maps the view feature sequence into a query matrix, a key matrix, and a value matrix, respectively; obtaining the attention weight distribution between each orthogonal section by calculating the similarity distribution between the query matrix and the key matrix; and performing weighted aggregation and multi-head feature concatenation on the value matrix based on the attention weight distribution, finally outputting a section fusion feature that integrates view information from different orthogonal sections.

6. The method according to claim 1, characterized in that, The bidirectional cross-modal cross-attention fusion module in step S5 includes a first cross-attention path and a second cross-attention path, specifically including: S51. In the first cross-attention pathway, the cross-sectional fusion features of the structural magnetic resonance imaging modality are used as the query vector, and the cross-sectional fusion features of the positron emission tomography modality are used as the key vector and value vector. The extraction of metabolic features guided by structural information is achieved through cross-attention calculation. S52. In the second cross-attention pathway, the cross-sectional fusion features of the positron emission tomography mode are used as the query vector, and the cross-sectional fusion features of the structural magnetic resonance imaging mode are used as the bond vector and value vector. The structural feature localization guided by metabolic information is achieved through cross-attention calculation. S53. Combine the output features of the first cross-attention path and the second cross-attention path to obtain cross-modal enhancement features.

7. The method according to claim 1, characterized in that, Step S6 includes: constructing a content-aware dynamic gating network for each modality; concatenating the cross-modal fusion features and cross-modal enhancement features corresponding to the modality along the channel dimension and inputting them into the gating network, whereby the gating network extracts joint feature content and outputs an adaptive gating weight vector; based on the gating weight vector, performing weighted fusion on the cross-modal fusion features and cross-modal enhancement features to obtain the final fusion features for each modality; finally, concatenating the final fusion features of the two modalities and inputting them into the classifier network to output the predicted probabilities of each category and determine the final classification prediction result.

Images