Retinal multi-modal image classification method and system based on spatial perception and modal routing

Abstract

The application discloses a retinal multi-modal image classification method and system based on spatial perception and modal routing, comprising: obtaining registered fundus 2D OCT and OCTA images, and extracting modal features through a double-flow network; generating a spatial heat map weighted feature by using macular anatomical prior, focusing on lesions and weakening registration error interference; extracting structural and blood vessel dominant features through asymmetric cross attention, and adaptively fusing multi-modal features through a gating mechanism; adopting a hierarchical uncertainty classification head to first distinguish normal / abnormal, combining evidence deep learning to quantitatively predict the uncertainty of abnormal samples, and completing four classifications of AMD, DR and other eye diseases. The application can adapt to the pathological characteristics of AMD dependent structure and DR dependent blood vessel image, adaptively adjust the modal weight, and improve the recognition accuracy of the two diseases; through uncertainty quantification, insufficient evidence samples are classified into other eye diseases, which greatly reduces the clinical misdiagnosis, and can be integrated into an ophthalmic device and a diagnosis platform to assist intelligent screening of fundus diseases.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, specifically to a retinal multimodal image classification method and system based on spatial perception and modal routing. Background Technology

[0002] Eye diseases are a leading cause of vision impairment and blindness worldwide, with age-related macular degeneration (AMD) and diabetic retinopathy (DR) being the leading causes of blindness in both developed and developing countries. In clinical ophthalmological diagnosis, optical coherence tomography (OCT) and optical coherence tomography angiography (OCTA) have become indispensable non-invasive imaging techniques. 2D OCT images can present high-resolution cross-sectional morphological changes in retinal tissue (such as retinal edema and drusen); while 2D OCTA images can non-invasively reflect the hemodynamic characteristics of retinal microvessels (such as microaneurysms and capillary non-perfusion areas). Accurately classifying fundus images into four categories—AMD, DR, other fundus diseases (such as atypical diseases like glaucoma and retinal vein occlusion), and normal—is of paramount clinical significance for early screening, differential diagnosis, and tiered management of these diseases.

[0003] In recent years, deep learning-based computer vision technology has made significant progress in medical image analysis. To fully utilize the complementary information between structure and blood vessels, existing advanced solutions often employ dual-stream multimodal neural networks, simultaneously inputting OCT and OCT images for joint classification. However, in complex clinical scenarios, existing multimodal retinal image classification techniques still suffer from the following significant technical limitations: First, existing multimodal fusion mechanisms do not consider the modality specificity of disease pathology, resulting in limited feature expression. AMD and DR show significant differences in their dependence on different imaging modalities in terms of pathological mechanisms: the core pathological features of AMD (such as retinal pigment epithelial detachment and drusen) are more intuitively represented on OCT structural images; while the early core pathology of DR (such as microvascular leakage and enlargement of the avascular area in the fovea of ​​the macular region) is more sensitive on OCTA vascular images. However, existing multimodal classification models usually use symmetrical feature concatenation or global cross-attention mechanisms for feature fusion. This "one-size-fits-all" symmetrical fusion approach fails to dynamically adjust modality weights according to disease specificity, easily leading to noise from weak modalities interfering with the effective features of strong modalities, thereby reducing the model's accuracy in distinguishing between AMD and DR.

[0004] Second, existing feature extraction methods lack spatial focusing capabilities for key lesion regions and are highly sensitive to registration errors. Key lesions in retinal diseases (such as macular edema and neovascularization) are highly concentrated in the fovea and surrounding areas. Existing deep learning models typically perform global convolution or global token pooling on 2D OCT and OCTA images. This approach not only introduces background noise from irrelevant areas in the peripheral retina, diluting the features of core lesions, but also amplifies registration errors between modalities. In actual acquisition, minute eye tremors can cause slight spatial misalignment between 2D OCT slices and 2D OCTA projection images. Existing pixel-level or simple feature-level alignment methods are prone to failure when faced with such non-rigid registration errors, leading to feature mismatch during cross-modal attention calculations and severely affecting the robustness of classification.

[0005] Third, existing classification networks struggle to effectively handle the ambiguous boundaries of the open set category "Others." In real-world clinical screening, besides typical AMD and DR, a large number of other atypical or rare fundus diseases (collectively referred to as the OTHERS category) are encountered. Existing classification models typically employ a single-layer Softmax classifier head, forcibly normalizing the probability distribution of input samples to four predefined categories. This closed-set classification paradigm lacks the ability to quantify the "uncertainty" of model predictions. When the input image belongs to the OTHERS category or is of poor quality, the model often makes "blindly confident" erroneous predictions, forcibly classifying it as AMD or DR, leading to extremely high false positive rates and the risk of clinical misdiagnosis.

[0006] Therefore, there is an urgent need for a retinal multimodal image classification method that can adaptively perform asymmetric modal fusion based on disease specificity, overcome registration errors and focus on core lesion areas, and has the ability to quantify uncertainty to accurately define the boundaries of atypical diseases. Summary of the Invention

[0007] Purpose of the Invention: In order to overcome the above shortcomings, the purpose of this application is to provide a retinal multimodal image classification method and system based on spatial perception and modal routing, which can solve the technical problems of insufficient modal fusion, poor resistance to registration error, and easy misdiagnosis of atypical diseases (OTHERS) in the joint analysis of 2D OCT and 2D OCTA images by existing deep learning models.

[0008] To address the aforementioned technical problems, this application provides a retinal multimodal image classification method based on spatial awareness and modal routing, including: S1: Acquire 2D OCT and 2D OCTA images corresponding to the fundus region of the same subject and perform image quality verification, visual field cropping, size uniform resampling, grayscale standardization and two-dimensional spatial registration processing to construct a multimodal retinal image dataset with a unified spatial coordinate system; S2: Input the 2D OCT image and 2D OCTA image into the dual-stream feature extraction network respectively, extract the structural feature representation through the structural feature encoding branch and extract the vascular feature representation through the vascular feature encoding branch to generate the corresponding OCT structural feature map and OCTA vascular feature map; S3: Based on the OCT structural feature map and OCTA vascular feature map, a spatial saliency heatmap is generated using the anatomical and physical prior of the fovea of ​​the retina. The OCT structural feature map and OCTA vascular feature map are then spatially weighted according to the spatial saliency heatmap to generate an OCT enhancement feature map and an OCTA enhancement feature map focusing on the macular region. S4: Based on the OCT enhanced feature map and the OCT enhanced feature map, an asymmetric cross-attention mechanism is used to construct a structure-dominant feature extraction path and a blood vessel-dominant feature extraction path to obtain structure-dominant features and blood vessel-dominant features. According to the preset modal routing gating weight, the structure-dominant features and blood vessel-dominant features are dynamically aggregated to generate multimodal fusion features. S5: Input the multimodal fusion features into the hierarchical uncertainty classification network to perform binary classification of normal samples and abnormal samples. When the judgment result is a normal sample, the normal classification result is directly output. When the judgment result is an abnormal sample, the classification uncertainty score is calculated by combining evidence deep learning to perform tri-class classification of age-related macular degeneration, diabetic retinopathy and other fundus diseases, so as to output a four-class classification result including NORMAL, AMD, DR and OTHERS.

[0009] As a preferred embodiment of this application, in S1, the method includes the following steps: S11: 2D OCT and 2DOCTA images of the fundus region of the same subject were acquired using optical coherence tomography (OCT) equipment; S12: Perform image quality detection on the acquired 2D OCT images and 2D OCTA images, remove abnormal images with severe motion artifacts, image blur, local occlusion and field of view loss, and retain image samples that meet the preset quality requirements. S13: Perform background region cropping on the 2D OCT image and 2D OCTA image after quality inspection, remove invalid black areas at the image edges and retain the effective retinal area including the fovea region of the macula; S14: The cropped 2D OCT image and 2D OCTA image are uniformly resampled to a preset input size and the image grayscale distribution is normalized using the Z-score normalization method to reduce the image intensity differences caused by different devices, different subjects and different acquisition conditions. S15: Perform data augmentation processing on the standardized image samples. The data augmentation processing includes at least random horizontal flipping, random angle rotation, and random brightness and contrast adjustment to improve the model's generalization ability to retinal image morphological changes. S16: Using the registration information built into the device or an image registration method based on mutual information criteria, perform two-dimensional spatial registration processing on the 2D OCT image and the 2D OCTA image to achieve a spatial correspondence between the macular region and retinal anatomy in the two modal images, so as to construct a multimodal retinal image dataset with a unified spatial coordinate system.

[0010] As a preferred embodiment of this application, in S2, the method includes the following steps: S21: Construct a two-stream feature extraction network including a first encoder and a second encoder; S22: Input the registered 2D OCT image into the first encoder to extract structural features that characterize the retinal layer structure, macular morphological changes and abnormal information of lesion tissue, and input the registered 2D OCTA image into the second encoder to extract vascular features that characterize the capillary distribution status, blood perfusion status and abnormal changes of microvessels. S23: Extract multi-scale feature maps from multiple layers of different depths of the first encoder and the second encoder respectively, and use the feature pyramid fusion mechanism to fuse shallow texture information, mid-level structural information and deep semantic information. S24: Through unified channel mapping and unified spatial scale processing, multi-scale features at different levels are mapped to a unified feature space, and learnable fusion weights are used to weight and aggregate the features at each scale to generate an OCT structure feature map with preset channel dimensions and spatial size. And OCTA vascular feature maps: Where H is the feature map height and W is the feature map width. For the feature map channel dimension.

[0011] As a preferred embodiment of this application, in S3, the method includes the following steps: S31: Input the OCT structural feature map into a lightweight convolution branch, and extract the physical curvature features that reflect the curvature state between retinal layers and the fovea morphology through convolution operations. S32: Generate an initial response map based on the physical bending morphological features, and perform Sigmoid function activation and spatial smoothing on the initial response map to generate a spatial saliency heatmap with values ​​in the range [0,1]. ,in It is a lightweight convolution mapping function. It is the Sigmoid activation function. Spatial saliency heatmap; S33: Perform spatial smoothing on the spatial saliency heatmap to create a continuous high-response region in the fovea region of the macula and maintain a low response value in the peripheral region; S34: Using the spatial saliency heatmap, the OCT structural feature map and the OCTA vascular feature map are multiplied element-wise to generate an OCT enhancement feature map and an OCTA enhancement map focusing on the macular lesion region: , Where ⊙ represents the element-wise multiplication operation under the broadcast mechanism. For OCT enhanced feature maps, Enhance the feature map for OCTA.

[0012] As a preferred embodiment of this application, in step S4, the method includes the following steps: S41: Using the OCT enhanced feature map as the structural modality input and the OCTA enhanced feature map as the vascular modality input, construct the structural-dominated cross-attention branch and the vascular-dominated cross-attention branch respectively; S42: In the structure-dominant cross-attention branch, using the OCT enhanced feature map as the query feature and the OCT enhanced feature map as the key and value features, a first query vector, a first key vector, and a first value vector are generated through linear mapping: , , ,in A learnable linear projection matrix for the structure-dominant cross-attention branch; S43: Calculate cross-modal attention weights using the first query vector, the first key vector, and the first value vector, and generate dominant structural features that reflect structural pathological characteristics. ,in The dominant structural feature Let be the dimension of the key vector. This is the transpose of the first key vector; S44: In the blood vessel-dominated cross-attention branch, using the OCTA enhanced feature map as the query feature and the OCT enhanced feature map as the key and value features, a second query vector, a second key vector, and a second value vector are generated through linear mapping: , , ,in A learnable linear projection matrix for blood vessel-dominated cross-attention branches; S45: Calculate cross-modal attention weights using the second query vector, the second key vector, and the second value vector, and generate vascular dominant features reflecting vascular pathology characteristics: ,in The dominant feature is the blood vessel. This is the transpose of the second key vector.

[0013] As a preferred embodiment of this application, in step S4, the method includes the following steps: S46: Perform global average pooling on the structural dominant features and the vascular dominant features respectively to obtain the corresponding global semantic representation vectors; S47: Concatenate the global semantic representation vector corresponding to the dominant structural feature and the global semantic representation vector corresponding to the dominant vascular feature in the channel dimension to form a joint feature; S48: Input the joint features into a multilayer perceptron, learn the synergistic relationship between structural pathological features and vascular pathological features through nonlinear mapping, and output scalar gating weights: ,in For the multilayer perceptron mapping function, To be and splicing along the channel dimension; S49: Dynamically weight and fuse the structural dominant features and vascular dominant features according to the gating weights to generate multimodal fusion features: ;in, This represents the final multimodal features after fusion.

[0014] As a preferred embodiment of this application, in step S5, the method includes the following steps: S51: Input the final multimodal fusion feature into the first-level classification network, compress the final multimodal fusion feature through the global average pooling layer, and output the binary classification prediction results corresponding to the normal category and the abnormal category through the first fully connected layer; S52: The samples corresponding to the abnormal probability prediction values ​​that are less than the preset value are classified as NORMAL, and the samples corresponding to the abnormal probability prediction values ​​that are greater than or equal to the abnormal probability prediction values ​​are input into the second-level classification network. S53: Utilize the evidence generation layer to output non-negative evidence values ​​corresponding to the AMD, DR, and OTHERS categories respectively, and construct the total amount of evidence required for disease category determination based on the evidence values: ,in The original evidence value output by the network. S represents the total number of categories in the second-level classification, and S represents the total amount of evidence collected by the model. For the concentration parameters of the Dirichlet distribution; S54: Calculate the prediction probability and overall uncertainty score for each category using subjective logic theory based on the total amount of evidence. ;in, To predict probabilities, The overall uncertainty score; S55: Based on the relationship between the cognitive uncertainty score and the preset safety threshold, make open set disease identification and classification decisions.

[0015] As a preferred embodiment of this application, in S5, a preset safety threshold is set. If the uncertainty score is less than the preset safety threshold, the AMD or DR classification result with the highest probability in the total evidence is output. If the uncertainty score is higher than or equal to the preset safety threshold, the model evidence is deemed insufficient, and the result is forcibly output as OTHERS.

[0016] As a preferred embodiment of this application, in step S5, the training method of the hierarchical uncertainty classification network includes the following steps: S101: Construct a binary classification loss term to constrain the determination results of normal and abnormal categories: ;in This represents the total number of samples in the current training batch. Let be the true label of the i-th sample in the first-level binary classification; Let be the predicted probability that the i-th sample, output by the first-level binary classification network, belongs to the anomaly category; S102: Construct an evidence-based classification loss term to constrain the classification results for the AMD, DR, and OTHERS categories: ;in One-hot encoding of the real label. It is the Digamma function; S103: Construct a concave prior regularization loss term based on the spatial saliency heatmap, forcing the center response to be higher than the edge response: ;in, and These are the sets of pixel coordinates for the image center and edges, respectively. This represents the total number of pixels in the set. The value of the spatial saliency heatmap at coordinates (x, y); S104: Construct an uncertainty penalty loss term to constrain the prediction results with high uncertainty; S105: The binary classification loss term, evidence classification loss term, concave prior regularization loss term, and uncertainty penalty loss term are weighted and combined to construct a joint loss function: ;in, To control the weights of the space regularization term, To control the weight of the uncertainty penalty term, For uncertainty penalty loss item; S106: The hierarchical uncertainty classification network is uniformly optimized using the joint loss function and the network parameters are updated using the backpropagation algorithm until the validation set loss converges.

[0017] This application also provides a retinal multimodal image classification system based on spatial awareness and modal routing that implements the method described above, comprising: The image acquisition module is used to acquire 2D OCT images and 2D OCTA images corresponding to the fundus region of the same subject and perform image quality verification, visual field cropping, size uniform resampling, grayscale standardization and two-dimensional spatial registration processing to construct a multimodal retinal image dataset with a unified spatial coordinate system. The feature extraction module is used to input the 2D OCT image and the 2D OCTA image into the dual-stream feature extraction network, extract structural feature representations through the structural feature encoding branch and extract vascular feature representations through the vascular feature encoding branch, so as to generate the corresponding OCT structural feature map and OCTA vascular feature map. The spatial guidance module generates a spatial saliency heatmap based on the OCT structural feature map and the OCTA vascular feature map, using the anatomical and physical priors of the fovea of ​​the retina, and spatially weights the OCT structural feature map and the OCTA vascular feature map based on the spatial saliency heatmap to generate an OCT enhancement feature map and an OCTA enhancement feature map focusing on the macular region. The modal routing module is used to construct structural dominant feature extraction paths and vascular dominant feature extraction paths respectively based on the OCT enhanced feature map and the OCT enhanced feature map using an asymmetric cross-attention mechanism to obtain structural dominant features and vascular dominant features. Based on the preset modal routing gating weights, the structural dominant features and vascular dominant features are dynamically aggregated to generate multimodal fusion features. The hierarchical classification module is used to input the multimodal fusion features into the hierarchical uncertainty classification network to perform binary classification of normal samples and abnormal samples. When the judgment result is a normal sample, the normal classification result is directly output. When the judgment result is an abnormal sample, the classification uncertainty score is calculated by combining evidence deep learning to perform tri-class classification of age-related macular degeneration, diabetic retinopathy and other fundus diseases, so as to output a four-class classification result including NORMAL, AMD, DR and OTHERS.

[0018] This application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the retinal multimodal image classification method based on spatial perception and modal routing.

[0019] This application also provides a computer-readable storage medium storing computer instructions for causing a computer to execute the above-described retinal multimodal image classification method based on spatial awareness and modal routing.

[0020] The technical solution described in this application has the following advantages over the prior art: 1. This application introduces a foveal perception spatial guidance mechanism, which utilizes the optical imaging characteristics and anatomical priors of the "physical concavity" of the fovea in 2D OCT images to automatically generate a spatial saliency heatmap. Compared with the peripheral noise interference caused by global convolution or global token pooling in the prior art, this application uses the heatmap to spatially weight the bimodal features, forcing the model to accurately focus on the core lesion area (such as macular edema and drusen) around the fovea. At the same time, since irrelevant features in the retinal edge area are effectively shielded, the negative impact of small non-rigid registration errors between 2D OCT and 2D OCTA images caused by micro-movements of the patient's eyeball on cross-modal fusion is greatly alleviated, and the model's sensitivity to the classification of early small lesions is significantly improved.

[0021] 2. This application breaks through the limitations of "symmetric splicing" or "global cross-attention" fusion in traditional multimodal deep learning models. It designs a disease perception modality routing. Targeting the clinical fact that the core pathology of AMD is biased towards changes in tissue structure and the early pathology of DR is biased towards changes in microvascular hemodynamics, it constructs "structure-dominated" and "vascular-dominated" feature branches respectively through an asymmetric cross-attention mechanism and uses dynamic gating weights for adaptive aggregation. This mechanism enables the network to automatically assign optimal weights to OCT structural features and OCT vascular features according to the actual pathological tendency of the input image, effectively avoiding the interference of weak modality noise on the effective features of strong modality, thereby improving the discrimination and classification accuracy of AMD and DR, two diseases that are easily confused.

[0022] 3. Addressing the prevalence of atypical or rare fundus diseases (OTHERS category) in real-world clinical screening, this application abandons the "blind confidence" flaw caused by the forced normalization of probabilities in traditional Softmax classification heads. Instead, it constructs a hierarchical uncertainty classification head. By introducing subjective logic theory and evidence deep learning (EDL), the network output is transformed into Dirichlet distribution parameters, achieving precise quantification of the "cognitive uncertainty" of model predictions. When the input image belongs to the OTHERS category or the image quality is poor and the model lacks sufficient diagnostic evidence, the uncertainty score automatically spikes and triggers a safe routing mechanism, reliably classifying it as an OTHERS. This mechanism defines a safe decision boundary for open-set diseases, reducing the false positive rate of misclassifying rare diseases as typical AMD or DR, and greatly improving the clinical safety and doctor trust of the AI-assisted diagnostic system.

[0023] 4. This application designs a multi-objective joint loss function that includes a foveal prior regularization term and an uncertainty penalty term, ensuring efficient collaboration between the three modules of spatial guidance, modal routing, and uncertainty classification in end-to-end training. This avoids the model becoming locally optimal or "lazy" in the complex loss space (such as over-outputting high uncertainty). In addition, the network structure of this application naturally supports clinical interpretability analysis. Doctors can intuitively confirm whether the model has focused on the correct macular lesion area through a visualized spatial saliency heatmap and understand whether the model relies more on structural changes or vascular changes to make a diagnosis by analyzing the modal routing gating weights. This meets the strict regulatory requirements of Medical Device Software (SaMD) for the transparency and interpretability of AI algorithms. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1 This is a schematic flowchart of an exemplary retinal multimodal image classification method provided in an embodiment of this application.

[0026] Figure 2 This is a schematic diagram of an exemplary standardized processing flow for fundus images provided in an embodiment of this application.

[0027] Figure 3 This is a flowchart illustrating an exemplary method for constructing a multimodal retinal image dataset provided in this application embodiment.

[0028] Figure 4 This is a schematic flowchart illustrating an exemplary OCT structure and an OCT vascular feature map generation method provided in this application embodiment.

[0029] Figure 5 This is an exemplary schematic diagram of a space guidance module structure provided in an embodiment of this application.

[0030] Figure 6 This is a schematic flowchart of an exemplary OCT and OCTTA enhanced feature map generation method provided in an embodiment of this application.

[0031] Figure 7 This is an exemplary schematic diagram of the adaptive fusion principle of a modal routing module provided in an embodiment of this application.

[0032] Figure 8This is a schematic flowchart of an exemplary method for generating blood vessel dominant features provided in an embodiment of this application.

[0033] Figure 9 This is a flowchart illustrating an exemplary multimodal fusion feature generation method provided in an embodiment of this application.

[0034] Figure 10 This is a flowchart illustrating an exemplary lesion classification method provided in an embodiment of this application.

[0035] Figure 11 This is an exemplary uncertainty classification header EDL inference principle diagram provided in the embodiments of this application.

[0036] Figure 12 This is a flowchart illustrating an exemplary training method provided in an embodiment of this application.

[0037] Figure 13 This is a schematic diagram of the structure of an exemplary retinal multimodal image classification system provided in an embodiment of this application.

[0038] Figure 14 This is a schematic diagram of the structure of an exemplary electronic device provided in an embodiment of this application. Detailed Implementation

[0039] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0040] This application aims to achieve the following three technical objectives: Firstly, this application aims to achieve disease-specific asymmetric modality fusion, thereby improving the identification accuracy of typical diseases. It seeks to overcome the limitations of traditional symmetric fusion in multimodal models by constructing a disease-aware modality routing unit. This allows the network to adaptively assign optimal cross-attention weights to OCT structural features (for AMD) and OCT vascular features (for DR) based on the pathological features of the input image. This fully leverages the dependence of different diseases on specific modalities, significantly improving the classification accuracy of AMD and DR.

[0041] Secondly, it achieves spatial focusing on the core lesion region, enhancing the model's robustness to registration errors. This application aims to overcome the noise interference and registration sensitivity issues caused by global feature extraction by introducing a foveal-sensing spatial guidance module, which utilizes the anatomical and physical priors of the macular region of the retina to generate a spatially saliency heatmap. This forces the model to focus on the core lesion region of the fovea and its surrounding area, effectively shielding away irrelevant peripheral noise and significantly reducing the negative impact of minor registration errors between 2D OCT and OCTA images caused by micro-movements of the patient's eyeball on the classification results.

[0042] Third, this application aims to effectively quantify classification uncertainty and accurately define the decision boundaries for open-set diseases. It addresses the deficiency of traditional Softmax classification heads in lacking confidence assessment when handling "Others" by designing a hierarchical uncertainty classification head and introducing Evidence Deep Learning (EDL) theory to quantify the cognitive uncertainty of model predictions. When the model lacks sufficient evidence for predicting typical diseases (AMD / DR), it can safely and automatically route them to the OTHERS category, thereby effectively clarifying the classification boundaries of atypical diseases and significantly reducing the false positive rate and misdiagnosis risk in clinical auxiliary diagnosis.

[0043] Specifically, refer to Figure 1 As shown, in some embodiments, the retinal multimodal image classification method based on spatial awareness and modal routing includes the following steps: S1: Acquire 2D OCT and 2D OCTA images corresponding to the fundus region of the same subject and perform image quality verification, visual field cropping, size uniform resampling, grayscale standardization and two-dimensional spatial registration processing to construct a multimodal retinal image dataset with a unified spatial coordinate system.

[0044] The 2D OCT image is a two-dimensional optical coherence tomography (OCT) structural image, and the 2D OCTA image is a two-dimensional optical coherence tomography (OCT) vascular imaging image.

[0045] refer to Figure 2 and Figure 3 As shown. Specifically, in S1, the method includes the following steps: S11: 2D OCT and 2DOCTA images of the fundus region of the same subject were acquired using optical coherence tomography (OCT) equipment; S12: Perform image quality detection on the acquired 2D OCT images and 2D OCTA images, remove abnormal images with severe motion artifacts, image blur, local occlusion and field of view loss, and retain image samples that meet the preset quality requirements. S13: Perform background region cropping on the 2D OCT image and 2D OCTA image after quality inspection, remove invalid black areas at the image edges and retain the effective retinal area including the fovea region of the macula; S14: The cropped 2D OCT image and 2D OCTA image are uniformly resampled to a preset input size and the image grayscale distribution is normalized using the Z-score normalization method to reduce the image intensity differences caused by different devices, different subjects and different acquisition conditions. S15: Perform data augmentation processing on the standardized image samples. The data augmentation processing includes at least random horizontal flipping, random angle rotation, and random brightness and contrast adjustment to improve the model's generalization ability to retinal image morphological changes. S16: Using the registration information built into the device or an image registration method based on mutual information criteria, perform two-dimensional spatial registration processing on the 2D OCT image and the 2D OCTA image to achieve a spatial correspondence between the macular region and retinal anatomy in the two modal images, so as to construct a multimodal retinal image dataset with a unified spatial coordinate system.

[0046] S2: Input the 2D OCT image and 2D OCTA image into the dual-stream feature extraction network respectively, extract the structural feature representation through the structural feature encoding branch and extract the vascular feature representation through the vascular feature encoding branch to generate the corresponding OCT structural feature map and OCTA vascular feature map.

[0047] refer to Figure 4 As shown. Specifically, in S2, the method includes the following steps: S21: Construct a two-stream feature extraction network including a first encoder and a second encoder; S22: Input the registered 2D OCT image into the first encoder to extract structural features that characterize the retinal layer structure, macular morphological changes and abnormal information of lesion tissue, and input the registered 2D OCTA image into the second encoder to extract vascular features that characterize the capillary distribution status, blood perfusion status and abnormal changes of microvessels. S23: Extract multi-scale feature maps from multiple layers of different depths of the first encoder and the second encoder respectively, and use the feature pyramid fusion mechanism to fuse shallow texture information, mid-level structural information and deep semantic information. S24: Through unified channel mapping and unified spatial scale processing, multi-scale features at different levels are mapped to a unified feature space, and learnable fusion weights are used to weight and aggregate the features at each scale to generate an OCT structure feature map with preset channel dimensions and spatial size. And OCTA vascular feature maps: Where H is the feature map height and W is the feature map width. For the feature map channel dimension.

[0048] S3: Based on the OCT structural feature map and OCTA vascular feature map, a spatial saliency heatmap is generated using the anatomical and physical prior of the fovea of ​​the retina. The OCT structural feature map and OCTA vascular feature map are then spatially weighted according to the spatial saliency heatmap to generate an OCT enhancement feature map and an OCTA enhancement feature map focusing on the macular region.

[0049] refer to Figure 5 and Figure 6 As shown. Specifically, in S3, the method includes the following steps: S31: Input the OCT structural feature map into a lightweight convolution branch, and extract the physical curvature features that reflect the curvature state between retinal layers and the fovea morphology through convolution operations. S32: Generate an initial response map based on the physical bending morphological features, and perform Sigmoid function activation and spatial smoothing on the initial response map to generate a spatial saliency heatmap with values ​​in the range [0,1]. ,in It is a lightweight convolution mapping function. This is the Sigmoid activation function, used to restrict the output value to the (0,1) interval; Spatial saliency heatmap; S33: Perform spatial smoothing on the spatial saliency heatmap to create a continuous high-response region in the fovea region of the macula and maintain a low response value in the peripheral region; S34: Using the spatial saliency heatmap, the OCT structural feature map and the OCTA vascular feature map are multiplied element-wise to generate an OCT enhancement feature map and an OCTA enhancement map focusing on the macular lesion region: , Where ⊙ represents the element-wise multiplication operation under the broadcast mechanism. For OCT enhanced feature maps, Enhance the feature map for OCTA.

[0050] S4: Based on the OCT enhanced feature map and the OCT enhanced feature map, an asymmetric cross-attention mechanism is used to construct a structure-dominant feature extraction path and a blood vessel-dominant feature extraction path to obtain structure-dominant features and blood vessel-dominant features. According to the preset modal routing gating weight, the structure-dominant features and blood vessel-dominant features are dynamically aggregated to generate multimodal fusion features.

[0051] Among them, the core pathology of AMD (such as drusen) is more obvious on the OCT structural map, while the core pathology of DR (such as microvascular abnormalities) is more sensitive on the OCTA vascular map. Traditional symmetrical splicing and fusion cannot distinguish this disease-modality dependence. Therefore, this step designed an asymmetric cross-attention mechanism to construct two feature branches, structure-dominant and vascular-dominant, respectively and dynamically aggregate them according to the actual pathological tendency of the image.

[0052] refer to Figure 7 and Figure 8 As shown. Specifically, in S4, the method for obtaining the dominant structural features and the dominant vascular features includes the following steps: S41: Using the OCT enhanced feature map as the structural modality input and the OCTA enhanced feature map as the vascular modality input, construct the structural-dominated cross-attention branch and the vascular-dominated cross-attention branch respectively.

[0053] S42: In the structure-dominant cross-attention branch, using the OCT enhanced feature map as the query feature and the OCT enhanced feature map as the key and value features, a first query vector, a first key vector, and a first value vector are generated through linear mapping: , , ,in It is a learnable linear projection matrix for the structure-dominant cross-attention branch.

[0054] Among them, OCT enhanced features As a query vector (representing the demand for structural features), OCTA is used to enhance features. As a key and a value (representing the context provided by vascular features).

[0055] S43: Calculate cross-modal attention weights using the first query vector, the first key vector, and the first value vector, and generate dominant structural features that reflect structural pathological characteristics. ,in The dominant structural feature is the dimension of the key vector, used to scale the dot product to prevent gradient vanishing; This is the transpose of the first key vector; S44: In the blood vessel-dominated cross-attention branch, using the OCTA enhanced feature map as the query feature and the OCT enhanced feature map as the key and value features, a second query vector, a second key vector, and a second value vector are generated through linear mapping: , , ,in A learnable linear projection matrix for blood vessel-dominated cross-attention branches; S45: Calculate cross-modal attention weights using the second query vector, the second key vector, and the second value vector, and generate vascular dominant features reflecting vascular pathology characteristics: ,in The dominant feature is the blood vessel. This is the transpose of the second key vector.

[0056] refer to Figure 9 As shown. Specifically, in S4, the method for generating multimodal features includes the following steps: S46: Perform global average pooling on the structural dominant features and the vascular dominant features respectively to obtain the corresponding global semantic representation vectors; S47: Concatenate the global semantic representation vector corresponding to the dominant structural feature and the global semantic representation vector corresponding to the dominant vascular feature in the channel dimension to form a joint feature; S48: Input the joint features into a multilayer perceptron, learn the synergistic relationship between structural pathological features and vascular pathological features through nonlinear mapping, and output scalar gating weights: ,in For the multilayer perceptron mapping function, To be and splicing along the channel dimension.

[0057] S49: Dynamically weight and fuse the structural dominant features and vascular dominant features according to the gating weights to generate multimodal fusion features: ;in, This represents the final multimodal features after fusion.

[0058] Among them, in the formula It achieves disease-specific adaptive routing; when the input image exhibits AMD characteristics, the network automatically learns to... Increase, thereby allowing More retention (Structural features); conversely, for DR images, It will decrease, making the model more dependent. (Vascular features), thus completely solving the problem of mutual interference between modal features.

[0059] S5: Input the multimodal fusion features into the hierarchical uncertainty classification network to perform binary classification of normal samples and abnormal samples. When the judgment result is a normal sample, the normal classification result is directly output. When the judgment result is an abnormal sample, the classification uncertainty score is calculated by combining evidence deep learning to perform tri-class classification of age-related macular degeneration, diabetic retinopathy and other fundus diseases, so as to output a four-class classification result including NORMAL, AMD, DR and OTHERS.

[0060] refer to Figure 10 and Figure 11 As shown. Specifically, in S5, the method includes the following steps: S51: Input the final multimodal fusion feature into the first-level classification network, compress the final multimodal fusion feature through the global average pooling layer, and output the binary classification prediction results corresponding to the normal category and the abnormal category through the first fully connected layer; S52: The samples corresponding to the abnormal probability prediction values ​​that are less than the preset values ​​are classified as NORMAL, and the samples corresponding to the abnormal probability prediction values ​​that are greater than or equal to the abnormal probability prediction values ​​are input into the second-level classification network. S53: Utilize the evidence generation layer to output non-negative evidence values ​​corresponding to the AMD, DR, and OTHERS categories respectively, and construct the total amount of evidence required for disease category determination based on the evidence values: ,in The original evidence value output by the network. S represents the total number of categories in the second-level classification, and S represents the total amount of evidence collected by the model. For the concentration parameters of the Dirichlet distribution; S54: Calculate the prediction probability and overall uncertainty score for each category using subjective logic theory based on the total amount of evidence. ;in, To predict probabilities, The overall uncertainty score; S55: Make open set disease identification and classification decisions based on the relationship between the cognitive uncertainty score and the preset safety threshold.

[0061] The system sets a preset safety threshold. If the uncertainty score is less than the preset safety threshold, the system outputs the AMD or DR classification result with the highest probability in the total evidence. If the uncertainty score is higher than or equal to the preset safety threshold, the system determines that the model evidence is insufficient and forces the output of the result as OTHERS.

[0062] In clinical screening, there are a large number of atypical diseases (OTHERS). Traditional Softmax classification heads force the normalization of probabilities, causing the model to "blindly and confidently" misclassify OHERS images as AMD or DR when faced with unseen OHERS images. This step introduces Evidential Deep Learning (EDL) to define a safe decision boundary for OHERS categories by quantifying the "cognitive uncertainty" of the model.

[0063] Specifically, the first-level network first performs a binary classification of Normal / Abnormal; if it is an anomaly, the second-level network does not directly output the probability, but instead outputs non-negative evidence vectors for three categories: AMD (c=1), DR (c=2), and OTHERS (c=3). Based on subjective logic theory, the evidence is transformed into Dirichlet distribution parameters. And the total amount of evidence S; where ≥0 represents the original evidence value of type c output by the network. =3 represents the total number of categories in the second-level classification, and S represents the total amount of evidence collected by the model. The concentration parameter is for the Dirichlet distribution.

[0064] Probability prediction and uncertainty quantification: Calculating the prediction probability of class c based on the expectation of the Dirichlet distribution. And calculate the overall cognitive uncertainty score u; where Let u be the model's predicted probability for class c; u∈(0,1] is the uncertainty score; the formula shows that the smaller the total evidence S (i.e., the more unfamiliar the features seen by the model), the greater the uncertainty u.

[0065] Uncertainty routing decision: if And u< If u ≥ 0, then output the category with the highest probability (AMD or DR); If so, it will be forcibly classified as OTHERS.

[0066] Specifically, The input is a hierarchical uncertainty classification head. The first-level binary classification network consists of a global average pooling layer (GAP) and a fully connected layer, and outputs the anomaly probability. ,like If <0.5, it is directly judged as NORMAL; if If the value is ≥0.5, the system proceeds to the second-level three-class classification network; the second-level network outputs three non-negative evidence values. The uncertainty score u is calculated based on subjective logic theory; in this embodiment, a preset safety threshold is used. Set the value to 0.25; if the calculated u ≥ 0.25, the system determines the model has "insufficient evidence" and forces the output to be OTHERS (other diseases); if u < 0.25, then output... The category with the highest probability (AMD or DR).

[0067] Through formula This application is able to honestly express "I don't know" when inputting rare OTHERS images, due to the lack of typical features, the evidence output by the network is insufficient. The value will be very small, causing S to decrease, which in turn causes the uncertainty u to surge and exceed... This allows them to be safely classified as OTHERS, significantly reducing the clinical misdiagnosis rate.

[0068] refer to Figure 12 As shown. In some embodiments of this application, the training method of the hierarchical uncertainty classification network includes the following steps: S101: Construct a binary classification loss term to constrain the determination results of normal and abnormal categories: ;in This represents the total number of samples in the current training batch. The ground truth is the ground label of the i-th sample in the first-level binary classification. Let be the predicted probability that the i-th sample output by the first-level binary classification network belongs to the abnormal category.

[0069] If the sample is a normal image, then =0; if it is an aberrant image, then =1; This represents the predicted probability that the i-th sample belongs to the anomalous (ABNORMAL) category, as output by the first-level binary classification network of the model (usually after passing through the Sigmoid activation function). ∈(0,1).

[0070] S102: Construct an evidence-based classification loss term to constrain the classification results for the AMD, DR, and OTHERS categories: ;in One-hot encoding of the real label. This is the Digamma function.

[0071] S103: Construct a concave prior regularization loss term based on the spatial saliency heatmap, forcing the center response to be higher than the edge response: ;in, and These are the sets of pixel coordinates for the image center and edges, respectively. This represents the total number of pixels in the set. Let be the value of the spatial saliency heatmap at coordinates (x, y); the formula constrains spatial focus by maximizing the center value and minimizing the edge value.

[0072] S104: Construct an uncertainty penalty loss term to constrain the prediction results with high uncertainty; S105: The binary classification loss term, evidence classification loss term, concave prior regularization loss term, and uncertainty penalty loss term are weighted and combined to construct a joint loss function: ;in, To control the weights of the space regularization term, To control the weight of the uncertainty penalty term, For uncertainty penalty loss item; S106: The hierarchical uncertainty classification network is uniformly optimized using the joint loss function and the network parameters are updated using the backpropagation algorithm until the validation set loss converges.

[0073] Specifically, during the training phase, a retina dataset containing 5000 samples was used (divided into training, validation, and test sets in an 8:1:1 ratio); the AdamW optimizer was employed, with an initial learning rate set to... The weight decays to Joint loss function The hyperparameters in the configuration are set as follows: =0.1 (Weight of control space regularization term), =0.05 (weight of the uncertainty penalty term), target uncertainty upper limit =0.15; The network parameters are updated end-to-end using the backpropagation algorithm until the loss on the validation set converges.

[0074] Therefore, in order to ensure that those skilled in the art can reproduce this application without ambiguity, the following provides detailed limitations on the underlying network structure, feature fusion weights at each scale, convolutional kernel parameters, and invocation rules: 1. Multi-scale structure and parameters of the dual-stream feature extraction network: Both the first and second encoders adopt an improved ResNet-50 architecture, and a Feature Pyramid Network (FPN) is introduced to fuse multi-scale features. Stage 1-4 convolution parameters: Stage 1 contains one 7×7 convolutional kernel (stride 2, output channels 64); Stages 2-4 contain 3, 4, and 6 Bottleneck residual blocks respectively, with internal convolutional kernels of 1×1 and 3×3 (stride 1 and 2), and output channels of 256, 512, and 1024 respectively. Feature fusion weights at each scale: To fully utilize shallow edge details and deep semantic information, feature maps of Stages 2, 3, and 4 are extracted (spatial dimensions 56×56, 28×28, and 14×14 respectively). The number of channels is uniformly reduced to 256 through 1×1 convolution, and then upsampled to a uniform size of 56×56 through bilinear interpolation; learnable scale fusion weights are used. Perform a weighted summation, with the initial weights set as follows: , , And dynamically updated during training using Softmax normalization; the final output and The dimensions are 56×56×256 (i.e., H=56, W=56, =256).

[0075] 2. Implementation details of the foveal perception spatial guidance module, lightweight convolutional branches. It consists of three convolutional layers: the first layer is a 3×3 convolution (256 input channels, 128 output channels, stride 1, padding 1) followed by ReLU activation; the second layer is a 3×3 convolution (128 input channels, 64 output channels) followed by ReLU activation; and the third layer is a 1×1 convolution (64 input channels, 1 output channel) followed by Sigmoid activation. The calling rule is that during the inference phase, if the input image has not been aligned to the hardware-level macular center, the module will automatically... The peak coordinates are offset from the geometric center of the image to ensure that the center of the heatmap is always aligned with the anatomical concave center.

[0076] 3. Disease perception modality routing and classification head parameters, cross-attention parameters: The projection matrix dimension is 256×64 (i.e., in multi-head attention, the number of heads h=4, and each head dimension is 256×64). =64); Multilayer perceptron It contains two linear layers; the first layer has an input dimension of 512 ( and The first layer is concatenated after global average pooling, with an output dimension of 128, followed by ReLU and Dropout (p=0.3); the second layer has an input of 128 and an output of 1, followed by Sigmoid to output γ; the hierarchical classification head consists of: a first-level binary fully connected layer with an input dimension of 256 and an output of 1; and a second-level ternary evidence output layer with an input dimension of 256 and an output of 3 (ensuring γ through Softplus activation). ≥0).

[0077] 4. Training hyperparameters and calling rules, optimizer and scheduler: The AdamW optimizer is used, with an initial learning rate of... Weight decay A Cosine Annealing learning rate scheduler is used, with a minimum learning rate. Warmup steps: 500; Batch Size and Epoch: Batch Size is set to 32, and the total training Epoch is set to 100; EarlyStopping strategy is adopted, and training is stopped if the validation set loss does not decrease for 15 consecutive Epochs.

[0078] Therefore, for reference Figure 13 As shown. This application also relates to a retinal multimodal image classification system based on spatial awareness and modal routing that implements the method described in any of the above claims, comprising: The image acquisition module is equipped with a data interface for acquiring 2D OCT and 2D OCTA images corresponding to the fundus region of the same subject and performing image quality verification, visual field cropping, size uniform resampling, grayscale standardization and two-dimensional spatial registration processing to construct a multimodal retinal image dataset with a unified spatial coordinate system. The feature extraction module, which incorporates a dual-stream convolutional neural network or Vision Transformer, is used to input the 2D OCTA image and the 2D OCTA image into the dual-stream feature extraction network respectively. It extracts structural feature representations through the structural feature encoding branch and extracts vascular feature representations through the vascular feature encoding branch to generate corresponding OCT structural feature maps and OCTA vascular feature maps. The spatial guidance module, which has a built-in lightweight convolutional branch and mask generator, is used to generate a spatial saliency heatmap based on the OCT structural feature map and OCTA vascular feature map, using the anatomical and physical priors of the fovea of ​​the retina, and to spatially weight the OCT structural feature map and OCTA vascular feature map based on the spatial saliency heatmap, thereby generating an OCT enhancement feature map and an OCTA enhancement feature map focusing on the macular region. The modal routing module has a built-in dual-branch cross-attention computing unit and a multilayer perceptron. It is used to construct structural dominant feature extraction path and vascular dominant feature extraction path respectively based on the OCT enhanced feature map and OCT enhanced feature map using an asymmetric cross-attention mechanism to obtain structural dominant features and vascular dominant features. According to the preset modal routing gating weight, the structural dominant features and vascular dominant features are dynamically aggregated to generate multimodal fusion features. The hierarchical classification module, which incorporates a binary classifier, an evidence generator, and an uncertainty calculation unit, is used to input the multimodal fusion features into a hierarchical uncertainty classification network to perform binary classification of normal and abnormal samples. When the judgment result is a normal sample, the normal classification result is directly output. When the judgment result is an abnormal sample, the classification uncertainty score is calculated by combining evidence deep learning to perform tri-class classification of age-related macular degeneration, diabetic retinopathy, and other fundus diseases, so as to output a four-class classification result including NORMAL, AMD, DR, and OTHERS.

[0079] In one embodiment of experimental verification and effect analysis of this application, in order to verify the technical effect of this application, this embodiment conducted rigorous comparative and ablation experiments on real clinical datasets, comparing this application (Full Model) with the following existing mainstream baseline models: OCT-only (ResNet-50), OCTA-only (ResNet-50), and Concat-Fusion. The specific performance comparison is shown in Table 1: Table 1: Comparison of Multiclass Classification Performance of Mainstream Models Comparing the quantitative metrics of single-modal, simple stitching fusion, and the proposed algorithm, we can see that the classification performance of the OCT-only and OCTTA-only single-modal models is similar. The accuracy (ACC) of OCT-only is 0.7514, while that of OCTTA-only is 0.7427. The BA and MCC values ​​of the two are almost identical, with only the F1-score of OCTTA-only (0.7578) being slightly higher than that of OCT-only (0.7460), proving that the two fundus modal images contain complementary lesion discrimination information. The Concat-Fusion multimodal scheme, which uses direct channel stitching, outperforms the two single-modal baselines in all four metrics. Among them, the ACC is improved by 1.18%, the F1-score by 2.41%, the BA by 1.58%, and the MCC by 5.12% compared with the best single-modal baseline, verifying that the fusion structure and blood flow features can effectively improve classification performance. The multimodal model designed in this paper... The modality fusion scheme achieved the best results in all four metrics: ACC, F1-score, BA, and MCC. Compared with Concat-Fusion, ACC improved from 0.7632 to 0.7876 (an improvement of 2.44%), F1-score improved from 0.7819 to 0.8135 (an improvement of 3.16%), BA improved from 0.7133 to 0.7343 (an improvement of 2.10%), and MCC improved from 0.7223 to 0.7523 (an improvement of 3.00%). All four metrics increased steadily and simultaneously, without any single metric showing excessively high values. It has stronger recognition accuracy and robustness in the class imbalance fundus image classification task.

[0080] To verify the independent contributions of the three core innovation modules in steps S3, S4, and S5, ablation experiments were conducted on the test set. The baseline model was "dual-stream ResNet-50 + feature concatenation + standard Softmax classification head"; the specific experimental results are shown in Table 2. Table 2: Ablation Experiment Results of Core Module Introducing any submodule individually can reduce the false diagnosis rate. Adding only the spatial guidance module S3 reduced the false diagnosis rate to 34.6%, a decrease of 3.0 percentage points from the baseline; introducing only the modal routing module S4 reduced the false diagnosis rate to 34.9%, a decrease of 2.7 percentage points from the baseline; and using only the uncertainty classification header S5 reduced the false diagnosis rate to 33.4%, a decrease of 4.2 percentage points. The comparison shows that the uncertainty classification header has the greatest single-module benefit in reducing false diagnoses, followed by the spatial guidance module, while the modal routing module has a relatively smaller single-module improvement. All three modules have independent optimization capabilities.

[0081] After embedding all three modules into the network, the misdiagnosis rate of the complete model decreased to 30.3%, the lowest among all experimental groups. Compared with the original baseline model, the total misdiagnosis rate decreased by 7.3 percentage points; compared with the best-performing single module (S5 only, 33.4%), it can still be further reduced by 3.1 percentage points. This result demonstrates that the three modules have complementary and synergistic effects: the spatial guidance module focuses on the effective feature regions of fundus lesions, the modal routing module adaptively adjusts the weights of dual-modal features, and the uncertainty classification head corrects the prediction bias of fuzzy and difficult-to-distinguish samples. The combination of the three can reduce lesion misdiagnosis at multiple levels, verifying the necessity and completeness of the overall architecture design in this paper.

[0082] Therefore, for reference Figure 14 As shown. This application also relates to an electronic device, comprising: at least one processor (such as a CPU, GPU, or NPU, preferably a GPU containing Tensor Cores to accelerate attention mechanisms and matrix multiplication operations); and a memory (such as RAM and ROM) communicatively connected to said processor; The memory stores a computer program that can be executed by the processor. When the program is executed by the processor, it enables the processor to implement all the steps of the retinal multimodal image classification method based on spatial awareness and modal routing described in the above embodiments.

[0083] Therefore, for reference Figure 14 As shown. This application also relates to a computer-readable storage medium storing computer instructions for causing a computer to execute the retinal multimodal image classification method based on spatial awareness and modal routing described in any of the above claims.

[0084] Therefore, in one exemplary clinical application scenario of this application, specifically in an actual ophthalmological physical examination or community screening scenario, the system of this application can be deployed on an edge computing terminal or a cloud server.

[0085] The system automatically acquires 2D OCT and 2D OCTA images by scanning the patient's fundus with an OCT / OCTA device.

[0086] The system automatically executes the above S1-S5 steps. For typical AMD or DR patients, the system not only outputs the disease category, but also visualizes the "spatial salience heatmap" (highlight macular lesions) and "modal routing weight" (such as displaying "structural feature contribution 80%) on the interface to assist doctors in quick review.

[0087] For patients with rare fundus diseases (such as retinitis pigmentosa) or whose images have severe artifacts, the uncertainty score u calculated by the system will exceed 0.25. The system will automatically mark them as "OTHERS (manual review recommended)", thereby effectively avoiding the risk of medical accidents caused by AI misdiagnosing rare diseases as common diseases.

[0088] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0089] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A retinal multimodal image classification method based on spatial perception and modal routing, characterized in that, Includes the following steps: S1: Acquire 2D OCT and 2D OCTA images corresponding to the fundus region of the same subject and perform image quality verification, visual field cropping, size uniform resampling, grayscale standardization and two-dimensional spatial registration processing to construct a multimodal retinal image dataset with a unified spatial coordinate system; S2: Input the 2D OCT image and 2D OCTA image into the dual-stream feature extraction network respectively, extract the structural feature representation through the structural feature encoding branch and extract the vascular feature representation through the vascular feature encoding branch to generate the corresponding OCT structural feature map and OCTA vascular feature map; S3: Based on the OCT structural feature map and OCTA vascular feature map, a spatial saliency heatmap is generated using the anatomical and physical prior of the fovea of ​​the retina. The OCT structural feature map and OCTA vascular feature map are then spatially weighted according to the spatial saliency heatmap to generate an OCT enhancement feature map and an OCTA enhancement feature map focusing on the macular region. S4: Based on the OCT enhanced feature map and the OCT enhanced feature map, an asymmetric cross-attention mechanism is used to construct a structure-dominant feature extraction path and a blood vessel-dominant feature extraction path respectively to obtain structure-dominant features and blood vessel-dominant features. According to the preset modal routing gating weight, the structure-dominant features and blood vessel-dominant features are dynamically aggregated to generate multimodal fusion features. S5: Input the multimodal fusion features into the hierarchical uncertainty classification network to perform binary classification of normal samples and abnormal samples. When the judgment result is a normal sample, the normal classification result is directly output. When the judgment result is an abnormal sample, the classification uncertainty score is calculated by combining evidence deep learning to perform tri-class classification of age-related macular degeneration, diabetic retinopathy and other fundus diseases, so as to output a four-class classification result including NORMAL, AMD, DR and OTHERS.

2. The retinal multimodal image classification method based on spatial perception and modal routing according to claim 1, characterized in that, In S1, the method includes the following steps: S11: 2D OCT images and 2D OCTA images of the fundus region of the same subject were acquired using optical coherence tomography (OCT) equipment; S12: Perform image quality detection on the acquired 2D OCT images and 2D OCTA images, remove abnormal images with severe motion artifacts, image blur, local occlusion and field of view loss, and retain image samples that meet the preset quality requirements. S13: Perform background region cropping on the 2D OCT image and 2D OCTA image after quality inspection, remove invalid black areas at the image edges and retain the effective retinal area including the fovea region of the macula; S14: The cropped 2D OCT image and 2D OCTA image are uniformly resampled to a preset input size and the image grayscale distribution is normalized using the Z-score normalization method to reduce the image intensity differences caused by different devices, different subjects and different acquisition conditions. S15: Perform data augmentation processing on the standardized image samples. The data augmentation processing includes at least random horizontal flipping, random angle rotation, and random brightness and contrast adjustment to improve the model's generalization ability to retinal image morphological changes. S16: Using the registration information built into the device or an image registration method based on mutual information criteria, perform two-dimensional spatial registration processing on the 2D OCT image and the 2D OCTA image to achieve a spatial correspondence between the macular region and retinal anatomy in the two modal images, so as to construct a multimodal retinal image dataset with a unified spatial coordinate system.

3. A retinal multimodal image classification method based on spatial perception and modal routing according to claim 1 or 2, characterized in that, In S2, the method includes the following steps: S21: Construct a two-stream feature extraction network including a first encoder and a second encoder; S22: Input the registered 2D OCT image into the first encoder to extract structural features that characterize the retinal layer structure, macular morphological changes and abnormal information of lesion tissue, and input the registered 2D OCTA image into the second encoder to extract vascular features that characterize the capillary distribution status, blood perfusion status and abnormal changes of microvessels. S23: Extract multi-scale feature maps from multiple layers of different depths of the first encoder and the second encoder respectively, and use the feature pyramid fusion mechanism to fuse shallow texture information, mid-level structural information and deep semantic information. S24: Through unified channel mapping and unified spatial scale processing, multi-scale features at different levels are mapped to a unified feature space, and learnable fusion weights are used to weight and aggregate the features at each scale to generate an OCT structure feature map with preset channel dimensions and spatial size. And OCTA vascular feature maps: Where H is the feature map height and W is the feature map width. For the feature map channel dimension.

4. The retinal multimodal image classification method based on spatial perception and modal routing according to claim 3, characterized in that, In S3, the method includes the following steps: S31: Input the OCT structural feature map into a lightweight convolution branch, and extract the physical curvature features that reflect the curvature state between retinal layers and the fovea morphology through convolution operations. S32: Generate an initial response map based on the physical bending morphological features, and perform Sigmoid function activation and spatial smoothing on the initial response map to generate a spatial saliency heatmap with values ​​in the range [0,1]. ,in It is a lightweight convolution mapping function. It is the Sigmoid activation function. Spatial saliency heatmap; S33: Perform spatial smoothing on the spatial saliency heatmap to create a continuous high-response region in the fovea region of the macula and maintain a low response value in the peripheral region; S34: Using the spatial saliency heatmap, the OCT structural feature map and the OCTA vascular feature map are multiplied element-wise to generate an OCT enhancement feature map and an OCTA enhancement map focusing on the macular lesion region: , Where ⊙ represents the element-wise multiplication operation under the broadcast mechanism. For OCT enhanced feature maps, Enhance the feature map for OCTA.

5. A retinal multimodal image classification method based on spatial perception and modal routing according to claim 4, characterized in that, In S4, the method includes the following steps: S41: Using the OCT enhanced feature map as the structural modality input and the OCTA enhanced feature map as the vascular modality input, construct the structural-dominated cross-attention branch and the vascular-dominated cross-attention branch respectively; S42: In the structure-dominant cross-attention branch, using the OCT enhanced feature map as the query feature and the OCT enhanced feature map as the key and value features, a first query vector, a first key vector, and a first value vector are generated through linear mapping: , , ,in A learnable linear projection matrix for the structure-dominant cross-attention branch; S43: Calculate cross-modal attention weights using the first query vector, the first key vector, and the first value vector, and generate dominant structural features that reflect structural pathological characteristics. ,in The dominant structural feature Let be the dimension of the key vector. This is the transpose of the first key vector; S44: In the blood vessel-dominated cross-attention branch, using the OCTA enhanced feature map as the query feature and the OCT enhanced feature map as the key and value features, a second query vector, a second key vector, and a second value vector are generated through linear mapping: , , ,in A learnable linear projection matrix for blood vessel-dominated cross-attention branches; S45: Calculate cross-modal attention weights using the second query vector, the second key vector, and the second value vector, and generate vascular dominant features reflecting vascular pathology characteristics: ,in The dominant feature is the blood vessel. This is the transpose of the second key vector.

6. The retinal multimodal image classification method based on spatial perception and modal routing according to claim 5, characterized in that, In S4, the method includes the following steps: S46: Perform global average pooling on the structural dominant features and the vascular dominant features respectively to obtain the corresponding global semantic representation vectors; S47: Concatenate the global semantic representation vector corresponding to the dominant structural feature and the global semantic representation vector corresponding to the dominant vascular feature in the channel dimension to form a joint feature; S48: Input the joint features into a multilayer perceptron, learn the synergistic relationship between structural pathological features and vascular pathological features through nonlinear mapping, and output scalar gating weights: ,in For the multilayer perceptron mapping function, To be and splicing along the channel dimension; S49: Dynamically weight and fuse the structural dominant features and vascular dominant features according to the gating weights to generate multimodal fusion features: ;in, This represents the final multimodal features after fusion.

7. A retinal multimodal image classification method based on spatial perception and modal routing according to claim 6, characterized in that, In S5, the method includes the following steps: S51: Input the final multimodal fusion feature into the first-level classification network, compress the final multimodal fusion feature through the global average pooling layer, and output the binary classification prediction results corresponding to the normal category and the abnormal category through the first fully connected layer; S52: The samples corresponding to the abnormal probability prediction values ​​that are less than the preset values ​​are classified as NORMAL, and the samples corresponding to the abnormal probability prediction values ​​that are greater than or equal to the abnormal probability prediction values ​​are input into the second-level classification network. S53: Utilize the evidence generation layer to output non-negative evidence values ​​corresponding to the AMD, DR, and OTHERS categories respectively, and construct the total amount of evidence required for disease category determination based on the evidence values: ,in The original evidence value output by the network. S represents the total number of categories in the second-level classification, and S represents the total amount of evidence collected by the model. For the concentration parameters of the Dirichlet distribution; S54: Calculate the prediction probability and overall uncertainty score for each category using subjective logic theory based on the total amount of evidence. ;in, To predict probabilities, The overall uncertainty score; S56: Based on the relationship between the cognitive uncertainty score and the preset safety threshold, make open set disease identification and classification decisions.

8. A retinal multimodal image classification method based on spatial perception and modal routing according to claim 7, characterized in that, In S5, a preset safety threshold is set. If the uncertainty score is less than the preset safety threshold, the AMD or DR classification result with the highest probability in the total evidence is output. If the uncertainty score is higher than or equal to the preset safety threshold, the model evidence is deemed insufficient, and the result is forced to be output as OTHERS.

9. A retinal multimodal image classification method based on spatial perception and modal routing according to claim 1 or 7, characterized in that, In S5, the training method for the hierarchical uncertainty classification network includes the following steps: S101: Construct a binary classification loss term to constrain the determination results of normal and abnormal categories: ;in This represents the total number of samples in the current training batch. Let be the true label of the i-th sample in the first-level binary classification; Let be the predicted probability that the i-th sample output by the first-level binary classification network belongs to the abnormal category; S102: Construct an evidence-based classification loss term to constrain the classification results for the AMD, DR, and OTHERS categories: ;in One-hot encoding of the real label. It is the Digamma function; S103: Construct a concave prior regularization loss term based on the spatial saliency heatmap, forcing the center response to be higher than the edge response: ;in, and These are the sets of pixel coordinates for the image center and edges, respectively. This represents the total number of pixels in the set. The value of the spatial saliency heatmap at coordinates (x, y); S104: Construct an uncertainty penalty loss term to constrain the prediction results with high uncertainty; S105: The binary classification loss term, evidence classification loss term, concave prior regularization loss term, and uncertainty penalty loss term are weighted and combined to construct a joint loss function: ;in, To control the weights of the space regularization term, To control the weight of the uncertainty penalty term, For uncertainty penalty loss item; S106: The hierarchical uncertainty classification network is uniformly optimized using the joint loss function and the network parameters are updated using the backpropagation algorithm until the validation set loss converges.

10. A retinal multimodal image classification system based on spatial perception and modal routing, implementing the method of any one of claims 1-9, characterized in that, include: The image acquisition module is used to acquire 2D OCT images and 2D OCTA images corresponding to the fundus region of the same subject and perform image quality verification, visual field cropping, size uniform resampling, grayscale standardization and two-dimensional spatial registration processing to construct a multimodal retinal image dataset with a unified spatial coordinate system. The feature extraction module is used to input the 2D OCT image and the 2D OCTA image into the dual-stream feature extraction network, extract structural feature representations through the structural feature encoding branch and extract vascular feature representations through the vascular feature encoding branch, so as to generate the corresponding OCT structural feature map and OCTA vascular feature map. The spatial guidance module generates a spatial saliency heatmap based on the OCT structural feature map and the OCTA vascular feature map, using the anatomical and physical priors of the fovea of ​​the retina, and spatially weights the OCT structural feature map and the OCTA vascular feature map based on the spatial saliency heatmap to generate an OCT enhancement feature map and an OCTA enhancement feature map focusing on the macular region. The modal routing module is used to construct structural dominant feature extraction paths and vascular dominant feature extraction paths respectively based on the OCT enhanced feature map and the OCT enhanced feature map using an asymmetric cross-attention mechanism to obtain structural dominant features and vascular dominant features. Based on the preset modal routing gating weights, the structural dominant features and vascular dominant features are dynamically aggregated to generate multimodal fusion features. The hierarchical classification module is used to input the multimodal fusion features into the hierarchical uncertainty classification network to perform binary classification of normal samples and abnormal samples. When the judgment result is a normal sample, the normal classification result is directly output. When the judgment result is an abnormal sample, the classification uncertainty score is calculated by combining evidence deep learning to perform tri-class classification of age-related macular degeneration, diabetic retinopathy and other fundus diseases, so as to output a four-class classification result including NORMAL, AMD, DR and OTHERS.

Images