Myopia maculopathy atn grading method and system based on multi-modal medical images

Abstract

This invention discloses a method and system for grading myopic macular degeneration (ATN) based on multimodal medical imaging. The system includes an image acquisition module for acquiring multimodal retinal images to be graded; a preprocessing module for preprocessing the multimodal images to obtain image sequences that meet preset grading requirements; and a grading module for inputting the image sequences into a pre-stored multimodal multi-task grading model. The model performs feature extraction, multimodal fusion, and multi-task classification on the image sequences, outputting a lesion grade prediction result for each grading task. This invention significantly improves the model's sensitivity to pathological regions by performing pixel-level segmentation of the input images and locating key anatomical structures. In subsequent steps, spatial weighting is applied to CFP image modalities, and specific regions of OCT and OCTA are enhanced, effectively improving the accuracy and robustness of lesion diagnosis and localization.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for classifying myopic macular degeneration (ATN) based on multimodal medical imaging. Background Technology

[0002] To more accurately assess myopic macular degeneration, the internationally widely adopted ATN grading system (Atrophy (A), Traction (T), and Neovascularization (N)) is used. This system assesses retinal structural damage from multiple dimensions, providing a structured basis for clinical decision-making and follow-up management.

[0003] With the development of medical imaging technology, various imaging techniques such as optical coherence tomography (OCT), optical coherence tomography angiography (OCTA), and color fundus photography (CFP) are widely used in the diagnosis and evaluation of retinal diseases. These images can reflect the structure, vascular status, and lesion characteristics of the retina from different dimensions. For example, OCT images can show the layered structure of the retina and its morphological changes, OCTA images can provide retinal blood flow information, and CFP images can intuitively present the lesion area and the overall retinal morphology. Therefore, in clinical practice, doctors usually need to integrate multimodal medical imaging information to analyze and classify myopic macular degeneration.

[0004] Currently, the grading of myopic macular degeneration mainly relies on clinicians to manually interpret medical images. This method not only requires extensive clinical experience but is also inefficient when dealing with large amounts of image data. Furthermore, the interpretation results may differ between different doctors, affecting the consistency and stability of the diagnostic results. With the development of artificial intelligence technology, deep learning-based medical image analysis methods are gradually being applied to the automatic detection and grading of retinal diseases, which has improved the efficiency of image analysis to some extent.

[0005] The patent document with publication number CN120809163A discloses a method and device for generating a multimodal intelligent diagnostic model of myopic macular degeneration. The feature fusion operation of this method is simply splicing or post-fusion, without considering the fine interaction and structure-function association across modalities. Therefore, it is difficult to achieve accurate diagnosis in modal missing scenarios.

[0006] DINOv3 ViT, with its advantages of self-supervised pre-training, strong dense features, frozen backbone, and lightweight head, has been rapidly deployed in medical imaging diagnosis. Its core applications include lesion segmentation, disease classification, and multimodal analysis, as shown in patent document CN120876444A. However, this self-supervised diagnostic method for medical imaging is trained based on a set of two-dimensional slice images, which has significant domain differences from CFP images. It is difficult to accurately capture subtle lesion features such as microaneurysms, hemorrhages, and exudations. It lacks targeted modeling for key areas such as the macula and optic disc, is easily affected by background interference, and direct transfer cannot fully adapt to the subtle lesion features of the fundus, thus limiting diagnostic accuracy and clinical applicability.

[0007] The patent document with publication number CN120809163A proposes a method for generating a multimodal intelligent diagnostic model for myopic macular degeneration. It mainly involves preprocessing multimodal data, concatenating or uniformly mapping text and image features, and constructing a diagnostic model by combining convolutional operations and fully connected layers. This method essentially adopts a fusion strategy based on feature concatenation or uniform mapping, lacking differentiated modeling of structural and functional information between different modalities, and also failing to address fine-grained interaction mechanisms between cross-modal features. Furthermore, in the case of missing modalities, it mainly relies on feature concatenation or data reconstruction, without establishing an explicit cross-modal feature inference mechanism, making it difficult to guarantee the stability of the model under incomplete input conditions.

[0008] Furthermore, patent document CN120876444A proposes a self-supervised diagnostic method for medical images based on a multimodal attention mechanism. This method pre-trains multimodal images using MAE and DINO self-supervised learning strategies and achieves feature fusion using a Perceiver model and a cross-modal attention mechanism. This method primarily focuses on improving general representation learning capabilities and unifying the modeling of multimodal features. Its fusion method involves encoding features from each modality, concatenating them in a unified feature space, and performing global attention modeling. However, it does not design differentiated fusion strategies for the structural and functional information differences between different modalities, nor does it establish a feature organization method oriented towards specific clinical grading tasks. In addition, this method does not consider modality-deficient scenarios, and the model relies on complete modality input. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for grading myopic macular degeneration using multimodal medical imaging, which addresses the problems of insufficient utilization of single-modal information, neglect of inter-task correlation, and insufficient robustness of models under modal absence in the prior art.

[0010] To achieve the above objectives, this invention proposes a multimodal medical imaging-based method for classifying myopic macular degeneration (ATN), comprising the following steps: S1. Acquire image data: Acquire the retinal multimodal image to be classified; the multimodal image includes at least one of OCT image, OCTA image and CFP image; S2. Image preprocessing and anatomical structure localization: Perform basic preprocessing and anatomical localization on the multimodal images to obtain an image sequence that meets the preset hierarchical processing requirements; S3. Image feature extraction: Input the image sequence into three independent visual Transformer networks to extract features from the images of the three modalities; if more than one modality is missing, proceed to S4; if all three modalities are present, proceed to S5. S4. Modality Missing Handling: Utilize existing modality features to dynamically generate feature estimates for missing modalities through a predictor; the predictor consists of two MLP layers and cross-attention; S5. Multimodal feature fusion: Intramodal differential fusion and cross-modal collaborative attention fusion of features from three modalities; S6. Enhance lesion features: Enhance the expression of lesion region features by using lesion probability maps to obtain lesion enhancement features; S7. Output grading results: Input the lesion enhancement features into three independent classification heads, and obtain the probabilities of each level (A, N, T) through softmax.

[0011] Preferably, S2 includes the following steps: S2.1 Basic preprocessing: Normalize and standardize the images of each modality, and perform center cropping on the CFP image to retain 80% of the central region; S2.2 Anatomical structure localization: The key structures are segmented at the pixel level using the Swin-UNet model. The key structures include the retinal layer boundary, the fovea of ​​the macula, the vascular network, the non-perfusion area, the avascular area of ​​the macula, the optic disc, the macula, the hemorrhage area, and the exudation area. S2.3, Anatomical Attention Map Generation: Encode the probability map or boundary map obtained from segmentation into an anatomical attention map.

[0012] Preferably, step S3 includes the following steps: For two or more OCT and OCTA images, extract the [CLS] token and patch feature sequences from the input image respectively; For a single CFP image, the patch features of the CFP image are spatially weighted using the anatomical attention map obtained in S2.3: the attention map is upsampled to the same size as the patch grid and then multiplied element-wise with the patch features to enhance the response of key regions.

[0013] Preferably, in step S5, the intra-modal differential fusion step is as follows: For the OCT modality: The [CLS] token features of each slice are fused through learnable attention pooling: a weighted sum of the features of each slice is calculated, with the weights generated by a small attention network based on the slice features; For the OCTA modality: The [CLS] token features of each layer image are input into the channel attention module: first, global average pooling is used to obtain the feature descriptions of each layer, then the weights of each layer are generated through two layers of MLP, and the weighted fusion is used to obtain the OCTA modality features; For CFP image modalities: the patch feature sequence is multiplied element-wise with the anatomical attention map obtained in S2.3 to obtain the enhanced patch features, which are then obtained by global average pooling to obtain the CFP image modal features; The steps of the cross-modal collaborative attention fusion are as follows: The enhanced patch feature sequences of the three modalities are concatenated and input into a shared cross-modal Transformer encoder. This encoder uses a self-attention mechanism to enable tokens from different modalities to interact with each other, learn cross-modal joint representations, and outputs the fused [CLS] token as the comprehensive feature \( f_{fusion} \).

[0014] As a preferred option, the specific steps of S5 are as follows: Based on the comprehensive feature \( f_{fusion} \), a lightweight segmentation head is introduced to perform pixel-level prediction of common lesions, generate a lesion probability map, multiply the lesion probability map with a feature map before cross-modal fusion, or re-weight the segmentation result as a weight through attention pooling to obtain lesion enhancement features.

[0015] As a preferred option, in S7, the grading results are converted into binary screening results and a structured report is generated.

[0016] This invention also proposes an ATN grading system for myopic macular degeneration based on multimodal medical imaging, used to implement the aforementioned ATN grading method for myopic macular degeneration based on multimodal medical imaging, comprising: The system image acquisition module is used to acquire retinal multimodal images to be classified, wherein the multimodal images include at least one of OCT images, OCTA images, and CFP images; The preprocessing module is used to preprocess the multimodal images to obtain an image sequence that meets the preset hierarchical processing requirements; The grading module is used to input the image sequence into a pre-stored multimodal multitask grading model, and to perform feature extraction, multimodal fusion and multitask classification on the image sequence through the multimodal multitask grading model, and output the lesion level prediction result for each grading task. The multimodal, multi-task hierarchical model includes three parallel feature extraction backbone networks, one multimodal fusion module, and three classification heads. The three feature extraction backbone networks correspond to three modalities: OCT, OCTA, and CFP images, respectively. The multimodal fusion module includes two sub-modules: intramodal differential fusion and cross-modal collaborative attention fusion. The three classification heads correspond to three hierarchical tasks: A-level, N-level, and T-level, respectively.

[0017] Preferably, the feature extraction backbone network adopts the DINOv3-ViT-B / 16 architecture; the multimodal fusion module adopts a learnable weighted fusion strategy; and the multi-task classification head consists of three independent linear classifiers.

[0018] This invention also proposes a multimodal, multi-task grading model for diagnosing myopic macular degeneration, the training steps of which are as follows: Step 1: Collect multimodal retinal images with A, N, and T grade annotations as training datasets. Divide patients into training and validation sets according to a preset ratio. The multimodal retinal images include OCT, OCTA, and CFP images. Step 2: Load the weights pre-trained on ImageNet into the three DINOv3 ViT-B / 16 backbone networks, and initialize the remaining modules with Xavier. Step 3: Perform end-to-end fine-tuning on the DINOv3 ViT-B / 16 backbone network. All layer parameters participate in gradient updates. Differential learning rates are used. New modules use higher learning rates, while other modules use lower learning rates. Step 4: Form a multimodal, multi-task hierarchical model based on the updated parameters.

[0019] Preferably, in step three, the learning rate of the DINOv3 ViT-B / 16 backbone network is set to 1e. -5 The learning rates for the cross-modal predictor, the fusion Transformer, the segmenter head, and the classifier head are set to 1e. -3 All modules use the AdamW optimizer with β1=0.9, β2=0.999, and weight decay of 1e. -4 .

[0020] Preferably, during training, 20% of the samples are randomly selected and one modality is randomly discarded. The [CLS] features of the remaining modalities are used to generate features of the discarded modality through a lightweight predictor consisting of two MLP layers and cross attention. At the same time, the L2 reconstruction loss between the generated features and the true features is calculated and jointly optimized with the main task loss, so that the network learns the intrinsic complementary relationship between modalities. During inference, if a modality is truly missing, the existing modal features are directly used to dynamically generate feature estimates of the missing modality through the predictor, replacing the traditional fixed learnable tokens or zero-padding.

[0021] As a preferred approach, the segmentation task and the grading task are optimized in parallel during training: the segmentation loss (Dice + cross-entropy) and the grading loss (Focal Loss) are trained together, enabling the model to explicitly learn the causal relationship between lesions and grading.

[0022] As a preferred option, the specific training process for the missing modality is as follows: (1) Randomly select 20% of the training samples, and randomly discard one modality in each sample to simulate a real missing scenario; (2) Remove the image of the discarded modality from the computation graph and input the features of the remaining modalities (obtained by the feature extraction step in S3) into a lightweight cross-modal predictor, which consists of two MLP layers and a cross attention module, and is able to predict the [CLS] features of the discarded modality based on the existing modal features; (3) At the same time, the true features of the discarded modalities are extracted using the complete image (all modalities exist), and the L2 loss between the predicted features and the true features is calculated as the reconstruction loss \( ​​L_{recon} \). (4) Jointly optimize the reconstruction loss and the main task loss (hierarchical loss, segmentation loss) so that the predictor learns the intrinsic relationship between modes and provides self-supervised auxiliary signals for the whole model.

[0023] As a preferred approach, data augmentation is applied during the training phase: random horizontal flipping, color jitter (brightness ±0.2, contrast ±0.2, saturation ±0.2), and random erasure (p=0.1); the CFP image is center-cropped to retain 80% of the central area to focus on the macula.

[0024] The beneficial effects of this invention are: (i) This invention proposes a structure that combines intramodal differential fusion and cross-modal collaborative attention fusion. It designs feature aggregation methods for the three modalities of OCT, OCTA and CFP respectively, and realizes fine-grained interaction at the patch level through cross-modal Transformer, thereby establishing explicit associations between structural information, blood flow function information and overall morphological information. It not only preserves the intramodal specific information, but also fully explores the semantic associations between OCT structure, OCTA function and CFP image as a whole. At the same time, it solves the problem of modality missing, and can give accurate grading results when there are no images of other examinations. (ii) This invention adopts a mechanism that combines domain-adapted pre-training for ophthalmic images with task-driven fine-tuning. Based on the labeled ophthalmic dataset, the DINOv3 ViT model is fine-tuned end-to-end with all parameters, enabling the model to better capture fine-grained lesion features such as microvascular abnormalities, hemorrhage and exudation, and to form a close coupling relationship with the ATN grading task for myopic macular degeneration, thus obtaining a customized model adapted to the specific ophthalmic dataset. (iii) This invention constructs a cross-modal feature prediction mechanism based on cross-attention, which dynamically generates missing modal features through existing modalities and performs joint optimization in combination with reconstruction constraints, enabling the model to learn the intrinsic correspondence between modalities, thereby maintaining high diagnostic accuracy even when the modalities are incomplete; (iv) This invention introduces an attention-guided mechanism based on anatomical structure segmentation. By locating key anatomical structures and generating attention maps, the expression of lesion areas is enhanced during the feature extraction stage, thereby improving the model's sensitivity to key pathological areas. (V) This invention constructs a cross-modal feature completion network. During training, 20% of the samples are randomly selected and one modality is randomly discarded. The [CLS] features of the remaining modalities are used to generate the features of the discarded modalities through a lightweight predictor composed of two MLP layers and cross attention. At the same time, the L2 reconstruction loss between the generated features and the real features is calculated and jointly optimized with the main task loss, so that the network learns the intrinsic complementary relationship between modalities. During inference, if a modality is truly missing, the existing modal features are directly used to dynamically generate the feature estimate of the missing modality through the predictor, replacing the traditional fixed learnable tokens or zero-filling. This not only more accurately completes the missing information, but also enhances the model's ability to model multimodal associations through self-supervised reconstruction tasks. (vi) This invention directly outputs grading results and visualization reports, with high adaptability and can be used directly in clinical practice. It can achieve ATN grading and generate risk reports based on multi-dimensional comprehensive examination results, and also supports independent grading and risk screening based on a single examination result. It is suitable for both tertiary specialized hospitals and primary healthcare institutions. It can achieve accurate grading and is also suitable for large-scale rapid screening, and has wide applicability.

[0025] The features and advantages of the present invention will be described in detail through embodiments and in conjunction with the accompanying drawings. Attached Figure Description

[0026] Figure 1 A schematic diagram of the ATN grading system for myopic macular degeneration based on multimodal medical imaging; Figure 2 This is a flowchart illustrating the ATN grading method for myopic macular degeneration based on multimodal medical imaging. Figure 3 A schematic diagram of a multimodal, multi-task hierarchical model framework for diagnosing myopic macular degeneration; Figure 4 This is a visualization of the lesion identification based on multimodal medical imaging according to the present invention; Figure 5 This is a schematic diagram comparing the receiver operating characteristic (ROC) curves of three graded tasks tested on the dataset in this invention. Detailed Implementation

[0027] See Figures 1 to 3 A multimodal medical imaging-based ATN (Advanced Myopic Neuropathy) grading system, comprising: The system image acquisition module is used to acquire retinal multimodal images to be classified, wherein the multimodal images include at least one of OCT images, OCTA images, and CFP images; The preprocessing module is used to preprocess the multimodal images to obtain an image sequence that meets the preset hierarchical processing requirements; The grading module is used to input the image sequence into a pre-stored multimodal multitask grading model, and to perform feature extraction, multimodal fusion and multitask classification on the image sequence through the multimodal multitask grading model, and output the lesion level prediction result for each grading task. The multimodal, multi-task hierarchical model includes three parallel feature extraction backbone networks, one multimodal fusion module, and three classification heads. The three feature extraction backbone networks correspond to three modalities: OCT, OCTA, and CFP images, respectively. The multimodal fusion module includes two sub-modules: intramodal differential fusion and cross-modal collaborative attention fusion. The three classification heads correspond to three hierarchical tasks: A-level, N-level, and T-level, respectively.

[0028] The feature extraction backbone network adopts the DINOv3-ViT-B / 16 architecture; the multimodal fusion module adopts a learnable weighted fusion strategy; and the multi-task classification head consists of three independent linear classifiers.

[0029] The working steps of the ATN hierarchical system are as follows: S1. Acquire image data: Acquire the retinal multimodal image to be classified; the multimodal image includes at least one of OCT image, OCTA image and CFP image; S2. Image preprocessing and anatomical structure localization: Perform basic preprocessing and anatomical localization on the multimodal images to obtain an image sequence that meets the preset hierarchical processing requirements; S2.1 Basic preprocessing: Normalize and standardize the images of each modality, and perform center cropping on the CFP image to retain 80% of the central region; The specific steps are as follows: The size of each modal image was adjusted, and all modal images were uniformly adjusted to 224×224 pixels; The adjusted image is normalized using the mean and standard deviation of ImageNet to ensure that the pixel values ​​are distributed within a specified range. Perform image validity verification on the normalized images, and remove or mark invalid images. The image sequence for each modality is fixed in number. For each patient sample, each modality m is allowed to contain a variable number of images, which we determine using the hyperparameter K. m They were standardized to a fixed number.

[0030] The specific strategy is: if the actual number of images N m >K m Then randomly sample K m Zhang; Ruo N m <K m Then, filling is done by repeated sampling.

[0031] This process can be formally represented as: ; in m is the original image set of mode m.

[0032] The processed modal image sequences are organized into a model input format.

[0033] S2.2 Anatomical structure localization: The key structures are segmented at the pixel level using the Swin-UNet model. The key structures include the retinal layer boundary, the fovea of ​​the macula, the vascular network, the non-perfusion area, the avascular area of ​​the macula, the optic disc, the macula, the hemorrhage area, and the exudation area. S2.3, Anatomical Attention Map Generation: Encode the probability map or boundary map obtained from segmentation into an anatomical attention map.

[0034] This step uses the Swin-UNet model pre-trained on the REFUGE, IDRID, and OCT2017 datasets to directly segment the input image without local fine-tuning.

[0035] The input image size is 224×224, and the output is a 7-layer segmentation map of the retinal boundary, fovea, vascular / non-perfusion area, optic disc / macula / hemorrhage / exudation. The segmentation map is converted into an attention map (0-1 value) and upsampled to 14×14 for subsequent weighting.

[0036] S3, Image Feature Extraction: The image sequence is input into three independent DINOv3 ViT-B / 16 networks to extract features from the images of the three modalities; if more than one modality is missing, proceed to S4; if all three modalities are present, proceed to S5. For two or more OCT and OCTA images, extract the [CLS] token and 14×14 patch feature sequence of the input image respectively; For a single CFP image, the patch features of the CFP image are spatially weighted using the anatomical attention map obtained in S2.3: the attention map is upsampled to 14×14 and then multiplied element-wise with the patch features to enhance the response of key regions.

[0037] The feature extraction backbone network adopts the DINOv3-ViT-B / 16 architecture.

[0038] For the input image First, it is divided into a series of non-overlapping image blocks {Pi}i=1N, where N=HW / P2 and P=16 is the block size.

[0039] Each image patch is linearly projected into an embedding vector: ; These image patch embeddings are concatenated with a learnable [CLS] token and positional encoding to form the input sequence for the Transformer: ; The sequence is processed through a series of Transformer encoding layers.

[0040] Each layer contains a multi-head self-attention (MSA) and multilayer perceptron (MLP) module, and applies layer normalization (LN) and residual connections: ; Finally, the feature vector corresponding to the [CLS] token is extracted from the output of the last layer. 768 is used as a global semantic representation of the input image.

[0041] S4. Modality Missing Handling: Utilize existing modality features to dynamically generate feature estimates for missing modalities through a predictor; the predictor consists of two MLP layers and cross-attention; S5. Multimodal feature fusion: Intramodal differential fusion and cross-modal collaborative attention fusion of features from three modalities; Based on the imaging characteristics of different modalities, this step designs differentiated fusion strategies to integrate the features of multiple images of the same modality into a single-modality feature vector.

[0042] Intramodal fusion: The two [CLS] tokens of OCT are obtained by attention pooling (weights are generated by a small network) to get f_oct; the four [CLS] tokens of OTA are obtained by weighting the SE module to get f_octa; the enhanced patch features of the CFP image are obtained by global average pooling to get f_cfp.

[0043] Cross-modal fusion: The patch feature sequences (14×14=196 tokens) of each modality are concatenated and input into a 6-layer Transformer encoder (768 hidden layers, 12 heads). The output [CLS] token is taken as the fusion feature f_fusion.

[0044] S6. Enhance lesion features: Enhance the expression of lesion region features by using lesion probability maps to obtain lesion enhancement features; Add an FPN segmentation head to the fusion feature to output a lesion probability map (5 categories: background, hemorrhage, exudation, non-perfusion area, and others). Multiply this map with the patch feature of OCTA and then perform global average pooling to obtain the lesion enhancement feature f_seg.

[0045] The multimodal fusion module employs a learnable weighted fusion strategy; Modal weight parameters are provided. The normalized contribution weights of each mode are obtained through the softmax function: The joint features of fused CFP images are represented as follows: ; Where h m The dynamics of the sample determine the result: if valid data exists for mode m of the sample, then h... m This is a characteristic of the mode; otherwise h m = t m This design enables the model to adaptively weigh the contributions of different modalities and is robust to missing data.

[0046] S7. Output results: Input the lesion enhancement features \( f_{seg} \) into three independent classification heads (fully connected layers), output the logits of A, N, and T levels, and obtain the probability of each category through softmax.

[0047] Each classification head maps the fused features (hfused) to the corresponding class space: ; Among them W g and b g These are learnable parameters, and the predicted probabilities for each category are ultimately obtained through the softmax function.

[0048] Screening conversion (optional): Based on clinical needs, the grading results can be converted into binary screening results (normal / lesion).

[0049] A normal condition is defined as A∈{0,1} with N=0 and T=0; otherwise, it is considered a lesion.

[0050] The rule-based or probability-based method can be used (calculate the joint probability of normality \( p_{\text{normal}} = P(A\in\{0,1\}) \cdot P(N=0) \cdot P(T=0) \)), and set a threshold θ for judgment.

[0051] Uncertainty estimation: During inference, enable MC Dropout (10 forward iterations) and calculate the mean and standard deviation of the predictions for each sample as confidence indicators.

[0052] Based on the confidence level, the screening results can be further divided into three levels: "automatic pass" (high confidence normal), "automatic referral" (high confidence lesion), and "manual review" (low confidence), thereby improving the reliability of the screening.

[0053] Output format: Outputs A, N, T classification results, probability of each category, normal probability, and confidence level for each patient, and can generate structured reports containing visualization information such as attention heatmaps and lesion segmentation overlay maps.

[0054] The training steps for the multimodal, multi-task hierarchical model are as follows: Step 1: Collect multimodal retinal images with A, N, and T grade annotations as training datasets. Divide patients into training and validation sets according to a preset ratio. The multimodal retinal images include OCT, OCTA, and CFP images. This step involves acquiring multimodal ophthalmic images of the patient, including OCT B-scan sequences (2-3 images per eye), OCTA en-face projections at different depths (4-6 images per eye), and CFP images (1 image per eye), and associating them with corresponding A (atrophy), N (neovascularization), and T (traction) grading labels.

[0055] By covering imaging modalities across three dimensions—structure, function, and overall picture—it lays the data foundation for subsequent fusion analysis; at the same time, it supports the inclusion of historical images from multiple time points, facilitating longitudinal disease tracking.

[0056] Step 2: The three DINOv3 ViT-B / 16 backbone networks are loaded with weights pre-trained on ImageNet, while other modules (cross-modal predictor, fusion Transformer, segmentation head, classification head, etc.) are initialized using Xavier. Step 3: Perform end-to-end fine-tuning on the DINOv3 ViT-B / 16 backbone network. All layer parameters participate in gradient updates, but differential learning rates are used to balance the pre-trained knowledge with the adaptation to new data. New modules use higher learning rates, while other modules use lower learning rates.

[0057] Step 4: Form a multimodal, multi-task hierarchical model based on the updated parameters.

[0058] Specifically, firstly, a large number of multimodal retinal images with A, N, and T hierarchical annotations are collected as a training dataset.

[0059] The dataset comes from multiple medical institutions, and the images have been annotated by professional ophthalmologists.

[0060] Five-fold cross-validation was performed based on patient ID, with the ratio of training set to validation set in each fold being approximately 80%:20%, ensuring that all images of the same patient appear only in the same dataset.

[0061] Secondly, the network model is trained by employing a multi-task joint loss function, with Focal Loss used for each task to mitigate the class imbalance problem. ; Where p i,c =softmax(sg) c Let y be the predicted probability that sample i belongs to class c. i,c For the true label of one-hot encoding, α c γ represents the class weights, and γ is the focusing parameter.

[0062] The total loss is the weighted sum of the losses from the three tasks: ; In terms of optimization strategy, a hierarchical differential learning rate is adopted.

[0063] Use a low learning rate for the backbone network Fine-tune; apply a learning rate to the A-class header. Use a higher learning rate for other classification heads. To quickly adapt to new tasks.

[0064] The optimizer chosen is AdamW, and its parameter update rules are as follows: ; Where m tand v t λ is the bias correction estimate for the first and second moments of the gradient, and λ is the weight decay coefficient.

[0065] Automatic Mixed Precision (AMP) was used during training to accelerate training and reduce memory usage. The batch size was 16, and the training lasted for a total of 200 epochs. The gradient clipping threshold was 1.0, and the model selection was based on performance on the internal validation set.

[0066] After training, the model is used to predict new samples. For multimodal images to be classified, after preprocessing, they are input into the model to obtain the prediction results for the three classification tasks of A, N, and T.

[0067] Figure 4 This is a visualization of the lesion recognition based on multimodal medical images in this invention. The figure shows the original images, attention overlay, and attention heatmap of different patient samples in three modalities: OCT, OCTA, and CFP.

[0068] As can be seen from the figure: (1) In OCT images, the model’s attention is mainly focused on areas of abnormal retinal structure, such as changes in the morphology of the macular region and disordered retinal layer structure. (2) In OCTA images, the attention area is clearly focused on the abnormal blood vessel area or the area with enhanced blood flow signal, which can effectively locate the neovascularization or abnormal blood flow area; (3) In CFP images, attention is mainly focused on lesion-related areas, such as areas of hemorrhage, exudation or atrophy, and these areas are highly consistent with clinically observed lesion areas.

[0069] Furthermore, by comparing the "original image - attention overlay map - attention heatmap", it can be found that the model of the present invention can adaptively focus on key pathological areas under different modalities and effectively suppress the interference of background areas, demonstrating good spatial localization ability and cross-modal consistency.

[0070] Figure 5 This is a schematic diagram of the receiver operating characteristic (ROC) curves of the multimodal, multi-task classification model on the test dataset. In the figure, the horizontal axis represents the false positive rate, and the vertical axis represents the true positive rate, corresponding to the classification performance of the three tasks: A-level, N-level, and T-level, respectively.

[0071] As can be seen from the figure: All three ROC curves show a clear shift towards the upper left corner, indicating that the model can achieve a high true positive rate with a low false positive rate, demonstrating excellent overall classification performance.

[0072] The area under the curve (AUC) of the model in this invention on the three hierarchical tasks are as follows: Grade A task: 0.977; N-level tasks: 0.983; T-level task: 0.985; The AUC values ​​are all close to 1, indicating that the model has a high discrimination ability.

[0073] Depend on Figure 5 The present invention can be further explained by the following advantages: Excellent overall classification performance: The ROC curves for all three tasks are close to the upper left corner, and the AUC is greater than 0.97, indicating that the model has high accuracy in different graded tasks.

[0074] Multi-task learning effectiveness: The model performed stably on the three tasks of A (atrophy), N (neovascularization), and T (traction), indicating that the multi-task joint learning strategy can effectively improve the model's comprehensive discrimination ability.

[0075] Multimodal fusion has significant advantages: A high AUC value indicates that the model can fully integrate structural and functional information from OCT, OCTA, and CFP images, thereby improving the classification accuracy.

[0076] Strong potential for clinical application: It maintains a low false positive rate while ensuring high sensitivity, making it suitable for clinical screening and auxiliary diagnostic scenarios.

[0077] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention.

[0078] Anyone skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention.

[0079] Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this invention shall still be covered by the claims of this invention.

Claims

1. A method for classifying myopic macular degeneration (ATN) based on multimodal medical imaging, characterized in that, Includes the following steps: S1. Acquire image data: Acquire the retinal multimodal image to be classified; the multimodal image includes at least one of OCT image, OCTA image and CFP image; S2. Image preprocessing and anatomical structure localization: Perform basic preprocessing and anatomical localization on the multimodal images to obtain an image sequence that meets the preset hierarchical processing requirements; S3. Image feature extraction: Input the image sequence into three independent visual Transformer networks to extract features from the images of the three modalities; if more than one modality is missing, proceed to S4; if all three modalities are present, proceed to S5. S4. Modality Missing Handling: Utilize existing modality features to dynamically generate feature estimates for missing modalities through a predictor; the predictor consists of two MLP layers and cross-attention; S5. Multimodal feature fusion: Intramodal differential fusion and cross-modal collaborative attention fusion of features from three modalities; S6. Enhance lesion features: Enhance the expression of lesion region features by using lesion probability maps to obtain lesion enhancement features; S7. Output results: Input the lesion enhancement features into three independent classification heads, and obtain the probabilities of each level (A, N, T) through softmax.

2. The ATN grading method as described in claim 1, characterized in that: S2 includes the following steps: S2.1 Basic preprocessing: Normalize and standardize the images of each modality, and perform center cropping on the CFP image to retain 80% of the central region; S2.2 Anatomical structure localization: The key structures are segmented at the pixel level using the Swin-UNet model. The key structures include the retinal layer boundary, the fovea of ​​the macula, the vascular network, the non-perfusion area, the avascular area of ​​the macula, the optic disc, the macula, the hemorrhage area, and the exudation area. S2.3, Anatomical Attention Map Generation: Encode the probability map or boundary map obtained from segmentation into an anatomical attention map.

3. The ATN grading method as described in claim 1, characterized in that: S3 includes the following steps: For two or more OCT and OCTA images, extract the [CLS] token and patch feature sequences from the input image respectively; For a single CFP image, the patch features of the CFP image are spatially weighted using the anatomical attention map obtained in S2.3: the attention map is upsampled to the same size as the patch grid and then multiplied element-wise with the patch features to enhance the response of key regions.

4. The ATN grading method as described in claim 1, characterized in that, In step S5, the intra-modal differential fusion steps are as follows: For the OCT modality: The [CLS] token features of each slice are fused through learnable attention pooling: a weighted sum of the features of each slice is calculated, with the weights generated by a small attention network based on the slice features; For the OCTA modality: The [CLS] token features of each layer image are input into the channel attention module: first, global average pooling is used to obtain the feature descriptions of each layer, then the weights of each layer are generated through two layers of MLP, and the weighted fusion is used to obtain the OCTA modality features; For CFP image modalities: the patch feature sequence is multiplied element-wise with the anatomical attention map obtained in S2.3 to obtain the enhanced patch features, which are then obtained by global average pooling to obtain the CFP image modal features; The steps of the cross-modal collaborative attention fusion are as follows: The enhanced patch feature sequences of the three modalities are concatenated and input into a shared cross-modal Transformer encoder. This encoder uses a self-attention mechanism to enable tokens from different modalities to interact with each other, learn cross-modal joint representations, and outputs the fused [CLS] token as the comprehensive feature \( f_{fusion} \).

5. The ATN grading method as described in claim 4, characterized in that, The specific steps of S5 are as follows: Based on the comprehensive feature \( f_{fusion} \), a lightweight segmentation head is introduced to perform pixel-level prediction of common lesions, generate a lesion probability map, multiply the lesion probability map with a feature map before cross-modal fusion, or re-weight the fusion feature by using the segmentation result as a weight through attention pooling to obtain the lesion enhancement feature.

6. A multimodal medical imaging-based ATN (Advanced Myopic Neuropathy) grading system, characterized in that, To implement the ATN grading method for myopic macular degeneration based on multimodal medical imaging according to any one of claims 1 to 5, comprising: System image acquisition module: used to acquire retinal multimodal images to be classified, wherein the multimodal images include at least one of OCT images, OCTA images and CFP images; Preprocessing module: used to preprocess the multimodal images to obtain image sequences that meet preset hierarchical processing requirements; The grading module is used to input the image sequence into a pre-stored multimodal multitask grading model, and to perform feature extraction, multimodal fusion and multitask classification on the image sequence through the multimodal multitask grading model, and output the lesion level prediction result for each grading task. The multimodal, multi-task hierarchical model includes three parallel feature extraction backbone networks, one multimodal fusion module, and three classification heads. The three feature extraction backbone networks correspond to three modalities: OCT, OCTA, and CFP images, respectively. The multimodal fusion module includes two sub-modules: intramodal differential fusion and cross-modal collaborative attention fusion. The three classification heads correspond to three hierarchical tasks: A-level, N-level, and T-level, respectively.

7. The ATN hierarchical system as described in claim 6, characterized in that: The feature extraction backbone network adopts the DINOv3-ViT-B / 16 architecture; the multimodal fusion module adopts a learnable weighted fusion strategy; and the multi-task classification head consists of three independent linear classifiers.

8. A multimodal, multi-task grading model for diagnosing myopic macular degeneration, characterized in that, The training steps for the model are as follows: Step 1: Collect multimodal retinal images with A, N, and T grade annotations as training datasets. Divide patients into training and validation sets according to a preset ratio. The multimodal retinal images include OCT, OCTA, and CFP images. Step 2: Load the weights pre-trained on ImageNet into the three DINOv3 ViT-B / 16 backbone networks, and initialize the remaining modules with Xavier. Step 3: Perform end-to-end fine-tuning on the DINOv3 ViT-B / 16 backbone network. All layer parameters participate in gradient updates. Differential learning rates are used. New modules use higher learning rates, while other modules use lower learning rates. Step 4: Form a multimodal, multi-task hierarchical model based on the updated parameters.

9. The hierarchical model as described in claim 8, characterized in that: In step three, the learning rate of the DINOv3 ViT-B / 16 backbone network is set to 1e -5 , and the learning rate of the cross-modal predictor, the fusion Transformer, the segmentation head, and the classification head is set to 1e -3 ; all modules use the AdamW optimizer with β1=0.9, β2=0.999, and weight decay 1e -4 .

10. The hierarchical model as described in claim 8, characterized in that: During training, 20% of the samples are randomly selected and one modality is randomly discarded. The [CLS] features of the remaining modalities are used to generate features of the discarded modality through a lightweight predictor consisting of two MLP layers and cross attention. At the same time, the L2 reconstruction loss between the generated features and the true features is calculated and jointly optimized with the main task loss, so that the network learns the intrinsic complementary relationship between modalities. During inference, if a modality is truly missing, the existing modal features are directly used to dynamically generate feature estimates of the missing modality through the predictor. During training, the segmentation task and the classification task are optimized in parallel: the segmentation loss (Dice + cross entropy) and the classification loss (Focal Loss) are jointly trained, so that the model explicitly learns the causal relationship between lesions and classification.

Images