Multi-mode retina disease intelligent auxiliary diagnosis system

Abstract

The invention provides a multi-modal retinal disease intelligent auxiliary diagnosis system, which comprises an image preprocessing module used for processing a binocular fundus image uploaded by a user; the disease classification module is used for extracting global features and local lesion details by adopting a dual-channel DINOv2 model, initializing weights in a data set through transfer learning, performing fine adjustment on the fundus image data set, and performing fundus multi-label classification; and the batch processing module is used for disassembling batch binocular eye fundus image processing tasks into independent sub-tasks based on a distributed task scheduling and dynamic resource allocation technology, and distributing the independent sub-tasks to a plurality of edge computing nodes for parallel processing. According to the method, a multi-label independent classifier is designed, eight independent three-layer MLP classifiers are adopted, and each classifier focuses on single pathological feature modeling. Through parameter space decoupling design, gradient conflicts among multiple labels are avoided, and accurate capture of heterogeneity pathologies such as diabetes microvascular leakage characteristics and glaucoma optic cup morphological parameters is ensured.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence diagnostic technology, specifically to an intelligent auxiliary diagnostic system for multimodal retinal diseases. Background Technology

[0002] Current intelligent diagnostic systems only support monocular image input, neglecting the symmetry and pathological correlation of binocular anatomy (such as the binocular microvascular leakage characteristics of diabetic retinopathy and the asymmetry of optic cup enlargement in glaucoma), resulting in insufficient diagnostic accuracy for comorbidities (such as diabetes + glaucoma). Mainstream algorithms (such as ResNet and EfficientNet) cannot model binocular correlation, leading to low sensitivity to symmetrical lesions. Furthermore, existing intelligent diagnostic systems only output disease labels, lacking interpretable decision-making basis (such as reasonable descriptions of the condition and diagnostic logic), requiring additional review by doctors and reducing efficiency. Summary of the Invention

[0003] To address the aforementioned problems, this invention proposes a multimodal intelligent auxiliary diagnostic system for retinal diseases. Based on a DINOv2 dual-stream architecture, it extracts global features from both eyes and quantifies the contribution of each eye through a dynamic weighting module, thereby improving the accuracy of identifying symmetrical lesions. The technical solution provided by this invention is as follows:

[0004] A multimodal intelligent assisted diagnostic system for retinal diseases, comprising:

[0005] The image preprocessing module is used to process user-uploaded binocular fundus images;

[0006] The disease classification module uses a dual-channel DINOv2 model to extract global features and local lesion details. It initializes weights in the dataset through transfer learning and fine-tunes them for the fundus image dataset to perform multi-label fundus classification.

[0007] The batch processing module, based on distributed task scheduling and dynamic resource allocation technology, decomposes the batch binocular fundus image processing task into independent sub-tasks and distributes them to multiple edge computing nodes for parallel processing.

[0008] The DINOv2 model is a teacher-student architecture consisting of parameter-shared visual Transformers, where the student network... The teacher network is updated via gradient descent. Updated using exponential moving average.

[0009] Preferably, the specific process of the fundus multi-label classification is as follows: patient The left and right eye images are DINOv2 extracts depth features respectively:

[0010]

[0011]

[0012] Feature fusion is performed using channel splicing: ;

[0013] Features are fused through independent classifier groups Perform multi-label prediction:

[0014]

[0015] in This is the Sigmoid function.

[0016] Preferably, the independent classifier group Each pathology category Corresponding independent three-layer MLP classifier:

[0017]

[0018]

[0019]

[0020] in , and For learnable parameters, , and For bias parameters, The activation function is used; the independent classifier group updates the parameters through decoupling, and each classifier adjusts its feature weights autonomously.

[0021] Preferably, a dynamic reweighted training strategy is used, the specific process of which is as follows: Let the training set... The Middle The number of samples in each class is Define the category weights as follows:

[0022]

[0023] Dynamic resampling is achieved through weighted random sampling:

[0024]

[0025] In the formula Indicates the first The probability that a sample will be sampled.

[0026] Preferably, in the fine-tuning process for multi-label classification of fundus images with small sample sizes, a progressive hierarchical optimization strategy is adopted: Let the backbone network parameters be... The classifier parameters are The optimization objective is to minimize the weighted cross-entropy loss:

[0027]

[0028] The AdamW optimizer is used to update parameters in a hierarchical manner:

[0029]

[0030]

[0031] Among them, the basic learning rate The stratification coefficient is , Weight decay .

[0032] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention designs a multi-label independent classifier, employing eight independent three-layer MLP classifiers to address the classification needs of seven types of eye diseases (diabetic retinopathy, glaucoma, etc.) and normal fundus, with each classifier focusing on modeling a single pathological feature. Through parameter space decoupling design, gradient conflicts between multiple labels are avoided, ensuring accurate capture of heterogeneous pathologies such as diabetic microvascular leakage features and glaucoma optic cup morphology parameters. Breaking through the limitations of traditional monocular image analysis, this invention combines geometric alignment and CLAHE enhancement techniques to effectively suppress illumination differences and field-of-view bias, improving model robustness. Attached Figure Description

[0033] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0034] Figure 1 This is a system framework diagram of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] To make the above-mentioned objectives, features and effects of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1: A multimodal intelligent auxiliary diagnostic system for retinal diseases, such as Figure 1As shown, it includes:

[0038] The image preprocessing module is used to process user-uploaded binocular fundus images. This includes brightness equalization using OpenCV, contrast adjustment using the CLAHE algorithm, automatic segmentation of the optic disc, retinal vessels, and lesion areas using the RetinaNet model, and spatial alignment of the binocular images using a feature point matching algorithm. The preprocessed images are standardized and cropped to 512×512 pixels. The image preprocessing module can solve the feature extraction problem caused by the heterogeneity of lesion shape, color, and size.

[0039] The disease classification module uses a dual-channel DINOv2 model to extract global features and local lesion details. It initializes weights in the dataset through transfer learning and fine-tunes them for the fundus image dataset (containing 7000+ training samples). It also uses the Focal Loss loss function to alleviate the class imbalance problem and achieve multi-label classification of the fundus.

[0040] The batch processing module, based on distributed task scheduling and dynamic resource allocation technology, uses RabbitMQ message queues to decompose batch binocular fundus image processing tasks into independent sub-tasks and distribute them to multiple edge computing nodes for parallel processing.

[0041] The DINOv2 model framework is a teacher-student architecture composed of parameter-shared Visual Transformers (ViTs), where the student network... The teacher network is updated via gradient descent. Then update using exponential moving average:

[0042]

[0043] in Ensure the smooth evolution of the teacher network's parameters to prevent rapid updates to the student network from causing instability in the supervision signal. Given an input fundus image... Data Augmentation Module Generate two local perspectives and a global perspective ViT-B / 8 divides the image into patch sequence:

[0044]

[0045] in Embed the matrix for patch. For learnable classification tokens, For position encoding, For the embedded dimension. After... The output features of the CLS token after processing by the Transformer block. via projection head Mapped to the unit sphere:

[0046]

[0047] in , For learnable parameters, and Here are the bias parameters. The output probability distributions of the teacher network and the student network are as follows:

[0048]

[0049]

[0050] In the formula and Controlling the sharpness of the distribution, centering the vector Updated via moving average:

[0051]

[0052] in It is the momentum coefficient, used for updating the centered vector via moving average. , Batch size, i.e., the number of samples processed in one training / computation session. Loss function. Minimize the KL divergence of the two view distributions :

[0053]

[0054] Compared to CNN models such as ResNet, DINOv2 exhibits three theoretical advantages in fundus image processing: First, ViT's global self-attention mechanism can model the long-range dependencies of retinal vessels:

[0055]

[0056] in , , , Given the input sequence, This mechanism directly establishes a connection between any two patches, overcoming the limitations of the local receptive field in CNNs. Secondly, self-supervised pre-training captures the essential features of fundus anatomy through implicit contrastive analysis, and its objective function can be reparameterized as follows:

[0057]

[0058] in Cosine similarity. Finally, the hierarchical structure of ViT is more suitable for handling multi-scale pathological features: superficial attention focuses on local lesions, while deep attention integrates global anatomical information.

[0059] In a multi-label fundus classification task, anatomical structure-aware feature interaction is achieved through geometrically aligned binocular image input. Given a patient... Left and right eye images DINOv2 extracts depth features respectively:

[0060]

[0061]

[0062] Feature fusion is achieved through channel splicing:

[0063]

[0064] The design is based on three medical priors: (1) symmetrical lesions (such as diabetic retinopathy) have mirror-image correspondence in the anatomical locations of both eyes, and the splicing operation preserves the spatial correspondence; (2) the feature independence of unilateral lesions (such as glaucoma) is not destroyed by the fusion process; and (3) the topological similarity of the vascular networks of both eyes can be naturally captured through linear combination of feature spaces. The fusion features are obtained through independent classifier groups. Perform multi-label prediction:

[0065]

[0066] in This is the Sigmoid function. Compared to fusion methods based on attention or correlation matrices, the computational complexity of this strategy is significantly lower. Reduce to In maintaining While enhancing the ability to represent features, it reduces the amount of floating-point operations.

[0067] In fundus multi-label classification tasks, independent classifier design achieves specific modeling of pathological features through parameter space decoupling. Given binocular fused features... Each pathology category Corresponding independent three-layer MLP classifier:

[0068]

[0069]

[0070]

[0071] in The function is a sigmoid function. This design stems from the heterogeneity of fundus pathological features: the decision boundary for diabetic retinopathy (D) primarily depends on microvascular leakage features, while the classification of glaucoma (G) depends on optic cup morphology parameters. When using a shared classification layer, the gradient update direction... Affected by the combined loss of all labels:

[0072]

[0073] This will lead to a distortion of the feature space, especially when there is a weak correlation between labels, such as the correlation coefficient between myopia and glaucoma being only 0.03. Independent classifiers update parameters by decoupling:

[0074]

[0075] Each classifier can autonomously adjust its feature weights. For example, a classifier for class D might enhance sensitivity to macular vessel density, while class H (hypertension) focuses on arterial stenosis features. Theoretical analysis shows that the independently designed VC dimension is... ( For a single classifier (VC dimension), compared to a shared design... Its model capacity is improved times ( Furthermore, parameter isolation avoids gradient conflicts, which is especially important during backpropagation.

[0076]

[0077] Independent architectures are particularly suitable for multi-label tasks with long-tailed distributions, allowing each classifier to independently adjust class balance without compromising the data distribution of other labels.

[0078] In multi-label classification tasks using highly imbalanced fundus data, the dynamic reweighting training strategy effectively mitigates class distribution skewness through adaptive sample space mapping. Let the training set... The Middle The number of samples in each class is Define the category weights as follows:

[0079]

[0080] The denominator term introduces a coefficient of 2 to ensure... Taking hypertensive retinopathy (Class H) as an example, when When the percentage is 2.3%, the positive sample weight negative sample weights By weight ratio This strategy amplifies the impact of minority classes. It achieves dynamic resampling through weighted random sampling.

[0081]

[0082] In the formula Indicates the first The probability of a sample being sampled. Compared to static oversampling, dynamic weighting has the following theoretical advantages: by replacing samples, it ensures that the extreme minority classes (such as class H) are fully accessed in each epoch; the adaptive adjustment mechanism can respond to changes in prediction confidence during training. For example, when the model's prediction for class O (accounting for 27.7%) tends to stabilize, its sample weights automatically decay as the loss decreases, releasing capacity to learn difficult samples such as class H. During backpropagation, the gradient of the loss function is recalibrated:

[0083]

[0084] in This is the batch size. This mechanism forces the optimizer to adjust the gradient norm of class H samples during parameter updates. Magnified 21.3 times (by (Calculation results show that) the geometric shape of the decision boundary is significantly improved in minority class regions.

[0085] In the fine-tuning process for multi-label fundus classification with small sample sizes, a progressive hierarchical optimization strategy achieves a balance between pre-trained knowledge retention and task-specific adaptation through differentiated parameter update rules. Let the backbone network parameters be... The classifier parameters are The optimization objective is to minimize the weighted cross-entropy loss:

[0086]

[0087] The AdamW optimizer is used to update parameters in a hierarchical manner:

[0088]

[0089]

[0090] The base learning rate is set as follows: The stratification coefficient is , Weight decay The design is based on three theoretical considerations: (1) the pre-trained parameters of DINOv2 have encoded a general representation of the retinal anatomy and need to be adjusted gently; (2) the classifier needs to adapt quickly to the decision boundary of multi-labels; and (3) the weight decay term prevents overparameterization in small sample scenarios.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal intelligent auxiliary diagnostic system for retinal diseases, characterized in that, include: The image preprocessing module is used to process user-uploaded binocular fundus images; The disease classification module uses a dual-channel DINOv2 model to extract global features and local lesion details. It initializes weights in the dataset through transfer learning and fine-tunes them for the fundus image dataset to perform multi-label fundus classification. The batch processing module, based on distributed task scheduling and dynamic resource allocation technology, decomposes the batch binocular fundus image processing task into independent sub-tasks and distributes them to multiple edge computing nodes for parallel processing. The DINOv2 model is a teacher-student architecture consisting of parameter-shared visual Transformers, where the student network... The teacher network is updated via gradient descent. Updated using exponential moving average.

2. The intelligent auxiliary diagnostic system for multimodal retinal diseases according to claim 1, characterized in that, The specific process of the fundus multi-label classification is as follows: Patient The left and right eye images are DINOv2 extracts depth features respectively: ； ； Feature fusion is performed using channel splicing: ; Features are fused through independent classifier groups Perform multi-label prediction: ； in This is the Sigmoid function.

3. The intelligent auxiliary diagnostic system for multimodal retinal diseases according to claim 2, characterized in that, The independent classifier group Each pathology category Corresponding independent three-layer MLP classifier: ； ； ； in , and For learnable parameters, , and For bias parameters, The activation function is used; the independent classifier group updates the parameters through decoupling, and each classifier adjusts its feature weights autonomously.

4. The intelligent auxiliary diagnostic system for multimodal retinal diseases according to claim 3, characterized in that, The dynamic reweighting training strategy is used, and the specific process is as follows: Let the training set... The Middle The number of samples in each class is Define the category weights as follows: ； Dynamic resampling is achieved through weighted random sampling: ； In the formula Indicates the first The probability that a sample will be sampled.

5. A multimodal intelligent auxiliary diagnostic system for retinal diseases according to any one of claims 1-4, characterized in that, In the fine-tuning process for multi-label classification of fundus images with small sample sizes, a progressive hierarchical optimization strategy is adopted: Let the backbone network parameters be... The classifier parameters are The optimization objective is to minimize the weighted cross-entropy loss: ； The AdamW optimizer is used to update parameters in a hierarchical manner: ； ； Among them, the basic learning rate The stratification coefficient is , Weight decay .

Images