Diabetic retinopathy detection system and device based on vision-language model

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a diabetic retinopathy detection system and device based on a vision-language model, and the system comprises an image obtaining unit, an image processing unit, an adjusting unit and an output unit. A large number of DR eye fundus images are collected, a corresponding diagnosis report or annotation text is added to each image, and an eye fundus image-diagnosis text multi-mode joint data set is constructed after preprocessing; an existing large vision-language pre-training model is adopted as the basis, the data set is combined, the LoRA low-rank adaptation technology is utilized, indexes such as reasoning accuracy serve as reward signals, and a group relative strategy optimization algorithm is adopted to conduct fine adjustment on the model, so that the model can learn image-text joint features of DR diagnosis; and deploying the fine-tuned intelligent prediction model, performing automatic prediction on the newly collected eye fundus image, and outputting a prediction result and corresponding explanation information to provide auxiliary decision support for doctors.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, specifically relating to a diabetic retinopathy detection system based on a vision-language model. Background Technology

[0002] Diabetic retinopathy (DR) is one of the most common ophthalmic complications of diabetes and a leading cause of blindness. Existing vision-based (image-based) methods still have several limitations: First, DR prediction and diagnosis rely not only on structural lesions in fundus images (such as microaneurysms, exudates, hemorrhages, drusen, etc.) but also frequently on the patient's clinical information (disease course, blood glucose control, comorbidities, etc.); simple image input may perform poorly in complex clinical situations or with low-quality images. Second, deep learning models are generally "black boxes," lacking sufficient clinical interpretability, hindering doctors' trust and acceptance of model conclusions. Third, high annotation costs, the subjectivity of annotation grading (consistency among different experts), and differences between real-world data distribution and research sets all affect the algorithm's generalization ability. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a diabetic retinopathy detection system based on a vision-language model. This system employs a large-scale vision-language model as its foundation, and improves detection accuracy and enhances the interpretability of prediction results by introducing a multimodal inference chain structure and reinforcement learning optimization strategies.

[0004] To achieve the above objectives, the technical solution is as follows: A vision-language model-based system for detecting diabetic retinopathy includes an image acquisition unit, an image processing unit, an adjustment unit, and an output unit. Image acquisition unit: Acquires image-text combined datasets, including color fundus images of the patient's retina and corresponding clinical description information; Image processing unit: Using a pre-trained visual-language base model as the prediction model, insert a prediction LoRA low-rank adapter into the language processing layer or image-language fusion module of the visual-language base model, and only update the parameters of the adapter; Adjustment Unit: The inference policy of the prediction model is optimized using the Group Relative Policy Optimization (GRPO) algorithm: multiple candidate prediction outputs are generated for each input, and the policy is updated through the reward function; Output unit: Outputs the inference result, which corresponds to the preset severity level of diabetic retinopathy.

[0005] Preferably, dataset acquisition includes the following: (1) Obtain data related to diabetic retinopathy from real patients, including at least the patient’s fundus predicted RGB color image and the corresponding diagnostic text or clinical examination report; (2) The collected fundus images are screened for quality and preprocessed, and the diagnostic text is cleaned and standardized. (3) The preprocessed fundus images are paired one by one with the corresponding diagnostic texts to form multimodal samples of image-text joint; (4) Based on clinical diagnostic rules and expert consensus, each sample was labeled as a grade I to VI diabetic retinopathy category. The grade includes: no diabetic retinopathy, mild nonproliferative diabetic retinopathy, moderate nonproliferative diabetic retinopathy, severe nonproliferative diabetic retinopathy, proliferative diabetic retinopathy, and diabetic retinopathy after photocoagulation. (5) Divide the dataset according to patient level to form training set and test set to avoid the same patient’s data appearing in different data subsets at the same time.

[0006] Preferably, the image processing unit processes the following content: A predictive LoRA low-rank adapter is inserted into the language processing module and image-language fusion module of the vision-language base model. Let the original weights of a certain linear layer be... LoRA prediction uses two low-rank matrices Prediction and This represents its correction term, where the prediction... The prediction is represented in low-rank dimension, and a scaling factor is introduced for prediction. The forward computation employs additive correction: ; LoRA prediction matrices A and B are initialized with small random values. The forward computation process of the model is as follows: Input encoding: The system receives fundus images. With clinical text prompts The image is processed by a visual encoder to extract features, and the text is converted into a vector sequence by a prediction embedding layer. Together, they constitute the input representation. ; Multi-level mapping: input The prediction is passed sequentially through several layers of the Transformer, and in each linear mapping layer, the original weights are... The forecast remains frozen, among which This represents the input feature dimension of the linear layer. Indicates the output feature dimension of the linear layer; LoRA prediction branch calculation: Predicting input Parallel input into the LoRA prediction adapter, predicting via a low-rank matrix. Prediction and Perform dimensionality reduction and dimensionality increase mappings, and multiply by a scaling factor. ; Final generation: After processing by all layers, the decoder layer outputs the predicted token sequence until a complete inference result is generated.

[0007] Preferably, the adjustment unit processes the following: For each system in the training set that receives fundus images... With clinical text prompts enter Based on the current strategy model conduct After random sampling, we obtain Candidate output sequences Each output contains a reasoning chain and a reasoning conclusion; define candidates. The category prediction is The correct predicted category is The reward function is designed using a binary approach. ; Calculate the average reward for the candidate group based on the reward values ​​of all candidate diagnostic outputs under the same input sample: ; Calculate the average reward for the candidate group. with standard deviation and define the first The relative advantage of the group output of each candidate diagnosis is ,in For a very small positive constant (such as The standard deviation is used to prevent division by zero errors caused by a standard deviation of 0 during the calculation process, thus ensuring numerical stability. The relative advantage is used to measure the degree of improvement of the candidate inference output relative to the overall level of the current candidate group.

[0008] Preferably, the objective function for policy updating is constructed based on the relative advantages of candidate groups. By weighting the advantages of each answer in the generated sequence and combining a truncation mechanism with reference policy constraints, the model parameters are updated, and the objective function predicts... for: ; in , This indicates that the reference strategy model is used to limit the update range of the current model; The KL divergence is used to measure the current policy. Reference Strategy The distribution differences between them prevent the model from deviating excessively from the initial language ability; This represents the hyperparameter coefficient of the KL divergence penalty term, used to adjust the penalty weight for the degree of deviation of the model.

[0009] Preferably, the output unit displays the reasoning steps wrapped in labels, and the reasoning includes the observation of image / text evidence, the corresponding medical judgment, and the inference derived therefrom; the predicted final reasoning is also displayed wrapped in labels, and the reasoning explicitly uses predefined six-level DR category labels for output, including no diabetic retinopathy, mild non-proliferative diabetic retinopathy, moderate non-proliferative diabetic retinopathy, severe non-proliferative diabetic retinopathy, proliferative diabetic retinopathy, and diabetic retinopathy after photocoagulation. Multiple results should not be confused within the same label.

[0010] Preferably, the output unit uses a standardized model output format to structurally express the reasoning process and the final reasoning result.

[0011] A vision-language model-based device for detecting diabetic retinopathy includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of this application when executing the executable instructions.

[0012] Compared with the prior art, the beneficial effects of this application are as follows: 1) This invention introduces a visual-language model to jointly model fundus images and medical text, and guides the model to perform explicit reasoning during the prediction process. This helps to fully explore the lesion features and semantic association information in fundus images, thereby improving the accuracy and stability of identifying different grades of diabetic retinopathy and reducing the risk of misjudgment and missed judgment.

[0013] 2) This invention introduces a prediction LoRA low-rank adaptation mechanism into the pre-trained vision-language basic model, which can achieve effective adaptation of the model to medical diagnosis tasks by only fine-tuning a small number of newly added parameters. This avoids the overall update of large-scale model parameters, significantly reduces the computing resources and storage overhead required for training, and facilitates deployment and promotion in actual medical scenarios.

[0014] 3) In the fine-tuning stage, this invention further adopts a reinforcement learning method based on group relative policy optimization (GRPO) to compare and optimize the multiple candidate diagnostic results generated by the model. The reward mechanism guides the model to improve the probability of generating correct diagnostic results, thereby improving the overall diagnostic performance while ensuring training stability.

[0015] 4) This invention standardizes the model output format, and expresses the reasoning process and the final diagnostic result in a structured manner, so that the model can clearly present its reasoning logic while giving a diagnostic conclusion, which helps doctors understand the basis of the model's decision-making. Attached Figure Description

[0016] Figure 1 This is a framework diagram of the application; Figure 2 This is a rendering of the application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Figure 1 As shown, a vision-language model-based diabetic retinopathy detection system includes an image acquisition unit, an image processing unit, an adjustment unit, and an output unit. Image acquisition unit: Acquires image-text combined datasets, including color fundus images of the patient's retina and corresponding clinical description information; Image processing unit: Using a pre-trained visual-language base model as the prediction model, insert a prediction LoRA low-rank adapter into the language processing layer or image-language fusion module of the visual-language base model, and only update the parameters of the adapter; Adjustment Unit: The inference policy of the model in S2 is optimized using the Group Relative Policy Optimization (GRPO) algorithm: multiple candidate prediction outputs are generated for each input, and the policy is updated through the reward function; Output unit: Outputs the inference result, which corresponds to the preset severity level of diabetic retinopathy.

[0019] This invention proposes a method for detecting diabetic retinopathy based on a vision-language model. This method uses a large-scale vision-language model as its foundation and improves the accuracy of diabetic retinopathy detection and enhances the interpretability of the judgment results by introducing a multimodal inference chain structure and reinforcement learning optimization strategy.

[0020] (1) Multimodal dataset construction phase: (1.1) Obtain real patient data related to diabetic retinopathy from a hospital with medical qualifications. The data includes at least the patient's fundus predicted RGB color image and the corresponding diagnostic text or clinical examination report. The diagnostic text may be an image description, medical history summary or auxiliary diagnostic explanation written by a doctor. (1.2) Perform quality screening and preprocessing on the acquired fundus images, including but not limited to resolution unification, size normalization and pixel value standardization; clean and standardize the diagnostic text to remove irrelevant symbols and information that can directly or indirectly identify the patient. (1.3) The preprocessed fundus images are paired one by one with the corresponding diagnostic texts to form multimodal samples of image-text joint; (1.4) Based on clinical diagnostic rules and expert consensus, each sample was labeled as a diabetic retinopathy category I to VI. The classification includes: no diabetic retinopathy, mild nonproliferative diabetic retinopathy, moderate nonproliferative diabetic retinopathy, severe nonproliferative diabetic retinopathy, proliferative diabetic retinopathy, and diabetic retinopathy after photocoagulation. (1.5) Divide the dataset according to patient level to form training set and test set, so as to avoid the same patient’s data appearing in different data subsets at the same time, thereby improving the objectivity of model evaluation results.

[0021] (2) Model initialization phase: prediction (2.1) Load the pre-trained visual-language basic model as the basis of the diagnostic model. The visual-language basic model has the ability to jointly model and semantically align image information and text information. In this embodiment, the prediction Qwen2.5-VL-7B-Instruct prediction is preferably used as the visual-language basic model. (2.2) Insert a predicted LoRA low-rank adapter into the language processing module and image-language fusion module of the vision-language basic model. Let the original weights of a certain linear layer be... LoRA prediction uses two low-rank matrices Prediction and This represents its correction term, where the prediction... The prediction is represented in low-rank dimension, and a scaling factor is introduced for prediction. The forward computation employs additive correction: ; Insertion locations include, but are not limited to, the prediction projection matrix for the Transformer, the interactive projection layer for Cross-attention, and the linear layer of the feedforward network. LoRA prediction matrices A and B are initialized with small random values; in a preferred embodiment, low-rank dimension prediction... The prediction is set to prediction 8, with a scaling factor for prediction. The prediction was set to 32. Experiments show that this parameter combination improves training efficiency while maintaining the model's generalization ability.

[0022] Overall, the forward computation process of the model is as follows: Input encoding: The prediction system receives fundus images. With clinical text prompts The image is processed by a visual encoder to extract features, and the text is converted into a vector sequence by a prediction embedding layer. Together, they constitute the input representation. ; Multi-layer mapping: predicting input The prediction is passed sequentially through several layers of the Transformer. In each linear mapping layer (e.g., predicting the projection matrix for Query, Key, and Value), the original weights... The forecast remains frozen; LoRA prediction branch calculation: Predicting input Parallel input into the LoRA prediction adapter, predicting via a low-rank matrix. Prediction and Perform dimensionality reduction and dimensionality increase mappings, and multiply by a scaling factor. ; Final generation: After all layers have processed the prediction, the decoder layer outputs the predicted token prediction sequence until a complete prediction inference result is generated.

[0023] GRPO forecast fine-tuning phase: (3.1) For each input prediction in the training set Based on the current strategy model conduct After random sampling, the prediction is obtained. Candidate output sequences Each output contains an inference chain and a prediction. The inference chain includes a detailed description of the observed pathological features, such as the presence of microaneurysms, hemorrhages, hard exudates, or fibrous proliferative membranes. Candidate predictions are defined. The predicted category is The reward function is designed using a binary approach. ; (3.2) Calculate the average reward of the candidate group based on the reward values ​​of all candidate prediction outputs under the same input sample: Calculate the average reward prediction for this candidate group. Prediction and Standard Deviation And define the prediction The relative advantage of the predicted output group is: The relative advantage is used to measure the degree of improvement of the candidate prediction output relative to the overall level of the current candidate group; (3.3) Based on the relative advantages of candidate groups, construct the objective function prediction for policy update. The model parameters are updated by weighting the advantages of each answer in the generated sequence and combining the truncation mechanism with reference strategy constraints.

[0024] .

[0025] (4) Deployment and Clinical Reasoning Stage: (4.1) Save the basic model parameters (frozen part) and the trained LoRA prediction parameters together as a loadable model package. Load it on the deployment side. and In resource-constrained scenarios, only the encoder can be deployed and a remote decoder can be used, or model pruning / quantization techniques can be employed.

[0026] (4.2) Input is , Among them, prediction Predicting from fundus images preprocessed in Stage 1, predicting The predicted text is clinical prompts. To ensure output stability, a fixed-temperature decoding strategy is used to control diversity.

[0027] (4.3) Reasoning steps <think> Prediction...prediction< / think> The label indicates that the reasoning includes observation of image / text evidence, corresponding medical judgments, and inferences derived therefrom. The language is concise and clearly structured, and vague or ambiguous expressions are prohibited. The final prediction is based on... <answer> Prediction...prediction< / answer> The label package shows that the prediction is explicitly output using a predefined six-level DR category label, including no diabetic retinopathy, mild non-proliferative diabetic retinopathy, moderate non-proliferative diabetic retinopathy, severe non-proliferative diabetic retinopathy, proliferative diabetic retinopathy, and diabetic retinopathy after photocoagulation. Multiple results should not be confused within the same label.

[0028] According to another aspect of this application, a diabetic retinopathy detection device is also provided. The diabetic retinopathy detection device according to an embodiment of this application includes a processor and a memory for storing processor-executable instructions. The processor is configured to implement the diabetic retinopathy detection method described in this application when executing the executable instructions.

[0029] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A system for detecting diabetic retinopathy based on a vision-language model, characterized in that, It includes an image acquisition unit, an image processing unit, an adjustment unit, and an output unit; Image acquisition unit: Acquires image-text combined datasets, including color fundus images of the patient's retina and corresponding clinical description information; Image processing unit: Using a pre-trained visual-language base model as the prediction model, insert a prediction LoRA low-rank adapter into the language processing layer or image-language fusion module of the visual-language base model, and only update the parameters of the adapter; Adjustment Unit: The inference policy of the prediction model is optimized using the Group Relative Policy Optimization (GRPO) algorithm: multiple candidate prediction outputs are generated for each input, and the policy is updated through the reward function; Output unit: Outputs the inference result, which corresponds to the preset severity level of diabetic retinopathy.

2. The diabetic retinopathy detection system based on a vision-language model according to claim 1, characterized in that, Dataset acquisition includes the following: (1) Obtain data related to diabetic retinopathy from real patients, including at least the patient’s fundus predicted RGB color image and the corresponding diagnostic text or clinical examination report; (2) The collected fundus images are screened for quality and preprocessed, and the diagnostic text is cleaned and standardized. (3) The preprocessed fundus images are paired one by one with the corresponding diagnostic texts to form multimodal samples of image-text joint; (4) Based on clinical diagnostic rules and expert consensus, each sample was labeled as a diabetic retinopathy category I to VI. The classification includes: no diabetic retinopathy, mild nonproliferative diabetic retinopathy, moderate nonproliferative diabetic retinopathy, severe nonproliferative diabetic retinopathy, proliferative diabetic retinopathy, and diabetic retinopathy after photocoagulation. (5) Divide the dataset according to patient level to form training set and test set to avoid the same patient’s data appearing in different data subsets at the same time.

3. The prediction system for diabetic retinopathy detection based on a vision-language model according to claim 1, characterized in that, The image processing unit processes the following: A predictive LoRA low-rank adapter is inserted into the language processing module and image-language fusion module of the vision-language base model. Let the original weights of a certain linear layer be... LoRA prediction uses two low-rank matrices Prediction and This represents its correction term, where the prediction... The prediction is represented in low-rank dimension, and a scaling factor is introduced for prediction. The forward computation employs additive correction: ； LoRA prediction matrices A and B are initialized with small random values. The forward computation process of the model is as follows: Input encoding: The system receives fundus images. With clinical text prompts The image is processed by a visual encoder to extract features, and the text is converted into a vector sequence by a prediction embedding layer. Together, they constitute the input representation. ; Multi-level mapping: Input The prediction is passed sequentially through several layers of the Transformer, and in each linear mapping layer, the original weights are... The forecast remains frozen, among which This represents the input feature dimension of the linear layer. Indicates the output feature dimension of the linear layer; LoRA prediction branch calculation: Predicting input Parallel input into the LoRA prediction adapter, predicting via a low-rank matrix. Prediction and Perform dimensionality reduction and dimensionality increase mappings, and multiply by a scaling factor. ; Final generation: After processing by all layers, the decoder layer outputs the predicted token sequence until a complete inference result is generated.

4. The method for detecting diabetic retinopathy based on a vision-language model according to claim 3, characterized in that, The adjustment unit processes the following: For each system in the training set that receives fundus images... With clinical text prompts enter Based on the current strategy model conduct After random sampling, we obtain Candidate output sequences Each output contains a reasoning chain and a reasoning conclusion; define candidates. The category prediction is The correct prediction result is The reward function is designed using a binary approach. ； Calculate the average reward for the candidate group based on the reward values ​​of all candidate diagnostic outputs under the same input sample: ; Calculate the average reward for the candidate group. with standard deviation and define the first The relative advantage of the group output of each candidate diagnosis is ,in It is a very small positive constant.

5. The method for detecting diabetic retinopathy based on a vision-language model according to claim 4, characterized in that, Based on the relative advantages of candidate groups, construct an objective function prediction for policy updates. By weighting the advantages of each answer in the generated sequence and combining a truncation mechanism with reference policy constraints, the model parameters are updated, and the objective function predicts... for: ； in , This indicates that the reference strategy model is used to limit the update range of the current model; The KL divergence is used to measure the current policy. Reference Strategy The differences in distribution between them; This represents the hyperparameter coefficient of the KL divergence penalty term.

6. The method for detecting diabetic retinopathy based on a vision-language model according to claim 1, characterized in that, Output Unit: The reasoning steps are displayed in labels, which include the observation of image / text evidence, the corresponding medical judgment, and the inferences derived therefrom; the predicted final reasoning is also displayed in labels, which explicitly use predefined six-level DR category labels for output, including no diabetic retinopathy, mild non-proliferative diabetic retinopathy, moderate non-proliferative diabetic retinopathy, severe non-proliferative diabetic retinopathy, proliferative diabetic retinopathy, and diabetic retinopathy after photocoagulation. Multiple results should not be confused within the same label.

7. The method for detecting diabetic retinopathy based on a vision-language model according to claim 1, wherein the output unit expresses the reasoning process and the final reasoning result in a structured manner by standardizing the model output format.

8. A device for detecting diabetic retinopathy based on a vision-language model, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of claims 1-7 when executing the executable instructions.

Images