Inference enhanced vision-language large model training and image processing method

Abstract

The invention relates to a reasoning enhanced vision-language large model training and image processing method, and the training method comprises the following steps: obtaining an ultra-wide-angle fundus image as an input image, and taking the manual annotation DR classification, the manual annotation lesion type and the clinical background of the ultra-wide-angle fundus image as cue words; utilizing a vision-language model with reasoning ability to generate reasoning enhanced image description and DR classification and lesion types obtained through reasoning; and taking the generated image description and the DR classification and lesion type obtained by reasoning as instructions, constructing a reasoning enhancement instruction data set in combination with the ultra-wide-angle fundus image, and finely adjusting the reasoning enhancement type vision-language large model. Compared with the prior art, the method has the advantages of being capable of effectively integrating clinical knowledge, high in recognition accuracy, high in interpretability and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a reasoning-enhanced vision-language large model training and image processing method. Background Art

[0002] Diabetic retinopathy (DR), a common and serious complication of diabetes, is one of the leading causes of blindness in adults. As the number of people with diabetes continues to rise globally, the need for early and accurate detection of diabetic retinopathy is becoming increasingly urgent. Fundus color photography is the primary diagnostic tool for diabetic retinopathy (DR). Traditionally, DR testing relies on manual interpretation of retinal images, such as fundus color photography. DR manifests as a series of lesions on the retina. Identifying the type and number of lesions based on clinical guidelines can further infer the severity of DR. However, this manual approach has numerous limitations. First, manual interpretation relies on the physician's professional experience and subjective judgment, resulting in significant variability in diagnostic results between physicians, making it difficult to ensure consistent diagnosis. Second, given the growing number of people with diabetes, manual interpretation is extremely inefficient and cannot meet the needs of large-scale screening. Furthermore, in areas with relatively scarce medical resources, the limited number of specialized ophthalmologists further exacerbates the difficulty of timely diagnosing diabetic retinopathy.

[0003] In recent years, with the rapid development of artificial intelligence technology, basic models trained using techniques such as self-supervised learning and visual language pre-training have gradually been applied to the field of retinal image analysis, providing new ideas and methods for the detection of diabetic retinopathy. These technologies aim to use large-scale data for pre-training to learn common image features and semantic information, which can then be transferred to specific downstream tasks to improve model performance and generalization capabilities. Currently, the main existing technical solutions are:

[0004] (1) Pre-trained models based on self-supervised learning: For example, RETFound uses a large number of unlabeled color fundus photography and optical coherence tomography images to pre-train masked autoencoders. Through this self-supervised pre-training method, the model can learn the potential features and structural information in the retinal images. In the subsequent diabetic retinopathy detection task, fine-tuning can be performed based on these pre-trained features to adapt to specific task requirements. However, this method often ignores the integration of multimodal data and the interpretability of model decisions when migrating the pre-trained model to a specific downstream classification task.

[0005] (2) Models using CLIP-like methods: KeepFIT and FLAIR use a CLIP (Contrastive Language-Image Pretraining)-like method to pretrain the visual encoder using paired retinal images and text for diabetic retinopathy detection tasks. This method improves the model's ability to recognize and diagnose retinal images by combining image and text information. However, when migrating to downstream classification tasks, the interpretability of the model's decision-making process is not fully considered. The model can only identify the type of lesions visible in the image and the grade to which it belongs. There is a risk that the co-occurrence relationship between lesions and grades does not comply with clinical guidelines, and its performance and reliability in practical applications are subject to certain limitations.

[0006] For example, CN117671422A discloses a method for constructing an ophthalmic large model based on fine-tuning of an ophthalmic vision model and an instruction set. The method includes constructing and training an ophthalmic vision model set, which includes an ophthalmic disease classification model and an ophthalmic lesion segmentation model; constructing an ophthalmic visual feature extraction and fusion module based on a pre-acquired visual text encoding module and a trained ophthalmic vision model set; constructing a large language model, and concatenating the ophthalmic visual feature extraction and fusion module with the large language model to obtain an ophthalmic large model; obtaining an ophthalmic instruction set, and fine-tuning the ophthalmic large model based on the ophthalmic instruction set to obtain the final trained ophthalmic large model.

[0007] However, the above existing methods have the following shortcomings: (1) Lack of clinical knowledge guidance: Clinical knowledge and guidelines are not fully integrated when generating image descriptions and instructions. In the diagnosis of diabetic retinopathy, the grading and identification of lesions need to be based on specific medical standards and clinical experience. However, this type of visual-language model based on image-text pairs cannot accurately incorporate this clinical knowledge into the model's decision-making process, so that the output results may be out of touch with actual clinical needs. For example, the correlation between the lesions and grading detected by the model may not be consistent with diagnostic guidelines and clinical experience. (2) Insufficient reasoning ability leads to poor interpretability of results: Existing visual-language models mainly focus on the matching and generation of images and texts, and have limited capabilities in handling complex medical reasoning tasks. It is difficult to clearly explain their decision-making basis and reasoning process to users. For example, in the detection of diabetic retinopathy, doctors cannot understand what features and logic the model uses to determine the existence and grading of lesions, which reduces doctors' trust in the model results and limits the widespread application of the model in clinical practice. Summary of the Invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and provide an inference-enhanced visual-language large model training and image processing method. The advantages of the visual-language model in combining visual understanding and natural language generation are utilized to realize the automatic generation of fundus image descriptions with inference logic, so that it can closely reflect the decision-making process of ophthalmologists; at the same time, the fundus images and the generated text are used to develop a better-performing inference-enhanced visual-language large model for DR detection, realize the effective integration of multimodal data, improve the model's recognition accuracy, interpretability and fit with clinical practice in DR grading and lesion identification tasks, and assist in more accurate and reliable medical diagnosis.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] The present invention aims to: (1) achieve effective integration of multimodal data. By designing prompt words embedded with clinical knowledge and guidelines, reasoning-enhanced image descriptions and explanations are generated based on DR grading and lesions, and multiple modal data (such as fundus images, medical text knowledge, etc.) are effectively integrated. When constructing a large reasoning-enhanced visual-language model for diabetic retinopathy detection, the synthetic reasoning-enhanced instruction text and fundus images are used for training, so that the model can fully utilize the advantages of multimodal data and improve its performance in diabetic retinopathy grading and lesion identification tasks. (2) Improve the interpretability of model decisions. Utilize the advanced visual understanding and language reasoning capabilities of the visual-language model to generate image descriptions and explanations that closely reflect the decision-making process of ophthalmologists. In this way, when the model performs diabetic retinopathy grading and lesion identification, its decision-making process and basis can be clearly presented, thereby enhancing the doctor's trust in the model results, improving the practicality and acceptability of the model in clinical practice, and assisting doctors in making more accurate and reliable diagnoses.

[0011] According to a first aspect of the present invention, a method for training a large inference-enhanced vision-language model is provided. The large inference-enhanced vision-language model is used for lesion recognition and DR grading in ultra-wide-angle fundus images. The method comprises the following steps:

[0012] An ultra-wide-angle fundus image is obtained as input. The manually annotated DR grade, manually annotated lesion type, and clinical context of the ultra-wide-angle fundus image are used as prompts. A visual-language model with reasoning capabilities is used to generate a reasoning-enhanced image description and the inferred DR grade and lesion type. The clinical context includes the grading standard and lesion explanation.

[0013] The generated image description and the inferred DR grade and lesion type are used as instructions, combined with the ultra-wide-angle fundus image to construct an inference-enhanced instruction dataset, and fine-tune the inference-enhanced vision-language large model.

[0014] As a preferred technical solution, the visual-language model performs three tasks of image description, lesion identification and DR grading in a clinical tone according to the prompt words. It adopts a structured and logic-driven format. It first performs a descriptive analysis based on the spatial position and appearance of clinically relevant visual features in ultra-wide-angle fundus images to generate image descriptions. Then, based on the image descriptions and predefined clinical contexts, lesion classification and severity grading are performed in sequence.

[0015] As a preferred technical solution, the instructions include one-stop instructions and sub-task instructions, wherein the one-stop instructions are a set of question-answer pairs corresponding to a certain ultra-wide-angle fundus image, the questions of the question-answer pairs are predefined comprehensive questions, and the answers are image descriptions generated by the visual-language model and the DR grades and lesion types obtained by reasoning; the sub-task instructions are three sets of question-answer pairs corresponding to a certain ultra-wide-angle fundus image, wherein the questions of the first set of question-answer pairs are predefined image description questions, and the answers are image descriptions generated by the visual-language model, the questions of the second set of question-answer pairs are predefined lesion classification questions, and the answers are lesion types obtained by reasoning the visual-language model, and the questions of the third set of question-answer pairs are predefined DR grading questions, and the answers are DR grades obtained by reasoning the visual-language model.

[0016] As an optimal technical solution, the reasoning-enhanced visual-language large model includes a visual encoder, a multi-layer perceptron, a text segmenter and a large language model. The ultra-wide-angle fundus image is input into a pre-trained visual encoder after image preprocessing to obtain a visual tag. The visual tag is mapped to the language tag space through a multi-layer perceptron to obtain a projected visual tag; the instruction is processed by the text segmenter to obtain a language tag; the language tag and the projected visual tag are connected to obtain an instruction tag, which is input into the pre-trained large language model.

[0017] As a preferred technical solution, the image preprocessing is specifically as follows:

[0018] Evenly dividing the ultra-wide-angle fundus image into a plurality of non-overlapping first partial images according to a preset layout;

[0019] The ultra-wide-angle fundus image is cropped according to a preset side length ratio with the centroid of the ultra-wide-angle fundus image as the center, to obtain a plurality of second partial images corresponding to different side length ratios;

[0020] The first partial image and the second partial image are resized to a uniform preset size and then spliced ​​together to serve as input of a visual encoder.

[0021] According to a second aspect of the present invention, there is provided an image processing method, the method comprising the following steps:

[0022] Acquire ultra-wide-angle fundus images;

[0023] The ultra-wide-angle fundus image and predefined questions are input into the reasoning-enhanced vision-language large model trained using the method described above, and the DR grade and lesion type are output.

[0024] According to a third aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the training method / image processing method when executing the program.

[0025] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the training method / image processing method is implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) Existing technologies lack effective integration of clinical knowledge. The present invention utilizes the advanced visual understanding and language reasoning capabilities of the visual-language model to achieve the automated generation of reasoning-enhanced image descriptions and instructions. By designing prompt words that embed clinical knowledge and guidelines related to DR grading, lesion identification, and their interrelationships, the generated descriptions and instructions can closely simulate the decision-making process of ophthalmologists. This makes the generated content more professional, logical, and clinically practical, and can more accurately reflect the reasoning process in medical diagnosis rather than a simple description of image content, which has greater advantages in the field of medical image analysis.

[0028] (2) In the DR detection task, other basic models have deficiencies in terms of grading accuracy, lesion recognition ability, interpretability of output results, and consistency with clinical practice. The reasoning-enhanced visual-language large model of the present invention is trained using ultra-wide-angle fundus images and synthetic reasoning-enhanced instructions. Compared with other basic models, the model shows comprehensive improvements in DR grading and lesion recognition tasks. Specifically, the present invention achieves higher recognition accuracy and can detect and grade lesions more accurately; it has better interpretability, making the decision-making process and basis of the model easier to understand; and it performs well in terms of consistency between the output results and clinical practice, and can more effectively assist doctors in making clinical decisions, meet the needs of clinical diagnosis, and is more valuable and reliable in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the training method of the present invention;

[0030] Figure 23 is a comparison chart of the prediction accuracy and consistency with clinical standards of the inference-enhanced visual-language large model of the present invention and the existing technology. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0033] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent.

[0034] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0035] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0036] Example 1

[0037] This embodiment provides a method for training a large inference-enhanced vision-language model. The large inference-enhanced vision-language model is used for lesion recognition and DR classification of ultra-wide-angle fundus images. Figure 1 As shown, the method includes the following steps:

[0038] Step 1) An ultra-wide-angle fundus image is obtained as the input image, and the manually annotated DR grade, manually annotated lesion type, and clinical background of the ultra-wide-angle fundus image are used as prompt words. A visual-language model with reasoning ability is used to generate an inference-enhanced image description and the inferred DR grade and lesion type.

[0039] To align DR diagnosis with clinical decision-making, this embodiment designs a vision-language model that generates clinically interpretable reports by simulating the two-step diagnostic process used by ophthalmologists: i) systematically identifying retinal abnormalities; ii) rule-based synthesis of findings into a standardized severity grade.

[0040] In this embodiment, the visual-language model uses GPT-4o. In other embodiments, any large model that supports visual-language multimodal input and has the ability to generate natural language can also be used to achieve the purpose of the invention.

[0041] In this step, GPT-4o leverages its advanced visual understanding and language reasoning capabilities to automatically generate reasoning-enhanced image descriptions and instructions. By designing a prompt that incorporates classification labels for DR lesion grading and identification, as well as clinical knowledge and guidelines describing the image characteristics and interrelationships of these categories, the generated image descriptions and instructions closely mimic the ophthalmologist's decision-making process, providing more professional and logical input for subsequent model training and application.

[0042] In one embodiment, the designed prompt words may be:

[0043] Imagine that you are a doctor.Analyze the ultra-wide-field fundusimage and perform the following tasks:1)Describe the image in detail.2)Given the image description, identify and explain any abnormalities from the list:..3)Given the abnormalities and the image,classify and explain the DRlevel..Doctors have observed microaneurysms on it and determine it to be MildNPDR, please pretend that you do not know that in advance. context:...(clinical criteria and explanations)..

[0044] In the above prompt words, microaneurysms and Mild NPDR are given labels, and (clinical criteria and explanations) represent the embedded clinical context. This embodiment prompts GPT-4o with detailed clinical requirements, so that it performs three tasks (image description, lesion identification, and DR classification) in a clinical tone. To ensure accuracy, the prompt template t(·,·,·) is for each fundus image u i Supplemented the grade g from the doctor's annotation i ∈{0,1,2,3,4} and lesion type l i∈{0,1} 7 , clinical background c (including grading standards and lesion explanations), etc., where lesion type 0 represents no lesion, 1 represents lesion, and the superscript 7 represents that there are 7 types of lesions, generating an average of 364.8 words of description r i =GTP4o(t(c,g i ,l i ),u i ), using a structured, logic-driven format, first performs descriptive analysis based on the spatial location and appearance of clinically relevant visual features in ultra-wide-angle fundus images to generate image descriptions, and then sequentially performs lesion classification and severity grading based on the image descriptions and predefined clinical context.

[0045] In one embodiment, the results generated by GPT-4o include three results: image description, lesion type, and DR grade. An example is as follows:

[0046] Image description:..The optic disc appears...The macula exhibits..Theperipheral retina..Small,round,red dots consistent with microaneurysms arevisible in the central to mid-peripheral retina.These are isolated and notwidespread.

[0047] Lesion identification:Microancurysm:Small,round,red dots observed on the retina.These are indicative of localized outpouchings of capillary walls,a hallmark of early diabetic retinopathy.

[0048] DR level classification:..(definition of Mild NPDR)..Based on the presence of microaneurysms and the absence of other abnormalities, the DRlevel can be classified as Mild NPDR,...(conclusion)...

[0049] Step 2) The generated image description and the inferred DR grade and lesion type are used as instructions, combined with the ultra-wide-angle fundus image to construct an inference-enhanced instruction dataset, and fine-tune the inference-enhanced vision-language large model.

[0050] Step 21) Build an inference enhancement instruction dataset

[0051] After generating image descriptions and reasoning results of lesion types and DR grades in step 1), a hybrid instruction dataset is constructed to fine-tune the reasoning-enhanced vision-language large model.

[0052] Instructions include one-stop instructions and task-specific instructions.

[0053] A one-stop instruction is a set of question-answer pairs corresponding to a certain ultra-wide-angle fundus image, which aims to provide a comprehensive task description for the model so that the model can give a complete analysis result based on the information of the entire image. For the i-th image, the corresponding one-stop instruction is recorded as (q,r i ), where q represents a one-stop instruction, r i represents the image description generated in step 1) and the DR grade and lesion type obtained by reasoning. The question in this question-answer pair is a predefined comprehensive question, for example:

[0054] Q:Describe this UWF image in detail.Provide the DR level of the imageand reason.

[0055] The answer is the image description generated by the vision-language model and the inferred DR grade and lesion type; that is:

[0056] A: ...(all results generated by the vision-language model)...

[0057] Since language models utilize previous contextual information when making incremental predictions, directly training the model to generate complete image descriptions in an autoregressive manner may result in error propagation. For example, incorrect identification of microaneurysms during lesion detection may seriously affect subsequent lesion classification, resulting in non-DR lesions being misidentified as mild DR lesions. Therefore, to reduce the impact of error propagation, the inference enhancement description content r of the i-th image is augmented. i Split into three sub-tasks, namely r i =r i1 ·r i2 ·r i3 "·" represents string concatenation, which is used to construct sub-task instructions, allowing the model to focus on specific tasks, reduce excessive dependence on previous steps, and enhance the ability to independently handle complex sub-tasks. The sub-task instructions are three sets of question-answer pairs corresponding to a certain ultra-wide-angle fundus image, denoted as (q i ,r ij ),j∈{1,2,3}, where

[0058] The first set of question-answer pairs consists of predefined image description questions, and the answers are image descriptions generated by the vision-language model, for example:

[0059] Q:Describe the findings on this UWF image.

[0060] A:....(image description)...

[0061] The second set of question-answer pairs consists of predefined lesion classification problems, and the answers are the lesion types inferred by the vision-language model, for example:

[0062] Q:Detect abnormalities on this UW'F image.

[0063] A:..(lesion identification)..

[0064] The third set of question-answer pairs consists of predefined DR grading questions, and the answers are DR grading obtained by visual-language model reasoning, for example:

[0065] Q:What is the DR level of this UWF image?

[0066] A:...(DR level classification)...

[0067] The final reasoning enhanced instruction dataset D consists of a mixture of two types of visual question answering (QA) pairs: one-stop instructions and task-specific instructions, namely Where N is the index set of the dataset, u i Represents the i-th image.

[0068] Step 22) Fine-tune the inference-enhanced vision-language model

[0069] First, a large, reasoning-enhanced vision-language model (UWF-VLMR) for DR detection is constructed based on a multimodal large model framework (e.g., InternVL). This model is fine-tuned and optimized using low-rank adaptation (LoRA) using ultra-wide-field fundus photography (UWF) and the reasoning-enhanced instructions generated in step 21). After training, the model can analyze the input fundus image and output the corresponding image feature description, detailed diagnostic reasoning process, DR grade, and lesion type diagnosis conclusion.

[0070] like Figure 1 As shown, the reasoning-enhanced visual-language large model includes a visual encoder, a multi-layer perceptron, a text segmenter and a large language model. The ultra-wide-angle fundus image is input into the pre-trained visual encoder after image preprocessing to obtain visual tags. The visual tags are mapped to the language tag space through the multi-layer perceptron to obtain projected visual tags; the instructions are processed by the text segmenter to obtain language tags; the language tags and the projected visual tags are connected to obtain instruction tags, which are input into the pre-trained large language model. In other embodiments, the large language model can be replaced by any open source large model that supports visual-language multimodal input, including InternVL, without affecting the realization of the purpose of the invention. During the training process of the reasoning-enhanced visual-language large model, the parameters of the visual encoder are frozen, and only the parameters of the multi-layer perceptron and the large language model are fine-tuned.

[0071] During the image preprocessing phase, a hybrid patch stitching strategy was designed to maximize the preservation of detailed information in ultra-wide-angle fundus images while taking into account key areas and global context. This strategy includes the following steps:

[0072] S1 , evenly dividing the ultra-wide-angle fundus image into a plurality of non-overlapping first partial images according to a preset layout; in this embodiment, the first partial images are arranged in a 4×3 layout, and a total of 12 first partial images are obtained.

[0073] S2, with the centroid of the ultra-wide-angle fundus image as the center, crop the ultra-wide-angle fundus image according to a preset side length ratio to obtain multiple second partial images corresponding to different side length ratios; in this embodiment, four square areas of different sizes are cropped around the centroid of the image, and their side lengths are 1.0 times, 0.75 times, 0.5 times, and 0.25 times the length of the short side of the original image, respectively, to obtain four second partial images.

[0074] S3, resizing the first partial image and the second partial image (a total of 16 image blocks) to a uniform preset size of 448×448 and then splicing them as input of the visual encoder.

[0075] In order to incorporate multimodal information, a pre-trained visual encoder is integrated into the backbone structure of the large language model and fine-tuned for 15 rounds using low-rank adaptation (LoRA). The training process is similar to the general LoRA fine-tuning process. The training data is input in the form of mini-batches. Each training step uniformly samples a batch of question-answer pairs (x) with corresponding ultra-wide angle (UWF) images from the constructed inference enhancement instruction dataset D. text ,y,x image )~Uniform(D) are the model inputs, where Uniform represents uniform sampling, x text represents the correct question, y represents the correct answer, and x imgge Represents the corresponding ultra-wide angle image. The visual encoder is frozen and a multi-layer perceptron (MLP) mapper M is used to map the visual tags to the language tag space. Subsequently, the language tag z is transformed into text and the projected visual marker z image Connect them to get the instruction tag z=[z text ,M(z image )]=[f tok (x text ),M(f ViT (x image ))], where f tok is a text tokenizer, f ViT is the visual encoder. Finally, these tokens are fed into a pre-trained large language model f with a low-rank adaptation (LoRA) module LLM The fine-tuning process is supervised by the cross entropy loss function.

[0076] Example 2

[0077] This embodiment provides an image processing method, which includes the following steps:

[0078] Acquire ultra-wide-angle fundus images;

[0079] The ultra-wide-angle fundus image and predefined questions are input into the reasoning-enhanced vision-language large model trained using the method described in Example 1 above, and the DR grade and lesion type are output.

[0080] Based on the above method, this example verifies the performance of the reasoning-enhanced vision-language large model proposed in the present invention as follows:

[0081] 1) Classification performance verification: This embodiment conducts DR grading and lesion classification performance experiments on ultra-wide-angle fundus image datasets. Figure 2 As shown, the vertical axis is the accuracy of DR grading. The results show that the present invention outperforms basic models such as RETFound, KeepFIT and FLAIR in both grading and lesion classification tasks. It is worth noting that the performance of GPT-4o with zero-shot learning is relatively low, indicating that it lacks prior knowledge about DR detection on ultra-wide-angle fundus images, and also shows that the performance improvement of the present invention does not come solely from the refinement of the GPT-4o model. At the same time, the results of reasoning using one-stop instructions and sub-task instructions are also reported. The difference between the two is not significant, indicating that the model is suitable for both one-stop DR diagnosis and sub-task DR recognition.

[0082] 2) Verification of consistency with clinical standards: The relevant methods were further evaluated by evaluating the DR grades and lesion prediction proportions that met the clinical standards, as well as the corresponding grading accuracy, such as Figure 2 The horizontal axis shows the proportion of DR grade and lesion predictions in the test set that met clinical standards. While baseline models such as RETFound for fundus images demonstrate strong performance, they do not fully encode the co-occurrence relationship between lesions and grade, resulting in 8.4%–20.9% of sample predictions being inconsistent with clinical standards. In contrast, our proposed method effectively captures these correlations, achieving 100% consistency with clinical standards in all test samples, achieving superior performance and the highest consistency accuracy among the compared methods.

[0083] In one embodiment, the electronic device includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). Various programs and data required for device operation can also be stored in the RAM. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0084] Many components in an electronic device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0085] The computing unit can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit performs the various methods and processes described above, such as image processing methods and / or model training methods. For example, in some embodiments, the image processing methods and / or model training methods can be implemented as computer software programs that are tangibly contained in machine-readable media, such as storage units. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of the image processing methods and / or model training methods described above can be performed. Alternatively, in other embodiments, the computing unit can be configured to perform image processing methods and / or model training methods in any other appropriate manner (e.g., by means of firmware).

[0086] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0087] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0088] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0089] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for training a large vision-language model with enhanced reasoning, characterized in that: The reasoning-enhanced vision-language large model is used to perform lesion recognition and DR grading in ultra-wide-angle fundus images, and the method includes the following steps: An ultra-wide-angle fundus image is obtained as input. The manually annotated DR grade, manually annotated lesion type, and clinical context of the ultra-wide-angle fundus image are used as prompts. A visual-language model with reasoning capabilities is used to generate a reasoning-enhanced image description and the inferred DR grade and lesion type. The clinical context includes the grading standard and lesion explanation. The generated image description and the inferred DR grade and lesion type are used as instructions, combined with the ultra-wide-angle fundus image to construct an inference-enhanced instruction dataset, and fine-tune the inference-enhanced vision-language large model.

2. The method for training a large-scale vision-language model with enhanced reasoning according to claim 1, wherein: The described visual-language model performs three tasks: image description, lesion identification, and DR grading in a clinical tone according to the prompt words. It adopts a structured, logic-driven format. It first performs a descriptive analysis based on the spatial position and appearance of clinically relevant visual features in ultra-wide-angle fundus images to generate image descriptions. Then, lesion classification and severity grading are performed in sequence based on the image descriptions and predefined clinical context.

3. The reasoning-enhanced vision-language large model training method according to claim 1, characterized in that: The instructions include one-stop instructions and sub-task instructions, wherein the one-stop instructions are a set of question-answer pairs corresponding to a certain ultra-wide-angle fundus image, the questions of the question-answer pairs are predefined comprehensive questions, and the answers are image descriptions generated by the visual-language model and the DR grades and lesion types obtained by reasoning; the sub-task instructions are three sets of question-answer pairs corresponding to a certain ultra-wide-angle fundus image, wherein the questions of the first set of question-answer pairs are predefined image description questions, and the answers are image descriptions generated by the visual-language model; the questions of the second set of question-answer pairs are predefined lesion classification questions, and the answers are lesion types obtained by reasoning the visual-language model; the questions of the third set of question-answer pairs are predefined DR grading questions, and the answers are DR grades obtained by reasoning the visual-language model.

4. The reasoning-enhanced vision-language large model training method according to claim 1, characterized in that: The inference-enhanced visual-language large model includes a visual encoder, a multi-layer perceptron, a text segmenter, and a large language model. The ultra-wide-angle fundus image is input into a pre-trained visual encoder after image preprocessing to obtain visual labels. The visual labels are mapped to a language label space through a multi-layer perceptron to obtain projected visual labels. The instruction is processed by a text word segmenter to obtain a language tag; The language tag and the projected visual tag are connected to obtain the instruction tag, which is input into the pre-trained large language model.

5. The method for training a large-scale vision-language model with enhanced reasoning according to claim 4, wherein: The image preprocessing is specifically as follows: Evenly dividing the ultra-wide-angle fundus image into a plurality of non-overlapping first partial images according to a preset layout; The ultra-wide-angle fundus image is cropped according to a preset side length ratio with the centroid of the ultra-wide-angle fundus image as the center, to obtain a plurality of second partial images corresponding to different side length ratios; The first partial image and the second partial image are resized to a uniform preset size and then spliced ​​together to serve as input of a visual encoder.

6. An image processing method, characterized in that: The method comprises the following steps: Acquire ultra-wide-angle fundus images; The ultra-wide-angle fundus image and predefined questions are input into the reasoning-enhanced vision-language large model trained using the method described in any one of claims 1 to 5, and the DR grade and lesion type are output.

7. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to claim 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to claim 6 is implemented.

Images