An inference-enhanced fundus reading large model training method, device, equipment and medium

Abstract

The application discloses a kind of inference enhanced fundus reading large model training method, device, equipment and medium, comprising: using the fundus reading dataset with explicit inference chain annotation to carry out supervised fine-tuning training to multiple modal base model and obtain basic model Fundus-base;Based on group relative strategy optimization GPRO reinforcement learning framework, and introduce the comprehensive reward mechanism of context consistency constraint, the basic model Fundus-base obtained by training is optimized, for constraint inference process and imaging features, the consistency of diagnosis and treatment knowledge, improve the medical credibility and explainability of inference chain.Therefore, the application can improve the training stability of fundus reading large model in reinforcement learning stage, solve the problems such as reward sparsity, insufficient discrimination, large learning signal fluctuation and training difficulty to converge caused by single reward of only relying on diagnostic accuracy in prior art, so that the model can continuously optimize inference ability under reliable learning signal.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image analysis technology, specifically to a reasoning-enhanced method, apparatus, equipment, and medium for training a large-scale fundus image reading model. Background Technology

[0002] Large-scale models for fundus image reading aim to perform various downstream tasks based on given retinal images, including detecting retinal lesions, locating lesion areas, and classifying diseases based on clinical knowledge. With the development of multimodal large-scale models in the field of general visual question answering, some studies have begun to explore their application in fundus image reading-assisted diagnosis. Existing large-scale models suitable for fundus image reading mostly employ a single-stage supervised fine-tuning (SFT) strategy, which involves fine-tuning the base model using visual question answering data based on fundus images to enable it to perform question answering and description based on fundus images.

[0003] Existing technologies disclose training on large-scale fundus image-text pairs and simulated doctor-patient dialogue data automatically generated from weakly labeled data; they also disclose methods that automatically convert pixel-level or lesion-level annotations into ophthalmic text descriptions and use large language models for semantic expansion and rewriting; and further disclose methods that pair fundus images with clinical reports and construct instruction data in the form of report generation and visual question-and-answer sessions. Compared to models that only output category labels, these methods can generate expert-style natural language explanations, but their training paradigm relies almost entirely on SFT. The language model is only optimized to mimic the target text, lacking explicit constraints on whether single-sentence reasoning truly corresponds to lesion evidence or meets clinical grading standards. Due to the widespread presence of weak annotations, automatic annotation conversions, and large model expansion content in the training data, the supervision signal provided by SFT is weak. While learning "expert expression," the model also learns imprecise or even erroneous reasoning patterns. The explanations generated by the model often fail to maintain strict consistency with actual imaging features and clinical guidelines, leading to problems such as unstable reasoning chains, insufficient interpretability, and poor clinical consistency.

[0004] In recent years, to improve the verifiability and reliability of general-purpose large-scale model outputs, existing technologies have proposed a reinforcement learning method based on verifiable rewards (RLVR). This method is applied to medical visual question answering and image understanding tasks, primarily involving medical imaging modalities such as MRI, CT, and some X-rays. It uses rewards based on answer correctness and output format conformity for post-training the model. The reward design in these works prioritizes the accuracy of the final answer and uses output format conformity as a secondary criterion. Therefore, the supervision signal generated by this reward primarily focuses on the correctness of the final output, lacking fine-grained and structured constraints on the intermediate inference chain. It neither mandates a one-to-one correspondence between the inference process and the lesion features presented in the image, nor considers whether the inference follows clinical diagnostic knowledge or guidelines. Therefore, even if the final answer is correct, the inference process generated by the model may still be inconsistent with real-world imaging evidence or clinical knowledge systems, making it difficult to fully guarantee the quality of the inference itself.

[0005] In summary, existing multimodal large model training methods based on fundus images have two implementation schemes: SFT and RLVR. However, they suffer from problems such as a lack of verification of the consistency between inference and images / guidelines, and insufficient attention to the quality of the inference chain and clinical consistency. Summary of the Invention

[0006] This invention aims to at least solve one of the technical problems existing in the prior art. Therefore, in response to the above-mentioned problems, the purpose of this invention is to provide a reasoning-enhanced method, apparatus, device, and medium for training a large-scale fundus image reading model. By jointly constraining diagnostic conclusions and reasoning processes, the generated reasoning chain is not only correct at the answer level, but also more consistent with real clinical practice in terms of lesion evidence citation and diagnostic and treatment knowledge logic.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0008] In a first aspect, the present invention provides a method for training a large-scale fundus image reading model with enhanced reasoning, comprising: supervising the training of a multimodal base model to obtain a base model Fundus-base by using a fundus image reading dataset with explicit inference chain annotations; optimizing the trained base model Fundus-base based on a reinforcement learning framework that optimizes GPRO using a group relative policy and introduces a comprehensive reward mechanism with context consistency constraints to obtain a large-scale fundus image reading model with enhanced reasoning, wherein the large-scale fundus image reading model with enhanced reasoning is used to output a structured inference chain and a final diagnostic result from the input retinal image to be analyzed and the corresponding question text.

[0009] Secondly, the present invention also provides a training device for a large-scale fundus image reading model with enhanced reasoning, comprising: a supervised fine-tuning training module configured to perform supervised fine-tuning training on a multimodal base model using a fundus image reading dataset with explicit inference chain annotations to obtain a base model Fundus-base; and a reinforcement learning module configured to optimize the trained base model Fundus-base using a reinforcement learning framework based on group relative policy optimization of GPRO and introducing a comprehensive reward mechanism with context consistency constraints to obtain a large-scale fundus image reading model with enhanced reasoning. This large-scale fundus image reading model with enhanced reasoning is used to output a structured inference chain and a final diagnostic result from the input retinal image to be analyzed and the corresponding question text.

[0010] Thirdly, the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the method described thereon.

[0011] Fourthly, the present invention also provides a computer-readable storage medium for storing one or more programs, characterized in that the one or more programs include computer instructions for causing a computer to perform the method.

[0012] Because the present invention adopts the above technical solution, it has the following characteristics: 1. When generating explicit chains of thought (CoTs), this invention overcomes the problem that traditional single-stage supervised fine-tuning (SFT) training lacks effective constraints on the reasoning process, avoids inconsistencies between reasoning steps and actual imaging features and clinical diagnostic knowledge, thereby improving the medical credibility and interpretability of the reasoning chain.

[0013] 2. This invention can improve the training stability of the fundus image reading model during the reinforcement learning stage, and solve the problems of reward sparsity, insufficient discrimination, large fluctuation of learning signal and difficulty in training convergence caused by the existing technology relying solely on the single reward of diagnostic accuracy. This enables the model to continuously optimize its reasoning ability under reliable learning signals.

[0014] 3. This invention constructs a reward mechanism designed specifically for the characteristics of fundus image reading tasks, overcoming the limitation that general medical reinforcement learning rewards cannot express the specific structured relationship between imaging features and fundus specialist knowledge. This solves the learning bias problem caused by the mismatch between rewards and task objectives, enabling rewards to effectively guide the model towards professional and accurate diagnostic reasoning.

[0015] In summary, this invention proposes a large-scale inference model training method that combines two-stage training with contextual consistency rewards for fundus image reading scenarios, which can be widely applied to the recognition of retinal images. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings: Figure 1 This is a schematic diagram of the large-scale model training method for fundus image reading according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the implementation of context consistency rewards in an embodiment of the present invention; Figure 3 The diagram illustrates that GRPO training with added contextual rewards in this embodiment of the invention exhibits faster convergence speed and higher steady-state reward level. Figure 4 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0017] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0018] To address the shortcomings of existing large-scale fundus image reading models that rely solely on single-stage supervised fine-tuning and lack sufficient connection between the reasoning process and imaging evidence and clinical knowledge, this invention provides a reasoning-enhanced training method, apparatus, device, and medium for large-scale fundus image reading models. Based on two-stage post-training plus contextual consistency rewards, this method enhances the medical credibility and interpretability of the reasoning chain.

[0019] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0020] Example 1: As Figure 1 As shown, the inference-enhanced fundus image reading large model training method provided in this embodiment includes setting a two-stage post-training process for inference enhancement. The two-stage post-training process for inference enhancement includes a supervised fine-tuning stage and a reinforcement learning stage executed sequentially. The specific process is as follows: S1, supervised fine-tuning stage.

[0021] In this embodiment, there is a supervised fine-tuning stage, including: 1) Construct a fundus image interpretation training dataset with explicit inference chain annotations. This involves annotating the intermediate inference steps followed by physicians during image interpretation, beyond the original fundus images and final diagnostic labels. These steps include locating key anatomical structures, identifying lesion features, associating evidence, and forming diagnostic conclusions. This allows the model to learn an interpretable inference path from visual cues to diagnostic conclusions. Each sample includes at least the following four tuples: (a fundus image; a question related to the image interpretation task; a reference answer; a structured inference chain). The inference chain explicitly annotates radiographic features (e.g., hemorrhage, exudation, cotton wool spots, neovascularization) and diagnostic knowledge (e.g., the relationship between feature combinations and disease classification and grading), organized in a step-by-step format. Example samples are shown in Table 1.

[0022] Table 1 Examples of Training Samples

[0023] 2) Based on the above sample set, supervised fine-tuning training is performed on the general pre-trained multimodal base model to enable the model to have the ability to read and reason about medical fundus images.

[0024] In this embodiment, a multimodal base model pre-trained on large-scale general image, text, or multimodal data is selected as the initial model. For example, the multimodal base model can be a vision-language pre-trained model or a multimodal large language model (such as LLaVA, Qwen-VL, InternVL, etc.). This type of model has general visual understanding and language generation capabilities, but it has not yet developed professional diagnostic knowledge and reasoning capabilities for fundus medical imaging scenarios. Based on the training sample set constructed above, the multimodal base model is subjected to supervised fine-tuning training. The training sample set includes fundus images, corresponding question descriptions, standard diagnostic answers, and explicit reasoning chain annotations, thereby guiding the model to learn the reasoning path from visual evidence to diagnostic conclusions in the medical image reading process while learning the diagnostic results. After training, when the model receives fundus images and corresponding question inputs, it can generate structured diagnostic results and their corresponding reasoning processes that conform to medical logic according to predefined output specifications. In the specific training process, each training sample is organized into an "input-target output" pair. The input includes at least fundus image data and corresponding text query information, while the target output includes disease classification or grading results and the corresponding stepwise inference text. By minimizing the supervised loss function between the model-generated output and the target output, the model is constrained to learn the mapping relationship between visual features, medical semantics, and diagnostic conclusions. Simultaneously, it learns the intermediate judgment and evidence integration steps in medical image interpretation, thereby enabling the model to acquire interpretable medical diagnostic reasoning capabilities, including the following: Capability 1: Output the analysis of image content in sequence; Ability 2: To reason by combining relevant medical knowledge; Capability 3: Provide the reasoning process and final diagnosis result in a predetermined format, as follows: <think> …reasoning process…< / think> <answer> …diagnostic results…< / answer> .

[0025] The basic model trained in this stage can output a structured and analyzable inference chain, and explicitly references imaging signs and diagnostic knowledge in the inference process, providing a structured information basis for reward calculation in the subsequent reinforcement learning stage. The above-mentioned basic model trained in this invention is named Fundus-base.

[0026] S2, Reinforcement Learning Phase.

[0027] In this embodiment, based on the Fundus-base model, this invention introduces a reinforcement learning framework based on Group Relative Policy Optimization (GRPO), and combines it with a validation reward mechanism to further optimize the model. The specific steps are as follows: (I) Copy the Fundus-base weights to the reference model Compared to reinforcement learning models (policy models) Among them: reference model The parameters of the reinforcement learning model remain fixed during training to provide a stable generation distribution as a regularization constraint. As the policy model to be optimized, its initial parameters are the same as those of Fundus-base, and are updated using the GRPO algorithm during subsequent training.

[0028] (II) For each input sample , by strategy model Perform group sampling to generate data in one go. 10 candidate answers: Among them, the input sample At least including: fundus image data (image); Text questions or instructions corresponding to diagnostic tasks. (question); Reference information used for verification (e.g., standard diagnostic labels, key sign annotations, or standardized medical knowledge). (answer). It should be noted that during the group sampling phase, the actual input to the policy model only includes fundus image data I and text question or instruction q. The model generates candidate answers based on the above input. Reference information 'a' does not participate in the model generation process, but is used for reward calculation and relative merit evaluation after candidate answers are generated. Each candidate answer... Each entry contains a complete reasoning chain text and a final reading conclusion, and conforms to a predefined output structure for subsequent reward evaluation.

[0029] (III) Calculate the comprehensive reward for each candidate answer. Within the candidate answer group corresponding to the same input sample, rank the candidate answers within the group to determine the relative merits of different candidates. Specifically, first, based on the comprehensive reward value of each candidate answer... The candidate set is sorted, with higher reward values ​​indicating higher overall quality of candidate responses. Based on this, various methods can be used to characterize the relative strengths and weaknesses of the candidate responses: Firstly, the relative advantage definition method based on within-group mean normalization measures relative performance by comparing the reward value of an individual candidate with the average reward value of the candidate group. ,in, Indicates the number of candidate answers, when When indicating candidate answers Better than the group average, when This indicates that it is relatively poor; Secondly: A relative advantage determination method based on ranking results, which involves ranking candidate answers from highest to lowest reward value and mapping the ranking to a relative advantage value. For example, let... Indicates candidate answers The ranking within the group (the smaller the value, the higher the ranking) can then be defined. To reflect the relative merits of different candidates: or ; Thirdly: The relative advantage determination method based on pairwise comparison measures relative merit by comparing the reward differences between candidate answers. For example, defining... Or adopt a smooth form ,in, For indicator functions, This is the Sigmoid function. Through the above ranking and relative advantage determination mechanism, high-quality and low-quality answers can be clearly distinguished within the candidate group, and transformed into relative optimization signals required for subsequent strategy optimization.

[0030] (IV) Strategy Model under the GRPO Framework The optimization objective is defined as follows: within the candidate answer group corresponding to the same input sample, maximize the log-likelihood weighted by the relative advantage within the group, and optionally introduce a reference model distribution as a regularization constraint to limit the policy update magnitude and improve training stability. Its objective function can be exemplarily expressed as:

[0031] in, Indicates the input sample. This represents the set of candidate answers generated by the policy model during the group sampling phase. For the corresponding candidate answers The relative advantage value within the group, For a reference model with fixed parameters, This represents the regularization weight coefficient. The first term in the objective function is the relative advantage-weighted policy optimization term, used to adjust the generation probability of the policy model based on the relative superiority or inferiority of candidate answers within the group: when a candidate answer has a large positive relative advantage, its corresponding log-likelihood term weight increases, thereby increasing the probability of such high-quality answers being generated subsequently; when the relative advantage of a candidate answer is negative, its log-likelihood term weight decreases accordingly, thereby suppressing the generation of low-quality or inconsistent answers. Through this mechanism, the model does not need to rely on the absolute reward scale, but rather performs stable optimization based on the relative comparison results between candidates. The second term in the objective function is the KL divergence regularization constraint term, used to measure the policy model. Compared with the reference model The output distribution differs under the same input conditions. By weighting and penalizing this term, the policy model can be effectively limited from deviating too much from the original distribution during the reinforcement learning stage, avoiding problems such as language degradation, distortion of medical expressions, or unstable generation. In some embodiments, when training stability has been ensured through other means, the regularization term may not be introduced, and the policy model may be updated only based on the relative advantage term. The policy model parameters are iteratively updated by performing gradient optimization on the above objective function. During multiple rounds of reinforcement learning training, the model gradually learns to favor generating answers with higher relative advantages, reasoning processes that are more consistent with medical imaging evidence, and diagnostic conclusions that are more clinically reliable within the candidate group, thereby achieving continuous optimization of medical reasoning ability.

[0032] Through the above two-stage training, this invention, while retaining the general capabilities of the base model, specifically improves the quality and clinical interpretability of the inference chain in fundus image reading scenarios.

[0033] Furthermore, a comprehensive reward mechanism including context consistency constraints is set up during the reinforcement learning phase.

[0034] In this embodiment, to address the problem that existing technologies, which only constrain the final diagnostic result during the reinforcement learning phase, result in a lack of medical consistency in the inference chain, this embodiment proposes a comprehensive reward mechanism that includes contextual consistency constraints. This reward mechanism applies to the output sequence generated by the model, simultaneously constraining the final diagnostic result, output format, and clinical consistency of the inference chain, thereby ensuring that the model obtains more stable, reasonable, and medically relevant optimization signals during the reinforcement learning phase.

[0035] Comprehensive Rewards It can be represented as: ,in, Rewards are given for accurate answers. Rewards for conforming to standardized formats This represents a context-consistency reward. These are the corresponding weight coefficients, which can be set and adjusted according to specific task requirements, training phase, or model stability requirements, where: 1) Reward for accurate answers: In this embodiment, the accuracy reward is used to determine whether the final interpretation conclusion output by the model is consistent with the reference answer. Let the input sample be... The complete response text generated by the model is The final diagnostic conclusion extracted from it is denoted as The corresponding reference answer is denoted as In one alternative implementation, the final diagnostic conclusion field is first parsed from the model's output text. If the output conforms to a predefined structured format, the diagnostic result is preferentially extracted from the conclusion field, for example, from... <answer> ...< / answer> Extract content from label pairs. When no conclusion field is detected or the label structure is incomplete, the entire text output by the model or its last paragraph can be used as the final diagnostic conclusion. By employing the above methods, we ensure that diagnostic conclusions for consistency determination can be obtained under different output format conditions. Model output conclusions. Compared with the reference answer Consistency determination can be achieved by combining one or more of the following methods.

[0036] (a) Complete match determination: When the disease category or classification result output by the model is completely consistent with the reference answer at the medical semantic level, it satisfies the condition. Then the model output is determined to be correct.

[0037] (b) Equivalence mapping determination: Pre-construct disease label mapping function:

[0038] in, This represents the set of diagnostic statements that the model may output. This represents a standardized set of diagnostic labels. This mapping function unifies different but semantically equivalent medical expressions (including synonymous diagnostic names and hierarchical descriptions under different naming conventions) to the same standard label. If the following conditions are met... Then it is also determined that the model output is consistent with the reference answer.

[0039] Based on the above consistency determination results, an accuracy-based reward will be given. It can be defined as a binary reward function:

[0040] The accuracy reward is the dominant optimization signal in the reinforcement learning process. It directly reflects the improvement of the model's diagnostic ability, is used to directly measure the correctness of the model's final image interpretation conclusion, and reflects the overall performance improvement of the model in medical fundus diagnosis tasks.

[0041] 2) Formatting compliance reward: In this embodiment, the format conformity reward is used to constrain whether the model output meets predefined structured format requirements. By explicitly constraining the output structure, the consistency and integrity of the generated content can be ensured, thereby facilitating subsequent automatic parsing, result verification, and streamlined processing. A structured format includes at least the following basic components: (a) Inference chain field: used to represent the intermediate inference process of the model from imaging evidence to diagnostic conclusion; (b) Conclusion field: used to represent the final reading conclusion or diagnosis result given by the model.

[0042] To achieve the above-mentioned structured constraints, this invention does not limit the specific format, and several optional exemplary implementation methods are given below: (a) Structured format based on explicit tag pairs: In this embodiment, the model output must satisfy the explicit label pair structure, for example: <think> ...< / think> and <answer> ...< / answer> Tag pairs. Among them, ` <think> ...< / think> The ` tag is used to wrap the complete inference chain text,` <answer> ...< / answer> The tags are used to encapsulate the final diagnostic conclusion. When the model output contains both of these tag pairs, and the tag structure is closed and the nesting relationship is correct, it is considered to meet the format specification requirements.

[0043] (ii) Format constraints based on field-based text structure: In this embodiment, the model output adopts a field-based text structure, which does not rely on specific symbol labels, but is distinguished by clear field identifiers. For example: Reasoning process: 1. Anatomical structure localization; 2. Imaging feature analysis; 3. Evidence integration; 4. Diagnostic conclusion: Severe non-proliferative diabetic retinopathy. Under this implementation, the format standardization is determined based on one or a combination of the following rules: (a) whether it contains predefined field titles (such as "reasoning process" or "diagnostic conclusion"); (b) whether the field order conforms to preset logic; (c) whether each field contains non-empty valid content.

[0044] (III) Output constraints based on structured data format: In this embodiment, the model output can be represented in a structured data format, such as JSON, XML, or other key-value pair formats, for example: { "reasoning": "Describes the inference chain process of the model", "diagnosis": "The corresponding final diagnosis result" } In this implementation, the output data structure is parsed and key fields are checked for existence, correct field types, and completeness of field content to determine whether the model output meets the format specification requirements.

[0045] A positive format conformance reward is given when the model output meets any predefined structured format requirement; no reward or penalty is given when the output is missing key fields, has unclosed fields, is structurally chaotic, or cannot be automatically parsed. This format conformance reward mechanism effectively guides the model to consistently produce well-structured, parsable, and reusable diagnostic and inference outputs during the reinforcement learning phase, thereby improving the overall system's engineering usability and clinical deployment value. The aforementioned structured formats and their determination methods are illustrative. In one possible implementation, the format conformance reward can be set using a binary reward mechanism: a positive reward (e.g., 1) is given when the model output fully conforms to the predefined structured format requirements; no reward is given or it is 0 when the model output does not meet the format requirements. This is just one example, but not limited to this; the output structure and corresponding format conformance reward mechanism can be equivalently replaced, adjusted, or combined according to specific application scenarios, system interface specifications, or downstream processing requirements.

[0046] 3) Context consistency reward.

[0047] In this embodiment, the consistency between the intermediate inference chain and the imaging features and related diagnostic knowledge presented in the image is measured. Unlike existing technologies that only constrain the final diagnostic result, this invention applies explicit supervision to the intermediate inference steps of the model, ensuring that the inference chain matches the actual fundus images and clinical knowledge system in both content and logic, thereby improving the interpretability, reliability, and clinical applicability of the model's inference process. Based on the correctness of the model's image interpretation conclusion, this invention employs a bi-branch strategy to calculate the context consistency reward, as follows: Scenario 1: The model's interpretation conclusion is the same as the reference answer, used to obtain the image consistency score. When the model's final interpretation conclusion is consistent with the reference answer, this embodiment focuses on examining whether the inference chain generated by the model accurately, sufficiently, and contradictorily covers key radiological features: (a) Identification and representation of key radiographic features; In this embodiment, for each input sample, a set of key radiographic features corresponding to the reference answer is pre-determined, denoted as: Each imaging feature This refers to a medical sign that can be observed or obtained through annotation in fundus images, such as microaneurysms, number of hemorrhage quadrants, beaded veins, intraretinal microvascular anomalies (IRMA), hard exudates, or subretinal fluid. The feature set can be derived from human expert annotations, automated detection model outputs, or a combination of both.

[0048] (b) Extraction of imaging features in the inference chain; In this embodiment, medical entities and attributes are extracted from the inference chain text generated by the model to obtain the set of imaging features explicitly or implicitly mentioned by the model during the inference process, denoted as: The extraction process can be assisted by rule matching, medical dictionaries, sequence labeling models, or large language models.

[0049] (c) Quantitative indicators of matching degree; Based on the two feature sets mentioned above, the following quantitative indicators are defined: (1) Feature Coverage: Used to measure the proportion of key radiographic features correctly covered in the model's inference chain.

[0050] (2) Feature Conflict Rate: in, This indicates that the inference chain contains feature descriptions that clearly contradict the facts of the image or reference annotations, such as denying the existence of lesions in the image or fabricating key signs that do not exist.

[0051] (d) Determining the reward tiers: Based on the above metrics, the context consistency reward can be defined in tiers as follows:

[0052] in, and For coverage threshold parameters (e.g.) The above design provides a positive reward when the reasoning chain completely and accurately describes the key lesion; no reward when it only partially covers or describes it vaguely; and a negative reward when the reasoning content clearly contradicts the image features, thereby effectively suppressing the model from generating fictitious or misleading reasoning.

[0053] Scenario 2: The model's interpretation conclusion differs from the reference answer, and is used to obtain a reasoning consistency score.

[0054] In this embodiment, it is no longer mandatory for the reasoning chain to cover specific imaging features. Instead, the examination focuses on whether the reasoning process has reasonable clinical diagnostic logic, such as starting from symptoms or signs and gradually conducting etiological analysis or differential diagnosis. (a) Whether it reflects a reasonable differential diagnosis approach, even if the final conclusion is wrong, the analysis process is in line with medical common sense; (b) Whether inferences that clearly contradict medical knowledge are avoided, such as skipping steps to conclusions without evidence or contradictory analyses.

[0055] Use a large model combined with RAG medical reference materials (such as EYE wiki). If the reasoning demonstrates an analytical path or differential diagnosis approach that aligns with common medical knowledge, a small positive reward can be given (for example, approximately 1 / 5 of the total reward is defined as a small positive reward); if the reasoning clearly deviates from clinical logic, no reward will be given.

[0056] Furthermore, to achieve the aforementioned context consistency reward, the present invention can be implemented in various ways, for example: (i) Rule-based matching implementation scheme.

[0057] (I) Construction of imaging feature database and disease knowledge rule set.

[0058] The imaging feature library is used to define "diagnostic imaging elements that can be referenced in the inference chain." The feature library is stored in structured record form, with each record containing at least: (a) feature identifier (fid); (b) standard name (canonical_name); (c) a set of synonyms (synonyms); (d) applicable modality (e.g., fundus / OCT / FFA / OCTA / UFW); (e) attribute constraints (attr_schema) (e.g., laterality, number, size, morphology, distribution range); and (f) anatomical location (anatomy_region). The feature library construction process includes: (a) compiling a candidate feature set from clinical guidelines, textbooks, and report templates; (b) merging synonyms and establishing fid mappings; and (c) generating matching templates for each feature, including keyword templates (synonyms) and pattern templates (regular expressions / fixed phrases), for subsequent extraction in the inference chain.

[0059] The disease knowledge rule set is used to express the deterministic relationship between diseases and imaging features. For each disease d, a rule entry is established, containing at least: (a) a set of necessary features Req(d): their presence / satisfaction supports the disease; (b) a set of excluded features Exc(d): their presence conflicts with the disease; (c) a set of optional features Opt(d): their presence enhances confidence; and (d) a set of relational constraints Rel(d): relationships that must be satisfied between features or between features and anatomical locations (e.g., "hemoptysis located_in macular region"). The rule entries are stored in an executable form, such as a set and weight table: w_req=1.0, w_opt=0.5, w_exc=1.0, w_rel=1.0 (exemplary values, which can be fixed as system parameters). The rule version number for each disease is also fixed to ensure consistency between training and reproduction.

[0060] (II) Extraction of entities and relations in the reasoning chain.

[0061] In this embodiment, the inference chain text output by the model is normalized (case-sensitive, synonym-based, unit-based, and negative word recognition), and extraction is performed according to the feature library template: (a) Entity extraction: The matched feature set is denoted as F_pred, where each feature is accompanied by polarity ∈ {POS, NEG} (e.g., “no bleeding” → bleeding is NEG); (b) Relation extraction: The matched relation triple set is denoted as R_pred (e.g., “exudation located_in macular region”).

[0062] (III) Calculation of indicators (hit, conflict, similarity).

[0063] In this embodiment, given that the model outputs a diagnostic conclusion of y_hat and the true / reference diagnosis is y, the target disease entry d=y_hat (for consistency checking) or d=y (for local rationality checking in error cases) is first retrieved from the rule set, and then calculated: (a) Number of hits (supporting factors): ; (b) Number of conflicts (exclusions): ; (c) Similarity (using Jaccard as an example): in .

[0064] One possible calculation method is the linear combination consensus score: ,in, , .

[0065] (IV) Reward determination, including: (1) Contextual consistency reward when the diagnosis result is correct: When y_hat = y, the reward only reflects whether the inference chain covers the key information in the input and is consistent with the disease rule. The reward value is limited to ({-0.2, 0, 0.2}) and is determined as follows: 1) High Consistency (Reward 0.2): Satisfies all of the following conditions: (a) Conf = 0; (b) Hit ≥ T_hit_high; (c) Sim ≥ T_sim_high. One possible threshold: T_hit_high = |Req(d)| + 1, T_sim_high = 0.5. Medium Consistency (Reward 0): Satisfies one of the following conditions: (a) Conf = 0 and Hit ≥ T_hit_mid; or (b) Sim ≥ T_sim_mid. One possible threshold: T_hit_mid = |Req(d)|, T_sim_mid = 0.3.

[0066] 2) Low consistency (penalty -0.2): For the remaining cases other than the above two types, that is, there are obvious omissions / misquotes: (a) Conf ≥ 1; or (b) Hit < T_hit_mid and Sim < T_sim_mid. Local rationality reward when the diagnostic result is incorrect.

[0067] (2) Context consistency reward when the diagnostic result is incorrect.

[0068] When y_hat ≠ y, this embodiment does not reward its overall conclusion, but evaluates whether the reasoning chain contains "an objective description that is locally reasonable for the input image". At this time, the reward value is limited to ({0, 0.1}) and is determined as follows: Having local rationality (reward 0.1): Meeting the following conditions: (a) Hit_true ≥ T_hit_local calculated with the true disease entry d = y; (b) and Conf_true ≤ T_conf_local. A possible threshold: T_hit_local = 1 (at least hitting one true relevant feature / relationship), T_conf_local = 0. Not having local rationality (reward 0): Not meeting the above conditions, that is, the reasoning chain does not reflect any key description consistent with the input, or there are feature references that strongly conflict with the true disease type.

[0069] Taking Diabetic Retinopathy (DR for short) as an example below, it illustrates how to perform entity extraction on the reasoning chain (CoT) and calculate the hit, conflict, and similarity metrics based on a pre-constructed imaging feature library and disease rule entries, so as to determine the discrete reward value. The following content is for illustrative purposes to clarify the implementable calculation process and value-taking method.

[0070] 1) Symbol and rule item definition: Let the true diagnosis of a certain case be Diabetic Retinopathy, denoted as disease A (DR). For disease A, establish disease rule entries, including the following sets: : The set of necessary imaging features of disease A (appearance / satisfaction strongly supports A).

[0071] : The set of optional enhanced features of disease A (appearance / satisfaction can enhance the credibility of A).

[0072] : The set of imaging features that conflict with / exclude disease A (appearance is inconsistent with A).

[0073] : The complete set of rule features of disease A.

[0074] : The set of features with positive polarity (POS) extracted from the inference chain (CoT). For ease of understanding and reproduction, a set of exemplary mappings is provided (not limited to the only choice): Microaneurysm; : Retinal hemorrhage (dot / blot hemorrhage); Hard exudate or cotton-woolspots; Typical signs of AMD include drusen / geographic atrophy.

[0075] 2) Calculation method of indicators: In the draw Then, calculate the number of hits, the number of conflicts, and the similarity as follows (the example uses set operations for easy reproduction): Number of hits ; Number of conflicts (Conf) ; Similarity (Sim, Jaccard) ; Discrete reward value range: when the diagnosis is correct ( When this happens, the reward is: When a diagnosis is incorrect ( When this happens, the reward is: .

[0076] Scenario 1: The model diagnosis is correct (DR), and the inference chain covers key signs → Reward 0.2; Let the model output diagnosis be DR, that is And extracted from the inference chain:

[0077] And no exclusion items were drawn. .

[0078] but:

[0079]

[0080]

[0081] Under one possible threshold setting (e.g., the high consistency condition is...), and and To achieve high consistency, the reward is: .

[0082] Example of a chain of inference: "Multiple microaneurysms and punctate hemorrhages were visible in the fundus, consistent with diabetic retinopathy." Scenario 2: The model diagnoses correctly (DR), but the inference chain generalizes and fails to reflect key image information → Reward 0.2; Let the model output diagnosis be DR, that is However, if the inference chain only contains generalized statements (e.g., "consider diabetic retinopathy, follow-up recommended") and does not hit any key features that can be matched with the feature library, then:

[0083] but:

[0084]

[0085]

[0086] Since no necessary / optional features were hit and the similarity is close to 0, it is judged as low consistency (exemplary condition: and ), Rewards will be awarded as follows: .

[0087] Scenario 3: The model diagnoses an error (not DR), but the inference chain still contains local objective signs of DR → reward 0.1. Let the true diagnosis be DR ( However, the model output is for another disease. (For example, AMD), that is:

[0088] Simultaneously, the inference chain extracted:

[0089] And no exclusion items were drawn. Note: The "local plausibility" here is evaluated based on the rule entries of the actual disease A.

[0090] but:

[0091]

[0092] (here) (This is not a necessary condition and is only used as an example) In the case of diagnostic errors, the exemplary local rationality condition is defined as: if and Then reward ;otherwise .

[0093] Therefore, this situation satisfies local rationality, and the reward is:

[0094] Example of an inference chain fragment: "Although macular degeneration is considered, microaneurysms are still visible in the image, suggesting diabetes-related changes." (The conclusion is incorrect, but the description of local signs is objectively consistent with the input, therefore a small positive reward is given.) Counterexample (also falls under case three, but should be rewarded with 0): If the model diagnoses an error ( ), and the inference chain extracted: ; If an exclusion item that conflicts with DR (such as drusen / geographic atrophy) appears, then: ,or ; If the local rationality condition is not met, the reward will be: .

[0095] (ii) Implementation plan for introducing Large Language Model (LLM) to assist scoring.

[0096] In this embodiment, the inference chain and the image features extracted in the pre-structured model are input into the LLM for auxiliary evaluation. The model outputs a consistency level or score according to a preset standard and maps it to the corresponding reward level. The score is judged and assigned by the large language model: 0.2 for completely correct, 0 for partially correct, and -0.2 for completely wrong.

[0097] (iii) Implementation plan for training the independent reward model.

[0098] In this embodiment, a specialized "inference chain-image feature consistency" discrimination model is constructed. The input image code, feature label and inference chain text are used as inputs. The discrimination model outputs a consistency classification or score, which is then discretized and used as a context consistency reward.

[0099] (I) Construction of the discriminant model.

[0100] (1) Training sample construction: The training samples of the consistency discrimination model are defined as triples: in The inference chain text generated by a generative model or a human report. These are the feature labels corresponding to the images. It can be obtained through any of the following methods (can be used alone or in combination): Manual annotation: Doctors / annotators annotate image features to form feature labels; Detection / segmentation model: Detection / segmentation of lesions such as microaneurysms, hemorrhages, and exudates are mapped to feature IDs; Structured report transcription: Doctor reports are transcribed into feature labels through rule extraction, information extraction models, or large models; Multi-model consistency fusion: The outputs of multiple detectors are combined / voted, and the confidence level is recorded before thresholding. , This indicates whether the feature exists. To ensure repeatability, the feature set definition must have a fixed version number, for example: feature dictionary version. Threshold version One possible construction method in this embodiment is to use positive samples + multiple types of negative samples to construct a diverse range of samples. Positive samples: Human-generated inference chains (doctor descriptions) or high-quality model inference chains; Negative samples (containing at least the following types): Feature permutation negative samples: ... The negative samples include: replacing some signs with others (e.g., replacing "hemorrhagic" with "drusen"); polarity reversal negative samples: changing "visible" to "not seen," or vice versa; attribute perturbation negative samples (optional): changing "macular region" to "peripheral region," "multiple" to "not seen," etc.; and cross-disease migration negative samples: pairing typical inference chains of other diseases with the current image. The construction rules and parameters of the negative samples (replacement ratio, reversal ratio, etc.) are fixed during training to ensure reproducibility.

[0101] (2) Construction of the supervisory signal: Consistency discrimination models require explicit supervision signals. This embodiment presents an implementable and quantifiable method for generating supervision signals: first, features are extracted from the inference chain, and then compared with the labels. The consistency score is obtained through comparison. It is used to train regression or classification.

[0102] (a) Feature extraction of the inference chain: Construct a feature dictionary and pattern library (which can be a manually maintained feature library) for... The following was obtained through the execution of the extraction: The extracted inference chain feature set: ,in, This indicates that the feature has an affirmative / negative polarity in the reasoning chain.

[0103] One possible polarity determination rule: If a negative word ("not seen / no / denied / excluded") appears near the feature, then... Otherwise, the default is... .

[0104] (b) Definitions of coverage and conflict rate.

[0105] Set real labels The "existing feature set" is: ; "No feature set exists" means: ; Let the set of affirmative features in the reasoning chain be: ; The set of negative features is: ; Define coverage: ; (like Then it is stipulated (Or customize according to task) Define conflict rate, which includes two types of conflict: describing a real feature as "unseen": To describe features that do not actually exist as "visible": ;therefore: ; (c) Consistency score Define (continuous tags).

[0106] In this embodiment, ;in, The conflict penalty coefficient is exemplified by the following values: Indicates truncation to (2.4) Consistent classification labels: If classification training is used, the consistent classification labels can be... Discrete can be categorized into three types: High consistency: Consistency: Low consistency: The example threshold is fixed as follows: ,

[0107] (3) Consistency discrimination model structure: input encoding and fusion.

[0108] The consistency discrimination model consists of three types of encoders and a fusion discrimination head: (a) Image encoder Input: Image Output: Image vector Implementation examples: Use a convolutional network or Visual Transformer (ViT) to obtain a global pooling vector; or output patch-level features and then pool them. .

[0109] (b) Feature label encoder Input: Feature label set Output: Feature vector Example of implementation (determined and easy to implement): Build an embedding table for each feature ID. ;by Weighted summation and normalization are performed using these weights: ; in, To prevent small constants from being divided by zero.

[0110] (c) Inference Chain Text Encoder Input: Inference chain text Output: Text vector ; Example of implementation: Use the Transformer text encoder to obtain the [CLS] vector or the average pooling vector.

[0111] (d) Fusion and discrimination head The three vectors are concatenated and then input into a multilayer perceptron to obtain the score. ; in, For Sigmoid, ensure If a classification output is used, the discriminant head output will be divided into three categories of logits, and the class probabilities will be obtained by Softmax.

[0112] (4) Training objectives and training process, specifically: (a) Regression training objective: with For monitoring the signal, mean squared error (MSE) or Huber loss is used: ; Optional sample weighting (e.g., increasing the weight of negative samples): ; in, Preset weights, such as positive samples negative samples .

[0113] (b) Classification training objective: To achieve consistent categories Cross-entropy is used as the monitoring signal: In order to make The threshold is stable and can be scaled for temperature or fine-tuned on the validation set.

[0114] (II) Contextual Consistency Reward: Output Score of Consistency Discriminant Model (or category) After that, based on the correctness of the diagnosis, it is mapped to different reward sets respectively (1) When the diagnosis is correct ( ): Three gears When outputting scores, a fixed threshold discretization method is used. , : like ,but ;like ,but ;like ,but When outputting by category: : ; : ; : A correct diagnosis does not necessarily mean a high-quality reasoning chain; here, we use... The chain of reasoning must "get to the point," or it will be penalized.

[0115] (2) When a diagnosis is incorrect ( Two gears When a diagnosis is incorrect, the overall conclusion is not rewarded; only the reasoning chain that includes a "locally objective and reasonable description" is rewarded. The score output uses a local reasonableness threshold. : like ,but ;otherwise When outputting by category: if : ;like : When the model makes a diagnosis error, if the inference chain still objectively describes some real signs in the image (such as "visible microaneurysm"), it is still valuable for subsequent manual review, and therefore a small positive reward is given. If the reasoning chain is significantly inconsistent with the image, the reward will be... .

[0116] The above schemes can be used individually or in combination to adapt to different data scales, hardware conditions, and application scenarios. To maintain training stability during the reinforcement learning phase, this invention limits the context consistency reward to a finite number of discrete levels and significantly lowers its magnitude, placing it below the answer accuracy reward. Through this design, the context consistency reward serves as an auxiliary signal to guide the quality of the inference chain structure and its medical consistency, without interfering with the optimization of the primary objective (diagnostic accuracy). This reward magnitude system allows the model to steadily improve diagnostic performance during training while gradually establishing a medically credible inference structure, effectively enhancing the structural regularity, image correspondence, and clinical reliability of the inference chain.

[0117] In summary, this invention improves the training process of a large-scale fundus image interpretation model through the proposed training method. It introduces constraints on the correspondence between the inference chain and imaging features and diagnostic knowledge, thus overcoming the shortcomings of existing methods that rely solely on SFT or only employ answer-level rewards in terms of inference reliability and clinical consistency. This invention offers the following beneficial effects: (1) Improving the verifiability and clinical consistency of the inference chain: This invention introduces explicit inference chain annotation in the supervised learning phase, enabling the model to output a reasoning process containing imaging features and clinical knowledge step by step; in the reinforcement learning phase, the correspondence between the inference content and the actual imaging features and clinical knowledge is constrained through context consistency rewards. The generated inference chain has a clear structure and well-defined elements, which facilitates manual review and verification of the intermediate inference process by doctors or developers, thereby improving the usability of the model in clinical scenarios.

[0118] (2) Enhance training signals to improve training efficiency and stability: In traditional reward designs that rely solely on the accuracy of the final answer, the model mainly receives "right / wrong" level supervision, and the gradient signals are relatively sparse. This invention introduces additional supervision related to physical characteristics and knowledge on the same sample through contextual consistency rewards, enabling the model to obtain richer feedback information in each update, which helps to alleviate problems such as slow convergence and large oscillations in training, and improves the stability and efficiency of the training process.

[0119] (3) Improving overall diagnostic performance under the constraint of reasoning quality: In the reinforcement learning stage, this invention distinguishes between two situations: correct and incorrect image interpretation conclusions. When the conclusion is correct, the focus is on whether the reasoning chain accurately reflects key imaging features. When the conclusion is incorrect, the focus is on whether the reasoning chain is still reasonable at the level of diagnostic and treatment knowledge. Through this design, two common error patterns can be suppressed: "the answer is correct but the reasoning does not match the imaging evidence" and "the reasoning seems reasonable but the conclusion is significantly biased." This improves the quality of reasoning and enhances overall diagnostic performance.

[0120] To verify the effectiveness of the method of this invention, experimental evaluation was conducted on FunBench, an authoritative benchmark dataset for fundus image interpretation. FunBench constructs a task system with four levels (L1–L4), covering multiple tasks such as modality perception, anatomical structure perception, lesion analysis, and disease diagnosis. The experiment was conducted in accordance with the official FunBench evaluation protocol and to avoid data leakage. The F1 score was used as the evaluation metric. The model trained by this invention based on Qwen2.5-VL-3B and Qwen2.5-VL-7B (denoted as "this invention-3B" and "this invention-7B" respectively) was compared with various existing general multimodal large models and medical multimodal large models. The results in Table 2 show that the overall F1 scores of this invention –3B and this invention –7B are 67.1 and 69.2, respectively, both higher than the existing high-performance model FundusExpert-8B (trained based on InternVL2.5-8B) of 62.5. Furthermore, they achieve higher scores in key hierarchical tasks such as anatomical structure perception (L2), lesion analysis (L3), and disease diagnosis (L4), indicating that the method of this invention has practical performance improvements in multi-level fundus image understanding tasks.

[0121] Table 2 Comparison of the present invention with existing large-scale models.

[0122] Furthermore, judging from the training process curve, such as Figure 3 As shown, GRPO training with context consistency reward (with context consistency reward) exhibits faster reward growth, a smoother accuracy curve, faster convergence speed, and less oscillation during training compared to standard GRPO without this reward. Furthermore, it maintains a higher steady-state reward level after convergence, further demonstrating that context consistency reward helps improve the stability of the training process and the final result.

[0123] The following detailed embodiments illustrate the practical application of the reasoning-enhanced fundus image reading large model training method of the present invention.

[0124] (I) Specific implementation of the model: This invention selects the Qwen2.5-VL series multimodal large model as the base model. The model includes a visual encoding module and a language reasoning module, which can accept fundus images and text input, and output a structured reasoning chain and diagnostic results. This invention sequentially performs a supervised fine-tuning stage and a reinforcement learning stage on this base model to achieve enhanced reasoning training.

[0125] (II) Training Method: SFT Stage: In this stage, training data containing explicit inference chain annotations is input into the base model, enabling the model to learn and output the inference process and final diagnosis step by step. The following main hyperparameters are used during training: Learning rate: 3 × 10 -5 Training batch size: 8, validation batch size: 8; training is performed using 8 GPUs in parallel, with a gradient accumulation step count of 8, corresponding to a total effective batch size of approximately 512; optimizer: AdamW, β1=0.9, β2=0.999, ε=1×10 -8 Learning rate scheduling: cosine annealing; Number of training rounds: 5. After this training stage, the model can stably output structured inference chains and diagnostic results, serving as the initial model for the reinforcement learning stage. RL stage: In the reinforcement learning stage, this invention adopts a policy optimization framework based on GRPO, starting with the SFT-prepared model and introducing a comprehensive reward for post-training. The comprehensive reward consists of three parts: answer accuracy, format conformity, and context consistency. The specific values ​​are designed as follows: Answer accuracy reward: 0.5 for correct diagnostic results, 0 for incorrect results; Format conformity reward: 0.5 for output conforming to the preset structured format, 0 otherwise; Context consistency reward is implemented using Qwen3-Max as the scorer: when the diagnostic result is correct, based on whether the CoTs reflect the key information in the input, The value is selected from three levels: 0.2, 0, and 0.2. When the diagnosis result is incorrect, based on the local rationality of CoTs, the value is selected from two levels: 0 or 0.1. To constrain the degree to which the model deviates from the initial strategy, a KL regularization term is introduced into the objective function, with the KL coefficient set to 1×10. - ², a low-variance KL estimation strategy is employed. The main training parameters for the reinforcement learning phase are as follows: Policy network (actor) learning rate: 1 × 10⁻⁶ -6 Weight decay coefficient: 1 × 10 - ²; the global training batch size is approximately 512, and the micro-batch size per device is 8; the maximum gradient norm is limited to 1.0, and gradient checkpointing is enabled to conserve GPU memory; three candidate answers are generated for each input sample during the rollout phase to calculate rewards and relative advantages; the generation phase temperature is typically set to 1.0, and top_p is set to 1.0; the validation phase temperature is lowered to approximately 0.6, and top_p is approximately 0.95 to obtain more stable evaluation output. Through these settings, while ensuring training stability, the contextual consistency reward is effectively used to constrain the inference process, thereby enhancing the inference ability of the large-scale fundus image reading model.

[0126] (III) Prediction scheme: In the reasoning stage, the retinal image to be analyzed and the corresponding question text are input. The model first outputs a structured reasoning chain to explain the diagnostic basis and steps; then it outputs the final diagnosis result. The reasoning chain structure generated by this invention is standardized and can be used for manual review and clinical verification.

[0127] Example 2: In contrast to Example 1, which provides a method for training a large-scale fundus image reading model with enhanced reasoning, this example provides a training device for a large-scale fundus image reading model with enhanced reasoning, comprising: a supervised fine-tuning training module configured to perform supervised fine-tuning training on a multimodal base model using a fundus image reading dataset with explicit inference chain annotations to obtain a base model Fundus-base; and a reinforcement learning module configured to optimize the trained base model Fundus-base using a reinforcement learning framework based on group relative policy optimization of GPRO and introducing a comprehensive reward mechanism with contextual consistency constraints to obtain a large-scale fundus image reading model with enhanced reasoning. This large-scale fundus image reading model with enhanced reasoning is used to output a structured inference chain and a final diagnostic result from the input retinal image to be analyzed and the corresponding question text.

[0128] Example 3: Figure 4 As shown, this embodiment provides an electronic device corresponding to the inference-enhanced fundus image reading large model training method provided in Embodiment 1. The electronic device can be an electronic device for client use, such as a mobile phone, laptop, tablet computer, desktop computer, etc., to execute the method of Embodiment 1.

[0129] Example 4: This example provides a computer-readable storage medium for storing one or more programs, the one or more programs including computer instructions, which, when executed by a computer, cause the computer to perform the method provided in Example 1 above.

[0130] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In the description of this specification, the terms "a preferred embodiment," "furthermore," "specifically," "in this embodiment," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments in this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a large-scale fundus image reading model with reasoning enhancement, characterized in that, include: The base model Fundus-base was obtained by supervised fine-tuning training of the multimodal base model using a fundus image dataset with explicit inference chain annotations. The reinforcement learning framework based on group relative policy optimization of GPRO and the comprehensive reward mechanism with context consistency constraints are used to optimize the trained base model Fundus-base to obtain a large-scale inference-enhanced fundus image reading model. This inference-enhanced fundus image reading model is used to output a structured inference chain and the final diagnostic result from the input retinal images to be analyzed and the corresponding question text.

2. The reasoning-enhanced fundus image reading large model training method according to claim 1, characterized in that, Construct a fundus image reading training dataset with explicit inference chain annotations, specifically as follows: In addition to the original fundus images and the final diagnostic labels, the intermediate reasoning steps followed by doctors during the image interpretation process are further annotated, including the localization of key anatomical structures, identification of lesion features, evidence association, and the process of forming a diagnostic conclusion. This enables the model to learn an interpretable reasoning path from visual cues to a diagnostic conclusion. Each sample includes at least the following four tuples: (a fundus image; a question for the image interpretation task; a reference answer; a structured reasoning chain). Based on the fundus image reading training dataset with explicit inference chain annotations, the multimodal base model is trained under supervision and fine-tuned to enable the trained base model Fundus-base to have medical fundus image reading and inference capabilities.

3. The reasoning-enhanced fundus image reading large model training method according to claim 2, characterized in that, The base model Fundus-base is obtained by supervised fine-tuning of the multimodal base model using a fundus image dataset with explicit inference chain annotations. The specific process is as follows: A multimodal pedestal model that has been pre-trained on large-scale general image, text or multimodal data is selected as the initial model. Based on the training sample set constructed above, the multimodal pedestal model is subjected to supervised fine-tuning training. The training sample set includes fundus images, question descriptions corresponding to the images, standard diagnostic answers and explicit inference chain annotations. During training, each training sample is organized into an "input-target-output" pair. The input includes at least fundus image data and corresponding text query information, while the target output includes disease classification or grading results and corresponding stepwise inference text. By minimizing the supervised loss function between the model's generated output and the target output, the model is constrained to learn the mapping relationship between visual features, medical semantics, and diagnostic conclusions. Simultaneously, it learns the intermediate judgment and evidence integration steps in medical image reading, enabling the trained base model Fundus-base to output structurally sound and parsable inference chains, possessing interpretable medical diagnostic reasoning capabilities. Capability 1: Output the analysis of image content in sequence; Ability 2: To reason by combining relevant medical knowledge; Capability 3: Provide the reasoning process and final diagnosis result in a predetermined format, as follows: <think> …reasoning process…< / think> <answer> …diagnostic results…< / answer> 。 4. The reasoning-enhanced fundus image reading large model training method according to claim 3, characterized in that, A reinforcement learning framework based on group-relative policy optimization of GPRO is used, and a comprehensive reward mechanism with context consistency constraints is introduced. The trained base model Fundus-base is then optimized to obtain a large-scale fundus image reading model with enhanced inference capabilities. The specific process is as follows: (I) Copy the Fundus-base weights to the reference model With reinforcement learning models , where: reference model The parameters of the reinforcement learning model remain fixed during training to provide a stable generation distribution as a regularization constraint. As the policy model to be optimized, its initial parameters are the same as those of Fundus-base, and are updated using the GRPO algorithm in subsequent training. (II) For each input sample , by strategy model Perform group sampling and generate at once 10 candidate answers: , where the input sample At least including: fundus image data Text questions or instructions corresponding to diagnostic tasks Reference information used for verification During the group sampling phase, the actual input to the policy model only includes fundus image data I and text questions or instructions q. The model generates candidate answers based on the above input. Reference information 'a' does not participate in the model generation process, but is used for reward calculation and relative merit evaluation after candidate answers are generated. Each candidate answer... All contain complete reasoning chain text and final reading conclusions, and conform to a predefined output structure for subsequent reward evaluation; (III) Calculate the comprehensive reward for each candidate answer. Within the candidate answer group corresponding to the same input sample, rank the candidate answers within the group to determine the relative merits of different candidates. (IV) Strategy Model under the GRPO Framework The optimization objective is defined as: within the candidate answer group corresponding to the same input sample, maximizing the log-likelihood weighted by the relative advantage within the group, and the objective function is expressed as: in, Indicates the input sample. This represents the set of candidate answers generated by the policy model during the group sampling phase. For the corresponding candidate answers The relative advantage value within the group, The first term in the objective function is the relative advantage-weighted policy optimization term, used to adjust the generation probability of the policy model based on the relative merits of candidate answers within the group; the second term in the objective function is the KL divergence regularization constraint term, used to measure the policy model. Compared with the reference model Given differences in output distribution under the same input conditions, and when training stability has been ensured through other means, the regularization term can be omitted. The policy model can be updated solely based on the relative advantage term, and the policy model parameters can be iteratively updated by optimizing the gradient of the objective function. .

5. The reasoning-enhanced fundus image reading large model training method according to claim 4, characterized in that, A comprehensive reward is calculated for each candidate answer. Within the candidate answer group corresponding to the same input sample, the candidate answers are ranked within the group to determine the relative merits of different candidates. The specific process is as follows: The overall reward value based on each candidate answer The candidate set is sorted, with higher reward values ​​indicating higher overall quality of candidate responses. Through sorting and relative advantage determination mechanisms, high-quality and low-quality responses can be clearly distinguished within the candidate group, and these can be transformed into relative optimizations required for subsequent strategy optimization. Multiple methods are used to characterize the relative strengths and weaknesses among candidate responses, including: Firstly, the relative advantage definition method is based on within-group mean normalization, which measures relative performance by comparing the reward value of an individual candidate with the average reward value of the candidate group. Secondly, the relative advantage determination method based on the ranking results, that is, ranking the candidate answers from high to low according to the reward value, and mapping the ranking to the relative advantage value; Thirdly, the relative advantage determination method based on pairwise comparison measures relative merit by comparing the reward differences between candidate answers.

6. The reasoning-enhanced fundus image reading large model training method according to claim 5, characterized in that, Comprehensive Rewards Represented as: , Represents the corresponding weight coefficient, where: This represents the accuracy reward for the answer, which is used to determine whether the final reading conclusion output by the model is consistent with the reference answer. This represents the format conformance bonus, which is used to constrain whether the model output meets the predefined structured format requirements. This represents the context consistency reward, which measures whether the intermediate inference chain is consistent with the imaging features and related diagnostic knowledge presented in the image.

7. The reasoning-enhanced fundus image reading large model training method according to claim 6, characterized in that, The context consistency reward is calculated using a two-branch strategy, the specific process of which is as follows: Scenario 1: The model's interpretation conclusion is the same as the reference answer, used to obtain the image consistency score: When the model's final interpretation conclusion is consistent with the reference answer, the focus is on whether the inference chain generated by the model accurately, sufficiently, and contradictorily covers the key radiological features. (a) Determination and representation of key radiographic features: For each input sample, a set of key radiographic features corresponding to the reference answer is pre-determined; (b) Extraction of imaging features from the inference chain: Medical entities and attributes are extracted from the inference chain text generated by the model to obtain the set of imaging features explicitly or implicitly mentioned by the model during the inference process, denoted as: (c) Quantitative indicators of matching degree: Based on the above two feature sets, the following quantitative indicators are defined: Feature coverage: Used to measure the proportion of key radiographic features correctly covered in the model inference chain; Feature conflict rate: in, This indicates that a feature description in the reasoning chain clearly contradicts the image facts or reference annotations; (d) Determining the reward tier: Based on the above indicators, the context consistency reward is defined as follows: in, and As the coverage threshold parameter, the above design gives a positive reward when the inference chain completely and accurately describes the key lesion; no reward when only partial coverage or a vague description is given; and a negative reward when the inference content is obviously contradictory to the image features. Scenario 2: The model's interpretation conclusion differs from the reference answer, used to obtain a reasoning consistency score: (a) Whether it reflects a reasonable differential diagnosis approach, even if the final conclusion is wrong, the analysis process is in line with medical common sense; (b) Whether inferences that clearly contradict medical knowledge are avoided; Using a large model and RAG medical reference materials, if the reasoning reflects an analytical path or differential diagnosis approach that conforms to medical common sense, a small positive reward can be given; if the reasoning clearly deviates from clinical logic, no reward will be given.

8. A large-scale training device for fundus image reading with reasoning enhancement, characterized in that, include: The supervised fine-tuning training module is configured to use a fundus reading dataset with explicit inference chain annotations to perform supervised fine-tuning training on the multimodal base model to obtain the base model Fundus-base; The reinforcement learning module is configured with a reinforcement learning framework based on group relative policy optimization of GPRO and introduces a comprehensive reward mechanism with context consistency constraints. The trained base model Fundus-base is optimized to obtain a large-scale fundus image reading model with enhanced reasoning. This large-scale fundus image reading model with enhanced reasoning is used to output a structured reasoning chain and the final diagnostic result from the input retinal images to be analyzed and the corresponding question text.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include computer instructions for causing a computer to perform the method according to any one of claims 1-7.

Images