Medical image tool enhancement processing method based on auto-reflection reinforcement learning

Abstract

The invention discloses a medical image tool enhancement processing method based on auto-reflection reinforcement learning. The method comprises the following steps: constructing a multi-modal reasoning model for vision-language joint reasoning; and performing supervision and fine tuning on the model by utilizing a cold start data set with tool calling track marks, so that the model has basic structured reasoning capability and normative tool use behaviors. An early-stage error reasoning track and a later-stage correct reasoning track which are generated by the same input at different training check points are collected, an auto-reflection training sample is constructed, and auto-reflection fine tuning is performed on the model. And based on the reinforcement learning data set containing the real tool calling scene, constructing a reward function and executing tool enhanced reinforcement learning training. After a medical image to be processed and a corresponding natural language problem are received, the trained model can generate gradual natural language thinking, a tool calling instruction and an external tool observation result, and outputs a medical image analysis conclusion, so that refined and explainable medical visual reasoning is realized.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a medical image tool enhancement processing method based on self-reflective reinforcement learning. Background Technology

[0002] Medical image analysis is a crucial component of clinical diagnosis, treatment planning, and disease monitoring. With the development of deep learning, technologies such as Convolutional Neural Networks (CNNs) and Visual Transformers (ViTs) have been widely applied to medical image feature extraction and segmentation tasks. Meanwhile, Natural Language Processing (NLP) based on Large Language Models (LLMs) has demonstrated powerful reasoning capabilities in multimodal tasks, enabling joint reasoning based on image and text information. For example, in the prior art, Chinese patent application CN110751627A discloses an image processing method, apparatus, computer device, and storage medium. The method includes: acquiring a medical image; inputting the medical image into a preset neural network model to obtain image-level processing results and pixel-level processing results of the medical image; the preset neural network model is used to perform at least two different functional level processing on the medical image.

[0003] However, existing automated medical image analysis methods still have the following problems: 1. Limited capabilities of a single model: Traditional methods typically rely on a single model to directly predict image features or diagnostic results, lacking the ability to flexibly combine multiple external tools (such as image segmentation, magnification, and morphological analysis tools), making it difficult to cope with complex, multi-step medical reasoning tasks.

[0004] 2. Lack of adaptability in tool usage strategies: When using external tools, existing multimodal inference models often rely on static or predefined tool call sequences and cannot adjust their strategies autonomously based on intermediate inference results, which can easily lead to redundant or erroneous tool calls.

[0005] 3. Insufficient error correction capability: In medical tasks, even with tool assistance, models may still produce erroneous inferences. Current multimodal inference models typically lack self-reflection and self-correction mechanisms, making it difficult to proactively correct errors based on tool output or inference history, resulting in low accuracy of the final results.

[0006] 4. Insufficient Application of Reinforcement Learning: Although reinforcement learning has been used to optimize policies in traditional natural language processing and image tasks, there is still a lack of effective methods for the joint optimization of tool call sequences, external observation information, and the final answer in the context of multimodal reasoning in medical images. Existing methods often only provide sparse rewards to the final answer, ignoring the optimization of tool usage efficiency and the structure of the reasoning process.

[0007] Therefore, there is an urgent need for a multimodal reasoning method that can achieve self-reflection and tool enhancement, so that the model can autonomously call external tools in medical image analysis and continuously correct errors through self-reflection mechanism to improve the accuracy and reliability of diagnosis and analysis. At the same time, through reinforcement learning optimization strategy, the efficiency of tool calling and the quality of reasoning can be improved simultaneously. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a medical image tool enhancement processing method based on self-reflective reinforcement learning.

[0009] The objective of this invention can be achieved through the following technical solutions: This invention provides a medical image enhancement method based on self-reflective reinforcement learning, comprising the following steps: A multimodal inference model is constructed, which includes an image encoder, a text encoder, an inference module, a tool interface module, and an observation encoding unit; Based on a cold start dataset with tool call trajectory annotations, the multimodal inference model is subjected to cold start supervised fine-tuning. In cold start supervised fine-tuning, early error trajectories and later correct trajectories generated by the same input at different training checkpoints are collected. Trajectory pairs that meet the self-reflection characteristics are constructed as self-reflection training samples, and the multimodal reasoning model after cold start supervised fine-tuning is self-reflected based on the self-reflection training samples. After completing the self-reflective fine-tuning, the multimodal reasoning model is trained using tool-enhanced reinforcement learning based on the reinforcement learning dataset to obtain the trained multimodal reasoning model. Acquire the medical images to be processed and their corresponding natural language questions; The medical image to be processed and the corresponding natural language question are input into the trained multimodal reasoning model to obtain reasoning output, including natural language thinking generated by the model, tool call instructions, observation information returned by external tools, and the final medical image analysis results.

[0010] Furthermore, the image encoder is a deep convolutional neural network used to process medical images and extract medical image features; The text encoder is used to perform word-level and sentence-level semantic encoding on natural language problems to obtain corresponding text semantic features; The inference module is an autoregressive sequence generation network based on a large language model, used to generate inference paths based on medical image features, text semantic features, and observation features. ; The tool interface module is an instruction parsing and tool scheduling unit, used to parse the tool call instructions generated by the inference module and send call requests to external image analysis tools. The observation encoding unit is an observation processing network used to receive observation information returned by external image analysis tools and encode it into observation features that can be read by the inference module.

[0011] Furthermore, the reasoning path Represented as: in, For the first n The natural language thinking content generated step by step For the first n The tool call instructions generated step by step, The observation features corresponding to the observation results returned by the external image analysis tool according to the tool call command; in the... n During step-by-step reasoning, the reasoning module is based on the currently input medical image. I ,question Q and before Step-by-step reasoning history autonomously determines the next step of natural language thinking. Tool call commands , is represented as: in, Indicates the preceding The history of step-by-step reasoning Represents the mapping function for the inference module; The reasoning module updates the reasoning path step by step through a loop mechanism of thinking – tool invocation – observation, until the termination step. The system generates a definite answer or reaches the preset maximum context length or maximum number of interaction rounds.

[0012] Furthermore, the external image analysis tool includes a first image processing tool, a second image processing tool, and a third image processing tool; The first image processing tool is a location-cue-based image segmentation tool, used to receive the target region bounding box provided by the multimodal inference model as input and output a fine segmentation mask of the target object; The second image processing tool is a semantic description-based image segmentation tool, which receives images and text prompts as input and outputs a segmentation mask of the corresponding target region, so as to achieve target region localization without the need for the model to explicitly predict the bounding box; The third image processing tool is an image local magnification tool, which is used to receive an image and a bounding box or segmentation mask as input. When the input is a bounding box, it outputs a magnified image of the corresponding region. When the input is a segmentation mask, it outputs a magnified image and explicitly draws the target outline in the image to highlight the morphological features of the lesion.

[0013] Furthermore, the cold start dataset includes multiple samples, each of which includes a medical image to be processed, a corresponding natural language question, a true answer, and an inference path with tool call trajectory annotations. The tool call trajectory annotations record the natural language thinking generated in each step of the inference, the tool call instructions, and the observation results returned by the corresponding tool.

[0014] Furthermore, the cold-start supervised fine-tuning of the multimodal inference model based on the cold-start dataset with tool call trajectory annotations specifically includes: For each sample in the cold start dataset, the input sample contains a medical image, a natural language question, and the corresponding labeled inference path. At most multimodal reasoning models will generate inference paths. With annotation path Perform step-by-step comparisons; think based on the natural language generated at each step. Tool call commands With annotation content and To assess consistency, the cold start supervision loss function is calculated using the following formula: in, For the first n The natural language thinking content generated step by step For the first n The tool call instructions generated step by step, The observation results returned by the tool call command in the labeled path; This represents the total number of tokens for natural language thinking and tool invocation instructions in the inference path generated by the model; Indicates the first n Step into natural language thinking The number of tokens; Indicates the first n Step tool call command The number of tokens; Indicates the total number of steps in the reasoning path; This represents the expectation of all samples in the cold start dataset; Indicates the first n Step-by-step model generates natural language thinking Tool call commands The conditional probability is determined by the inference module of the multimodal inference model based on medical image features, text semantic features, and prior information. The autoregressive sequence generation method of step observation features is used to calculate the sequence. Indicates the first n The inference history generated by the model before the step; The true answer for the sample; The cold start supervision loss function; By minimizing the cold start supervision loss function This enables supervised fine-tuning of cold starts for multimodal inference models.

[0015] Furthermore, in the cold-start supervised fine-tuning, early error trajectories and later correct trajectories generated by the same input at different training checkpoints are collected. Trajectory pairs that conform to the self-reflective characteristics are constructed as self-reflective training samples. Based on the self-reflective training samples, the multimodal inference model after cold-start supervised fine-tuning is subjected to self-reflective fine-tuning, specifically including: During the training process, for each sample in the cold start dataset, the inference path generated by the multimodal inference model at the early training checkpoint and the later training checkpoint is recorded respectively. The early training checkpoint is the training state when the model has not yet fully converged, and the later training checkpoint is the state when the model has been trained to convergence or is close to convergence. The inference path generated by the early training checkpoints is denoted as: ;in, These represent the early training checkpoints. The generated natural language thinking content, tool call commands, and observation features; This represents the total number of steps in the early reasoning path; The inference path generated by the later training checkpoints is denoted as: ;in, These represent the checkpoints in the later stages of training. The generated natural language thinking content, tool call commands, and observation features; Indicates the total number of steps in the later reasoning path; Determine if an arbitrary step index exists. and This makes the following formula: Established; among them, For the early reasoning path The tool call instructions generated step by step, For the later reasoning path The tool call instructions generated step by step, This is an indicator function for whether the final answer corresponding to the early reasoning path is correct; a value of 0 indicates an error, and a value of 1 indicates a correct answer. This is an indicator function for whether the final answer corresponding to the later reasoning path is correct; a value of 0 indicates an error, and a value of 1 indicates a correct answer. If it exists, then the condition will be satisfied. and Alignment and integration constitute a self-reflection training sample ; Self-reflection training samples The multimodal inference model, fine-tuned with cold-start supervision, is trained by minimizing the cold-start supervision loss function. This enables self-reflective fine-tuning of multimodal reasoning models.

[0016] Furthermore, the conditions that will be met and Alignment and integration constitute a self-reflection training sample Specifically, it includes: For each early error step Find the corresponding modification steps in the later inference path The modification refers to the existence of corresponding steps in the later path that satisfy... Or the early path leads to the wrong answer while the later path leads to the correct answer; Natural language thinking generated in early steps Correct tool call instructions for later steps This will combine to form new reflective steps, while also incorporating the observational features corresponding to later steps. Incorporate this reflection step; For steps in the early path that were not modified and corresponded to the correct final answer, their natural language thinking, tool call instructions and observation characteristics were preserved and directly added to the self-reflective training samples. Following the chronological order of the original reasoning path, all reflection steps are sequentially concatenated with the correct, unmodified steps to obtain a complete self-reflection training sample.

[0017] Furthermore, the step of training the multimodal reasoning model using tool-enhanced reinforcement learning based on the reinforcement learning dataset to obtain the trained multimodal reasoning model specifically includes: For each sample in the reinforcement learning dataset, including medical images I Corresponding natural language problem Q and the real answer A In the current policy parameters of the multimodal inference model Multiple candidate inference trajectories are generated below. ,in, They represent the first The first candidate inference trajectory n The generated natural language thinking content, tool call commands, and observation features; For the first The total number of steps for each candidate inference trajectory; During the generation of each reasoning trajectory, the multimodal reasoning model alternately generates natural language thinking according to the thinking-tool invocation-observation cycle mechanism. Tool call commands Based on the current image, the problem, and the existing trajectory Inferring the next output, represented as ; Calculate the comprehensive reward for each trajectory. , is represented as: in, The reward is given for the answer, used to measure the difference between the model's output answer and the actual answer. A The degree of matching is determined. For classification tasks, a fixed score is awarded when the predicted answer matches the true answer. For segmentation masking tasks, the score is awarded in segments based on the IoU value between the predicted mask and the true mask. The format reward is used to ensure that the structure of the reasoning trajectory conforms to the expected order. Specifically, it is checked whether the output trajectory includes natural language thinking, tool calls and final answer marking in sequence. Points are deducted for missing or incorrectly ordered markings to ensure that the trajectory generation conforms to the specified format. The tool usage reward is used to encourage the model to call external tools under reasonable circumstances. The reward is given only when the model generates the correct answer and calls the external tool at least once during the reasoning process. The reward value is positively correlated with the number of tool calls and the contribution of the tool calls to the completion of the task. Based on the overall reward for each trajectory The reinforcement learning loss function is calculated, and the policy parameters of the multimodal inference model are updated by minimizing the reinforcement learning loss function to obtain the trained multimodal inference model.

[0018] Furthermore, the reinforcement learning loss function is formulated as follows: in, To reinforce the learning loss function; This indicates the reinforcement learning dataset. All samples and in the old strategy The following generated G Take the expected value of each candidate inference trajectory; This represents the medical image input in the sample; This represents the natural language question input in the sample; This represents the true answer of the sample; Indicates the first i The total number of all tokens in the inference trajectory, excluding observations returned by the tool; Indicates the current strategy parameters Next, the i Trajectory number n The probability of the step action compared to the old strategy The ratio of the probabilities of corresponding actions; Indicates the first i The trajectory in the first n All previously generated actions and observed historical trajectories; Indicates the first i The advantage function of a trajectory; Indicates the first i The overall reward for each trajectory; , They represent G The mean and standard deviation of the overall reward for each candidate reasoning trajectory; This is the clipping function; These are the trimming parameters.

[0019] Compared with the prior art, the present invention has the following advantages: (1) Existing medical image analysis technologies generally rely on a single model to extract features from images in a one-time manner or directly output diagnostic results. They lack the ability to continuously interact with images during the reasoning process, preventing the model from thinking like a doctor. In complex tasks, the model cannot flexibly use external segmentation, localization, or magnification tools according to reasoning needs, nor can it actively examine local abnormal areas during reasoning. This leads to a rigid reasoning chain, insufficient visual evidence, and ultimately significantly limits the accuracy and interpretability of diagnostic results. This invention constructs a multimodal reasoning model that includes an image encoder, a text encoder, a reasoning module, a tool interface module, and an observation encoding unit, enabling the model to have the initiative of visual thinking. During the reasoning process, the model can actively decide when to view images, which areas to view, and which image analysis tools to call based on the current thinking content, and incorporate the local visual information returned by the tools into subsequent reasoning in real time. Through the closed-loop mechanism of thinking—tool calling—observation, the model maintains dynamic interaction with the image throughout the entire reasoning chain, realizing a step-by-step observation, verification, and correction capability similar to that of human experts. This mechanism significantly enhances the model's image understanding depth in complex, fine-grained medical tasks, and improves the accuracy, reliability, and interpretability of multi-step reasoning.

[0020] (2) In existing medical image analysis technologies, relying solely on cue engineering or zero-shot / few-shot reasoning makes it difficult for multimodal agents to stably select and invoke appropriate image analysis tools in real medical scenarios. This results in non-standard reasoning paths generated by the model, redundant or erroneous tool invocations, thus affecting the accuracy of the final answer. This invention supervises and fine-tunes the multimodal reasoning model on a cold-start dataset with tool invocation trajectory annotations. It uses the complete annotated reasoning path provided by each sample as the training objective, minimizing the negative log-likelihood loss of the model in generating natural language thinking and tool invocation instructions at each step. This guides the model to generate structured, legitimate, and progressively reasoning-reasonable tool usage trajectories. During training, the model generates the next thought and tool invocation instructions based on medical images, natural language questions, and reasoning history through autoregression, achieving multimodal information fusion. This solves the problem of the model lacking a stable tool invocation strategy in the cold-start phase, enabling the multimodal reasoning model to learn diverse and robust progressive reasoning patterns, significantly improving the model's tool invocation accuracy and overall reasoning performance in medical image analysis tasks.

[0021] (3) In existing medical image analysis technologies, even if a multimodal agent can invoke tools to complete basic reasoning, the model still has the problem of easily generating incorrect tool calls or incorrect answers in early steps, lacking self-correction capabilities, leading to incorrect final reasoning paths and making it difficult to guarantee high accuracy in complex medical tasks. This invention, through a self-reflective fine-tuning stage, collects erroneous trajectories generated at early training checkpoints and correct trajectories generated at later training checkpoints for the same input during training. It uses formulas to determine trajectory pairs with self-reflective characteristics, aligns and integrates them to form self-reflective training samples, and further trains the model on the cold-start supervised fine-tuning model. This solves the problem of the model lacking self-correction capabilities, enabling the multimodal reasoning model to learn how to adjust tool call strategies and correct reasoning ideas based on early erroneous steps, thereby improving the model's accuracy and robustness in complex medical image tasks, significantly reducing error propagation, and enhancing the model's reliability in practical application scenarios.

[0022] (4) In existing medical image analysis technologies, even after supervised fine-tuning and self-reflective training, models struggle to adaptively discover optimal tool combination strategies in multi-task environments. This is especially true when multiple tools need to be dynamically invoked to complete complex reasoning tasks, leading to redundant invocations, inefficient reasoning, or loop errors, resulting in low reasoning efficiency and inaccurate answers. This invention introduces explicit tool invocation and observational reasoning trajectory modeling through a tool-enhanced reinforcement learning stage, and designs a fine-grained reward function. It comprehensively evaluates the correctness of answers, the standardization of output format, and the rationality of tool use. Simultaneously, it utilizes a group advantage normalization strategy to optimize and update model strategy parameters, enabling the multimodal reasoning model to retain high-reward trajectories and suppress low-quality reasoning. This solves the problem of models struggling to adaptively select tools in complex multi-task environments, allowing the multimodal reasoning model to autonomously discover effective tool combinations during reasoning, significantly improving reasoning efficiency, the rationality of tool invocation, and the accuracy of the final medical image analysis results, achieving expert-level efficient reasoning capabilities. Attached Figure Description

[0023] Figure 1 This is a flowchart of a medical image enhancement processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the training of a multimodal inference model according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] Example 1: This embodiment specifically provides a medical image tool enhancement processing method based on self-reflective reinforcement learning, such as... Figure 1 As shown, it includes the following steps: Step S1: Construct a multimodal inference model; The multimodal inference model includes an image encoder, a text encoder, an inference module, a tool interface module, and an observation encoding unit. The image encoder is a deep convolutional neural network used to process medical images and extract medical image features. The text encoder is used to perform word-level and sentence-level semantic encoding on natural language problems to obtain corresponding text semantic features. The inference module is an autoregressive sequence generation network based on a large language model, used to generate inference paths based on medical image features, text semantic features, and observation features. The tool interface module is an instruction parsing and tool scheduling unit, used to parse the tool call instructions generated by the inference module and send call requests to external image analysis tools; the observation encoding unit is an observation processing network, used to receive the observation information returned by external image analysis tools and encode it into observation features that can be read by the inference module.

[0026] The training process of this invention includes three stages (e.g. Figure 2 (as shown) 1. Cold start supervision and fine-tuning phase, used to establish basic tool calls and image interaction capabilities; 2. In the self-reflection and fine-tuning stage, the self-correction ability is strengthened through a self-reflection sample sampling strategy; 3. In the tool-enhanced reinforcement learning stage, the inference strategy is further optimized through fine-grained rewards, enabling the model to transition from "imitating the tool usage trajectory" to "autonomously discovering effective tool usage strategies"; Specifically, it includes the following steps S2-S4: Step S2: Based on the cold start dataset with tool call trajectory annotations, perform cold start supervised fine-tuning on the multimodal inference model; Relying solely on prompting engineering makes it difficult for multimodal agents to reliably select and invoke appropriate tools in real-world medical scenarios. Therefore, this invention first addresses this issue using a cold-start dataset... D Supervised fine-tuning is performed on cold datasets to enhance inference capabilities by initializing the model's tools. D Cold provides an annotated inference process with tool call trajectories for each sample.

[0027] In this phase, the training objective is to minimize the average negative log-likelihood of all inferences and tool call tokens, and its objective function is defined as: in, This represents the total number of tokens used for reasoning and tool calls. A To provide the true answer. The goal is to guide the model to generate structured, legitimate tool-use trajectories and learn diverse and robust step-by-step reasoning patterns.

[0028] Step S3: In the cold start supervised fine-tuning, collect the early error trajectory and the later correct trajectory generated by the same input at different training checkpoints, construct the trajectory pairs that meet the self-reflection characteristics as self-reflection training samples, and perform self-reflection fine-tuning on the multimodal inference model after cold start supervised fine-tuning based on the self-reflection training samples; Given the complexity of real-world medical tasks, errors can occur even when using tools; therefore, models need to possess the ability to self-reflect and correct based on tool output. This invention proposes a self-reflective sample sampling strategy to mine representative self-correcting instances from the training process and construct a self-reflective dataset accordingly. D reflect.

[0029] Specifically, during training, when the model gives an incorrect answer at an early checkpoint but obtains the correct answer at a subsequent checkpoint by changing the tool call sequence, this invention treats the sample as a self-reflective inference trajectory. Formally, if there exists the same input ( I,Q A pair of trajectories ( R early ,R (late) satisfies: in, u and v These represent the indices of any step in the two reasoning trajectories. This indicates whether the answer is correct. Each sample that meets the criteria will be added. D reflect. For each of these samples, the invention relates to ( R early , R Alignment and integration are performed using the late method to obtain a high-quality reasoning trajectory that simultaneously embodies self-reflection and correctness. The model is then fine-tuned using an objective function similar to that in equation (1). The selective training strategy in this stage encourages the model to focus on learning reasoning paths that possess both reflective features and correct answers, laying the foundation for the subsequent reinforcement learning stage.

[0030] Step S4: After completing the self-reflective fine-tuning, the multimodal reasoning model is trained using tool-enhanced reinforcement learning based on a reinforcement learning dataset including tool invocation scenarios to obtain the trained multimodal reasoning model; After completing the supervised training described above, this embodiment further optimizes the model through reinforcement learning, enabling it to adaptively discover effective tool usage strategies in a multi-task environment and achieve combined tool invocation. The tool enhancement reinforcement learning process mainly includes three parts: inference trajectory modeling, reward design, and policy optimization.

[0031] (1) Unlike traditional chain-based reinforcement learning that only includes textual thinking, this invention explicitly introduces tool call tokens and external observation tokens into the inference trajectory modeling. For a given user query Q and image I, the inference trajectory before the k-th step can be represented as: In the k+1 step, the model is based on I , Q and current trajectory Generate the next step of thought With tool call This drives problem-solving, and the strategy is represented as follows: This process alternates between thinking, tool calls, and observation until a final answer is generated or the maximum number of tool calls is reached. To avoid inefficiency or circular behavior, this invention triggers early termination when a tool call is completely duplicated from a previous call. In implementation, this invention uses a pre-defined special marker in the model output. <think>,<tool_call> and <answer>These are respectively identified as the thought process, tool usage, and final answer. When detected...<tool_call> During the tagging process, the system parses the specific tool name and parameters from the output, calls the external function, and writes the return result to the file. <obs>This field is appended to the current trajectory. All observation tokens participate in decision-making only as state information and are not included in loss calculation.

[0032] (2) Unlike schemes that only provide sparse rewards based on correct or incorrect answers, this invention constructs a rule-driven fine-grained reward function that decomposes the overall reward into multiple signals to guide the model in learning more effective tool-use strategies. Final reward S It consists of three parts: format rewards S format, answer rewards S ans and tool usage rewards S tool. Among them, format rewards. S The `format` property is used to check if the output contains special markers arranged in the expected order; Answer reward S In multiple-choice questions, ANS scores based on whether the predicted answer matches the actual answer. In tasks requiring a segmented mask as the answer, scores are awarded in segments based on the IoU between the predicted and actual masks. (Tool usage reward) S The tool is only awarded when the model outputs the correct answer and the external perception tool is invoked at least once during the reasoning process. This encourages the model to use the tool appropriately when it helps complete the task, rather than calling it randomly or redundantly. The overall reward can be written as: The specific breakdown method can be adjusted according to different tasks.

[0033] (3) Policy Optimization: Based on the above reasoning modeling and reward design, this invention constructs a reinforcement learning objective function based on the policy optimization idea of ​​intra-group relative advantage normalization, and optimizes the policy parameters. Update. For the tool-enhanced reinforcement learning dataset D rl samples ( I,Q,A This invention first addresses the shortcomings of the old strategy. The following generation G Candidate inference trajectories And calculate their respective rewards. The reinforcement learning loss function is defined as: in in, Indicates the first i One reasoning path, This refers to the total length of the tokens excluding tool outputs. For the current strategy and the old strategy in the 1st n The probability ratio of each step action; Ai The advantage function, normalized to the within-group mean and standard deviation, reflects the relative quality of the trajectory within the group. By minimizing this loss function, the model tends to retain high-reward trajectories and suppress low-quality inferences, thereby gradually learning to utilize tools for key visual cues detection and expert-level efficient inference.

[0034] Furthermore, the dataset construction process in steps S2-S4 above includes: In the process of constructing the dataset, this invention follows the following three principles: (1) It covers a variety of medical image task types and imaging modalities, ensuring that the multimodal inference model can adapt to the analysis needs of different disease types and anatomical structures; (2) Focus on collecting and designing scenarios that can significantly improve reasoning performance through tool calls, so as to highlight the actual effect of tools in enhancing reasoning; (3) Provide as complete fine-grained annotations as possible, including natural language questions, tool call instructions and corresponding observation information returned by the tools, to support cold start supervised fine-tuning, self-reflective fine-tuning and tool-enhanced reinforcement learning training.

[0035] In specific embodiments, the present invention can utilize publicly available medical image segmentation datasets containing large-scale "image-segmentation mask-semantic label" triples, covering multiple imaging modalities and various anatomical or lesion objects; at the same time, it can combine high-quality datasets with "image-mask-text description" triples to provide fine-grained lesion attribute descriptions.

[0036] To ensure that tool calls are truly necessary in the problem-solving process, this invention employs a general multimodal generative model that automatically constructs question-answer pairs given the input image, mask, and corresponding semantic description. The generated question requires explicit localization of a specified mask region to obtain the correct answer, ensuring that the model must acquire local visual information through tool calls. Subsequently, this invention performs automated screening and manual verification of the generated samples, eliminating data where the answer does not match the image content or cannot be reliably verified, thereby ensuring the validity and accuracy of the data.

[0037] When constructing tool-enhanced inference data, this invention employs a large-scale multimodal inference model, generating example inference paths according to a cyclical mechanism of "thinking – tool invocation – observation". When the model generates a tool invocation instruction, the system provides a realistic mask or locally cropped image for the corresponding tool invocation, thereby obtaining a complete and high-quality tool-enhanced inference trajectory.

[0038] Subsequently, this invention filters the generated data based on the validity of the trajectory format and the correctness of the answer, specifically including: (1) Retain those samples that give incorrect results when answering directly in a single round, but can be corrected by using tools and fine-grained checking of local areas, so as to highlight the actual role of tool use; (2) At the same time, some questions that can be answered correctly without tools are retained for training the model. Tools are only called when necessary to avoid over-reliance on tools.

[0039] Through the aforementioned systematic data construction process, this invention obtains high-quality question-answering samples covering multiple task types and imaging modalities, and performs cross-validation using manual annotation to ensure data reliability. In implementation, the dataset can be divided into a subset Dcold for cold-start supervised fine-tuning, a subset Drl for tool-enhanced reinforcement learning, and a subset Dtest for in-domain testing. This invention does not limit the specific data size or partition ratio.

[0040] Step S5: Obtain the medical image to be processed and the corresponding natural language question; Natural language problems refer to specific task-oriented problems used to describe the target or lesion to be analyzed in medical images, including: (1) Lesion identification problems, such as "Is there a tumor region in this image?" or "Please indicate the location of the abnormal structure in the image"; (2) Lesion segmentation problems, such as "Please mark the precise boundaries of liver lesions in the image" or "Please segment lung nodules"; (3) Problems involving the description of lesion characteristics, such as "Please describe the shape, size and boundary characteristics of the lesion"; (4) Diagnostic reasoning problems, such as "determine the possible disease type and severity based on the image".

[0041] Natural language questions can be obtained through manual annotation or automatic generation. Their content must be directly related to the visual information of the corresponding medical image and be able to obtain accurate answers or segmentation results through model reasoning and tool calls.

[0042] Step S6: Input the medical image to be processed and the corresponding natural language question into the trained multimodal reasoning model to obtain the processed output, including the natural language thinking generated by the model, tool call instructions, observation information returned by external tools, and the final medical image analysis results.

[0043] Example 2: This embodiment provides a medical image tool enhancement processing system based on self-reflective reinforcement learning, including: The data construction module is used to collect and organize medical image data, mask data, and corresponding natural language questions, answers, and tool call inference trajectories according to a preset data construction strategy, and to build a multi-stage training dataset for cold start supervised fine-tuning, self-reflective fine-tuning, and tool-enhanced reinforcement learning. The Cold Start Supervision and Fine-Tuning module is used to minimize the negative log-likelihood between the inference token and the tool call token based on the labeled inference path in the cold start dataset, so that the multimodal inference model has basic tool call norms and structured inference capabilities.

[0044] The self-reflective fine-tuning module is used to construct self-reflective training samples based on the differences between the erroneous inference paths generated by the model in early checkpoints and the correct inference paths generated in later checkpoints, and to perform self-reflective reinforcement fine-tuning on the model to correct systematic errors in the tool invocation strategy, optimize inference logic, and improve the model's self-correction ability.

[0045] The tool-enhanced reinforcement learning module is used to enhance the multimodal reasoning model based on reinforcement learning datasets containing tool-calling scenarios. By calculating rewards for answer accuracy, format conformity, and reasonableness of tool calls, and combining policy gradient optimization methods, the module further improves the model's autonomous tool-calling ability and reasoning stability in real medical tasks.

[0046] The inference execution module is used to acquire the medical image to be processed and the corresponding natural language question, execute the trained multimodal inference model, generate step-by-step natural language thinking, tool call instructions and corresponding tool observation results, so as to obtain the final answer or segmentation result.

[0047] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.< / obs> < / answer> < / think>

Claims

1. A medical image enhancement method based on self-reflective reinforcement learning, characterized in that, Includes the following steps: A multimodal inference model is constructed, which includes an image encoder, a text encoder, an inference module, a tool interface module, and an observation encoding unit; Based on a cold start dataset with tool call trajectory annotations, the multimodal inference model is subjected to cold start supervised fine-tuning. In cold start supervised fine-tuning, early error trajectories and later correct trajectories generated by the same input at different training checkpoints are collected. Trajectory pairs that meet the self-reflection characteristics are constructed as self-reflection training samples, and the multimodal reasoning model after cold start supervised fine-tuning is self-reflected based on the self-reflection training samples. After completing the self-reflective fine-tuning, the multimodal reasoning model is trained using tool-enhanced reinforcement learning based on the reinforcement learning dataset to obtain the trained multimodal reasoning model. Acquire the medical images to be processed and their corresponding natural language questions; The medical image to be processed and the corresponding natural language question are input into the trained multimodal reasoning model to obtain the processed output, which includes the natural language thinking generated by the model, the tool calling instructions, the observation information returned by the external tool, and the final medical image analysis results.

2. The medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 1, characterized in that, The image encoder is a deep convolutional neural network used to process medical images and extract medical image features; The text encoder is used to perform word-level and sentence-level semantic encoding on natural language problems to obtain corresponding text semantic features; The inference module is an autoregressive sequence generation network based on a large language model, used to generate inference paths based on medical image features, text semantic features, and observation features. ; The tool interface module is an instruction parsing and tool scheduling unit, used to parse the tool call instructions generated by the inference module and send call requests to external image analysis tools. The observation encoding unit is an observation processing network used to receive observation information returned by external image analysis tools and encode it into observation features that can be read by the inference module.

3. The medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 2, characterized in that, The reasoning path Represented as: in, For the first n The natural language thinking content generated step by step For the first n The tool call instructions generated step by step, The observation features corresponding to the observation results returned by the external image analysis tool according to the tool call command; in the... n During step-by-step reasoning, the reasoning module is based on the currently input medical image. I ,question Q and before Step-by-step reasoning history autonomously determines the next step of natural language thinking. Tool call commands , is represented as: in, Indicates the preceding The history of step-by-step reasoning Represents the mapping function for the inference module; The reasoning module updates the reasoning path step by step through a loop mechanism of thinking – tool invocation – observation, until the termination step. The system generates a definite answer or reaches the preset maximum context length or maximum number of interaction rounds.

4. The medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 2, characterized in that, The external image analysis tool includes a first image processing tool, a second image processing tool, and a third image processing tool; The first image processing tool is a location-cue-based image segmentation tool, used to receive the target region bounding box provided by the multimodal inference model as input and output a fine segmentation mask of the target object; The second image processing tool is a semantic description-based image segmentation tool, which receives images and text prompts as input and outputs a segmentation mask of the corresponding target region, so as to achieve target region localization without the need for the model to explicitly predict the bounding box; The third image processing tool is an image local magnification tool, which is used to receive an image and a bounding box or segmentation mask as input. When the input is a bounding box, it outputs a magnified image of the corresponding region. When the input is a segmentation mask, it outputs a magnified image and explicitly draws the target outline in the image to highlight the morphological features of the lesion.

5. The medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 1, characterized in that, The cold start dataset includes multiple samples. Each sample includes a medical image to be processed, a corresponding natural language question, a true answer, and an inference path with tool call trajectory annotations. The tool call trajectory annotations record the natural language thinking generated in each step of inference, the tool call instructions, and the observation results returned by the corresponding tool.

6. The medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 1, characterized in that, The cold-start supervised fine-tuning of the multimodal inference model based on the cold-start dataset with tool call trajectory annotations specifically includes: For each sample in the cold start dataset, the input sample contains a medical image, a natural language question, and the corresponding labeled inference path. At most multimodal reasoning models will generate inference paths. With annotation path Perform step-by-step comparisons; think based on the natural language generated at each step. Tool call commands With annotation content and To assess consistency, the cold start supervision loss function is calculated using the following formula: in, For the first n The natural language thinking content generated step by step For the first n The tool call instructions generated step by step, The observation results returned by the tool call command in the labeled path; This represents the total number of tokens for natural language thinking and tool invocation instructions in the inference path generated by the model; Indicates the first n Step into natural language thinking The number of tokens; Indicates the first n Step tool call command The number of tokens; Indicates the total number of steps in the reasoning path; This represents the expectation of all samples in the cold start dataset; Indicates the first n Step-by-step model generates natural language thinking Tool call commands The conditional probability is determined by the inference module of the multimodal inference model based on medical image features, text semantic features, and prior information. The autoregressive sequence generation method of step observation features is used to calculate the sequence. Indicates the first n The inference history generated by the model before the step; The true answer for the sample; The cold start supervision loss function; By minimizing the cold start supervision loss function This enables supervised fine-tuning of cold starts for multimodal inference models.

7. The medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 1, characterized in that, In the cold-start supervised fine-tuning process, early error trajectories and later correct trajectories generated by the same input at different training checkpoints are collected. Trajectory pairs that meet the self-reflective characteristics are constructed as self-reflective training samples. Based on these self-reflective training samples, the multimodal inference model after cold-start supervised fine-tuning is subjected to self-reflective fine-tuning, specifically including: During the training process, for each sample in the cold start dataset, the inference path generated by the multimodal inference model at the early training checkpoint and the later training checkpoint is recorded respectively. The early training checkpoint is the training state when the model has not yet fully converged, and the later training checkpoint is the state when the model has been trained to convergence or is close to convergence. The inference path generated by the early training checkpoints is denoted as: ;in, These represent the early training checkpoints. The generated natural language thinking content, tool call commands, and observation features; This represents the total number of steps in the early reasoning path; The inference path generated by the later training checkpoints is denoted as: ;in, These represent the checkpoints in the later stages of training. The generated natural language thinking content, tool call commands, and observation features; Indicates the total number of steps in the later reasoning path; Determine if an arbitrary step index exists. and This makes the following formula: Established; among them, For the early reasoning path The tool call instructions generated step by step, For the later reasoning path The tool call instructions generated step by step, This is an indicator function for whether the final answer corresponding to the early reasoning path is correct; a value of 0 indicates an error, and a value of 1 indicates a correct answer. This is an indicator function for whether the final answer corresponding to the later reasoning path is correct; a value of 0 indicates an error, and a value of 1 indicates a correct answer. If it exists, then the condition will be satisfied. and Alignment and integration constitute a self-reflection training sample ; Self-reflection training samples The multimodal inference model, fine-tuned with cold-start supervision, is trained by minimizing the cold-start supervision loss function. This enables self-reflective fine-tuning of multimodal reasoning models.

8. The medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 7, characterized in that, The conditions that will be met and Alignment and integration constitute a self-reflection training sample Specifically, it includes: For each early error step Find the corresponding modification steps in the later inference path The modification refers to the existence of corresponding steps in the later path that satisfy... Or the early path leads to the wrong answer while the later path leads to the correct answer; Natural language thinking generated in early steps Correct tool call instructions for later steps This will combine to form new reflective steps, while also incorporating the observational features corresponding to later steps. Incorporate this reflection step; For steps in the early path that were not modified and corresponded to the correct final answer, their natural language thinking, tool call instructions and observation characteristics were preserved and directly added to the self-reflective training samples. Following the chronological order of the original reasoning path, all reflection steps are sequentially concatenated with the correct, unmodified steps to obtain a complete self-reflection training sample.

9. The medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 1, characterized in that, The step of training the multimodal reasoning model using tool-enhanced reinforcement learning based on a reinforcement learning dataset to obtain the trained multimodal reasoning model specifically includes: For each sample in the reinforcement learning dataset, including medical images I Corresponding natural language problem Q and the real answer A In the current policy parameters of the multimodal inference model Multiple candidate inference trajectories are generated below. ,in, They represent the first The first candidate inference trajectory n The generated natural language thinking content, tool call commands, and observation features; For the first The total number of steps for each candidate inference trajectory; During the generation of each reasoning trajectory, the multimodal reasoning model alternately generates natural language thinking according to the thinking-tool invocation-observation cycle mechanism. Tool call commands Based on the current image, the problem, and the existing trajectory Inferring the next output, represented as ; Calculate the comprehensive reward for each trajectory. , is represented as: in, The reward is given for the answer, used to measure the difference between the model's output answer and the actual answer. A The degree of matching is determined. For classification tasks, a fixed score is awarded when the predicted answer matches the true answer. For segmentation masking tasks, the score is awarded in segments based on the IoU value between the predicted mask and the true mask. The format reward is used to ensure that the structure of the reasoning trajectory conforms to the expected order. Specifically, it is checked whether the output trajectory includes natural language thinking, tool calls and final answer marking in sequence. Points are deducted for missing or incorrectly ordered markings to ensure that the trajectory generation conforms to the specified format. The tool usage reward is used to encourage the model to call external tools under reasonable circumstances. The reward is given only when the model generates the correct answer and calls the external tool at least once during the reasoning process. The reward value is positively correlated with the number of tool calls and the contribution of the tool calls to the completion of the task. Based on the overall reward for each trajectory The reinforcement learning loss function is calculated, and the policy parameters of the multimodal inference model are updated by minimizing the reinforcement learning loss function to obtain the trained multimodal inference model.

10. A medical image tool enhancement processing method based on self-reflective reinforcement learning according to claim 9, characterized in that, The reinforcement learning loss function is defined as follows: in, To reinforce the learning loss function; This indicates the reinforcement learning dataset. All samples and in the old strategy The following generated G Take the expected value of each candidate inference trajectory; This represents the medical image input in the sample; This represents the natural language question input in the sample; This represents the true answer of the sample; Indicates the first i The total number of all tokens in the inference trajectory, excluding observations returned by the tool; Indicates the current strategy parameters Next, the i Trajectory number n The probability of the step action compared to the old strategy The ratio of the probabilities of corresponding actions; Indicates the first i The trajectory in the first n All previously generated actions and observed historical trajectories; Indicates the first i The advantage function of a trajectory; Indicates the first i The overall reward for each trajectory; , They represent G The mean and standard deviation of the overall reward for each candidate reasoning trajectory; This is the clipping function; These are the trimming parameters.

Images