Medical image report generation method and system, electronic device and storage medium

Abstract

Embodiments of the present application provide a medical image report generation method and system, electronic equipment and storage medium, relating to the technical field of artificial intelligence. The method comprises: obtaining a plurality of candidate reasoning paths by reasoning intelligent agents on real medical image reports; performing report deduction on the candidate reasoning paths and reference path reports by deduction intelligent agents to obtain deduced medical image reports; filtering the candidate reasoning paths according to the deduced medical image reports and reference medical image reports by verification intelligent agents to obtain retained reasoning paths and determine retained medical image reports; training a model weight of a medical image report generation model according to the retained reasoning paths, the retained medical image reports and the reference medical image; and constructing a medical reasoning intelligent agent based on the model weight, which is used to generate a target reasoning path and a target medical image report based on a target medical image. The present application can significantly improve the quality of image reports and realize verifiable report generation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, system, electronic device and storage medium for generating medical image reports. Background Technology

[0002] In recent years, medical report generation technologies have made significant progress (such as R2GenGPT, PromptMRG, etc.), but existing technologies still lack verifiable evidence to support diagnostic statements in the generated reports.

[0003] Existing technologies such as R2GenGPT use traditional linguistic cross-entropy modeling (e.g., CE) to train on open medical image-text pairs such as IU X-Ray and MIMIC, which have the following drawbacks: the generated reports usually only provide conclusions without providing verifiable reasoning chains or evidence paths, resulting in insufficient clinical credibility; they cannot explicitly model the clinical reasoning process, leading to the inability to achieve verifiability in the end-to-end trained report generation; traditional linguistic cross-entropy modeling results in a lack of explicit causal dependency modeling between reasoning steps and the final conclusion, which is prone to inconsistencies and logical drift. Summary of the Invention

[0004] The main objective of this application is to propose a medical image report generation method, a medical image report generation system, an electronic device, and a computer-readable storage medium, which can significantly improve the quality of image reports and externalize the thought process in medical report generation to achieve verifiable report generation.

[0005] To achieve the above objectives, a first aspect of this application provides a method for generating medical image reports, the method comprising: The reasoning agent infers from the prompt words and real medical image reports to obtain multiple candidate reasoning paths. Each candidate reasoning path represents the process of reasoning from medical images to obtain the real medical image report. By performing report deduction on each candidate reasoning path and reference path report pair by a deductive agent, a deductive medical image report is obtained. The reference path report pair includes a reference reasoning path and a reference medical image report. The verification agent filters the candidate inference paths based on the deduced medical image report and the reference medical image report to obtain the retained inference path, and the reference medical image report corresponding to the retained inference path is used as the retained medical image report. Based on the preserved inference path, the preserved medical image report, and the reference medical image corresponding to the preserved medical image report, a triplet is constructed, and the medical image report generation model is trained using the triplet to obtain the model weights; Based on the model weights, a medical reasoning agent is constructed, which is used to generate a target reasoning path and a target medical image report based on the target medical image.

[0006] Optionally, the step of constructing a medical reasoning agent based on the model weights includes: Initialize the general agent based on the model weights; Obtain image report pairs, wherein the image report pairs include sample medical images and sample medical image reports; The old strategy is used to infer the sample medical images to obtain the output results, which include the output inference path and the output medical image report. Based on the sample medical image report, the old strategy, and the output results, the general agent is causally aligned to obtain the causal alignment loss function. The general agent is updated according to the alignment loss function to obtain the medical reasoning agent.

[0007] Optionally, the step of performing causal alignment on the general agent based on the sample medical image report, the old strategy, and the output result to obtain a causal alignment loss function includes: The output result is calculated based on the sample medical image report to obtain a scalar reward for the output result; The advantage estimate for each output is obtained by standardizing the scalar reward of the multiple outputs. Based on the parameters of the general agent, a policy is formulated to obtain the current policy, and the ratio of the current policy to the old policy is calculated to obtain the policy ratio. Based on the advantage estimate and the policy ratio, the truncated agent objective function is calculated; The KL divergence is calculated based on the current strategy and the old strategy to obtain the strategy divergence term; The causal alignment loss function is obtained by fusing the agent objective function and the policy divergence term.

[0008] Optionally, the step of calculating a reward for the output result based on the sample medical image report to obtain a scalar reward for the output result includes: Separator detection is performed on the output inference path and the output medical image report in the output results to obtain a format reward; The output medical image report is evaluated for similarity based on the sample medical image report, and a similarity reward is obtained; Based on the sample medical image reports, the longest common subsequence of the output medical image reports is evaluated to obtain an information rate coverage bonus. The scalar reward is obtained by fusing the format reward, the similarity reward, and the information rate coverage reward.

[0009] Optionally, the step of performing report deduction on each candidate inference path and the reference path report pair by the deductive agent to obtain a deductive medical image report includes: Get report style guidance keywords; The deductive agent generates a report based on the candidate inference path, the reference path report pair, and the report style guidance prompts, thereby obtaining the deductive medical image report.

[0010] Optionally, the step of filtering the candidate inference paths by the verification agent based on the deduced medical image report and the reference medical image report to obtain the retained inference paths includes: The verification agent performs entity extraction on the deduced medical image report to obtain a first entity set, and performs entity extraction on the reference medical image report to obtain a second entity set. Based on the correspondence between the first entity in the first entity set and the second entity in the second entity set, the entity semantic consistency score is calculated. The candidate reasoning paths are filtered based on the entity semantic consistency score to obtain the retained reasoning paths.

[0011] Optionally, training the medical image report generation model using the triples to obtain model weights includes: The reasoning path in the triples is linked with the medical image report to obtain the conditional generation target sequence; The medical image report generation model infers the images in the triplet to obtain a predictive inference path, generates a predictive medical image report based on the images in the triplet and the predictive inference path, and links the predictive inference path and the predictive medical image report to obtain a conditional generation prediction sequence. The target sequence and the predicted sequence generated under the conditions are compared in a domain to obtain a domain comparison loss function. The model parameters of the medical image report generation model are then adjusted based on the domain comparison loss function to obtain the model weights.

[0012] To achieve the above objectives, a second aspect of this application provides a medical image report generation system, the system comprising: The reasoning agent module is used to reason about prompt words and real medical image reports through the reasoning agent to obtain multiple candidate reasoning paths. Each candidate reasoning path represents the process of reasoning from medical images to obtain the real medical image report. The deductive agent module is used to perform report deduction on each candidate reasoning path and path report pair to obtain a deductive medical image report. The path report pair includes a reference reasoning path and a reference medical image report. The verification agent module is used to filter the candidate inference paths based on the deduced medical image report and the reference medical image report, obtain the retained inference path, and use the reference medical image report corresponding to the retained inference path as the retained medical image report; The domain alignment module is used to construct triples based on the preserved inference path, the preserved medical image report, and the reference medical image corresponding to the preserved medical image report, and to train the medical image report generation model using the triples to obtain model weights; The causal alignment module is used to construct a medical reasoning agent based on the model weights. The medical reasoning agent is used to generate a target reasoning path and a target medical image report based on the target medical image.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0015] This application proposes a method, system, electronic device, and computer-readable storage medium for generating medical image reports. First, this application utilizes a reasoning agent to mine the hidden thought processes in real medical image reports, i.e., candidate reasoning paths. Next, a deductive agent uses context learning with reference path-report pairs to reconstruct the mined thought processes into a deductive medical image report that conforms to medical standards. Then, a verification agent filters the candidate reasoning paths to obtain retained reasoning paths, eliminating unreasonable ones. Next, a medical image report generation model is trained based on a triple constructed from the retained reasoning paths, reports, and images to obtain model weights. Finally, based on the model weights, a medical reasoning agent is constructed, which can be used to generate target reasoning paths and target medical image reports based on target medical images. Therefore, this application can significantly improve the quality of image reports and externalize the thought processes in medical report generation to achieve verifiable report generation.

[0016] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0017] Figure 1 This is an optional flowchart of the medical image report generation method provided in the embodiments of this application; Figure 2 yes Figure 1 Flowchart for step 102; Figure 3 yes Figure 1 Flowchart for step 103; Figure 4 yes Figure 1 Flowchart for step 104; Figure 5 yes Figure 1 Flowchart for step 105; Figure 6 yes Figure 5 Flowchart for step 504; Figure 7 This is a block diagram of the module structure of the medical image report generation system provided in the embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and applications to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert devices. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0022] Backpropagation: The general principle of backpropagation is as follows: The training data is input into the input layer of the neural network, passes through the hidden layer, and finally reaches the output layer of the neural network to output the result; Since there is an error between the output result of the neural network and the actual result, the error between the estimated value and the actual value is calculated, and this error is backpropagated from the output layer to the hidden layer until it reaches the input layer; During the backpropagation process, the values ​​of various parameters are adjusted according to the error; The above process is iterated continuously until convergence.

[0023] The Multi-Agent Chain-of-Thought Reasoning (MACoTR) framework formalizes verifiable medical report generation as a specialized task and aims to address the inherent limitations of chain-of-thought reasoning. This application generates high-quality verifiable chain-of-thought data without relying on a single gold standard answer by embedding self-consistency into a structured multi-agent workflow, and facilitates a gradual transfer from a general agent to a medical reasoning agent using a dual-alignment learning strategy. It provides a pathway to verifiable medical reports. Extensive experiments on two representative medical report generation benchmarks demonstrate its robust verifiable report generation capabilities.

[0024] Current paradigms learn from image-report pairs without explicitly representing the clinical reasoning process, leading to inconsistencies between visual evidence and textual conclusions. The resulting reports lack verifiable diagnostic evidence. This deficiency undermines clinical trust and introduces diagnostic risks. Due to privacy concerns and annotation costs associated with medical data, clinical practice typically records diagnostic conclusions rather than step-by-step reasoning processes. Consequently, datasets lack process-level annotations that are strictly consistent with real reports, resulting in a lack of process supervision during model training.

[0025] Thought chain reasoning externalizes diagnostic logic by presenting step-by-step reasoning during report writing, thereby enabling verifiable medical report generation. However, despite its pressing clinical value, verifiable thought chain reasoning suffers from two limitations: 1) Lack of thought chain supervision in medical report data. Clinical practice typically records diagnostic conclusions rather than step-by-step reasoning processes, resulting in datasets lacking rigorous process-level annotations consistent with real reports, leading to a lack of process supervision during model training. Even with the introduction of thought chain cues during decoding, mismatches between general medical knowledge and patient-specific imaging evidence, as well as the propagation of noise, can lead to error accumulation and amplify illusions. 2) Lack of explicit causal dependencies between reasoning steps. When reasoning steps lack explicit causal connections, the generated interpretations can be inconsistent between local imaging observations and the final diagnostic statement. This limitation causes the model to rely on superficial textual relevance rather than interpreting visual evidence with causal connections. As these inconsistencies accumulate along the reasoning chain, they cause the logic to gradually deviate, weakening the verifiability of thought chain-based medical report generation. Therefore, how to achieve verifiable report generation modeling without introducing additional manual annotations is a key research topic.

[0026] Existing medical image report generation technologies can also employ an end-to-end paradigm, extracting global or local visual features through a visual encoder (such as CNN or Vision Transformer) and then using a text decoder to generate a complete report. Alternatively, they can utilize Large Language Models (LLM) or Visual Language Models (VLM) as generators, enhancing clinical readability through prompt templates, domain vocabulary injection, or instruction fine-tuning. While these methods are effective in improving report fluency and coverage, they still suffer from structural deficiencies regarding verifiability and traceable reasoning, often facing the following problems: 1) Medical report datasets typically do not contain inference chain annotations that are strictly consistent with real reports, and the lack of process supervision during model training makes it easy for the inference chain to become disconnected from real image evidence. 2) When deviations occur in early steps of the inference chain, errors accumulate along the chain and affect the final diagnostic statement, amplifying illusions and logical drift. 3) Single-agent thought chain generation lacks a consistency verification mechanism for medical entities and report content, resulting in low-quality generated thought chain data that is prone to introducing noise during training.

[0027] Existing medical image report generation technologies can be broadly categorized into three types: 1) End-to-end encoder-decoder models: These models extract image features using convolutional neural networks or visual Transformers, and then decode them using RNNs or Transformers to generate reports. They often incorporate attention mechanisms, hierarchical decoding, memory modules, or disease atlases to enhance completeness. 2) Retrieval enhancement or prompt enhancement models: These models retrieve similar cases or key phrases from a training database as prompts, guiding the generator to output reports that more closely resemble clinical descriptions. 3) Report generation incorporating Large Language Models (LLM) or Visual Language Models (VLM): These models utilize the expressive and reasoning priors of general language models, improving fluency and terminology compliance through the injection of instructions, templates, or medical knowledge. Some works attempt to introduce thought chains.

[0028] The methods described above are effective in improving report fluency and coverage, but they often face the following problems regarding verifiability: 1) Most methods are trained only on paired report and image data, without verifiable reasoning chains. Even with the introduction of thought chain cues during the decoding stage, the lack of process supervision during training leads to error accumulation and amplifies illusions. 2) There is a lack of data mechanisms to automatically construct high-quality reasoning chains from existing reports, making it difficult to train reliable reasoning without adding manual annotations. 3) The lack of training constraints on the causal consistency between reasoning steps and conclusions during training leads to inconsistencies between the reasoning chain and the report text, or logical drift.

[0029] To overcome the above problems, MACoTR proposes a ternary reasoning agent checker to simulate the clinical diagnostic process to generate high-quality verifiable thought chain reasoning data. It also proposes a dual alignment learning mechanism to progressively train a general agent to model the causal dependencies between reasoning steps, thereby improving verifiable diagnostic reasoning capabilities.

[0030] The medical image report generation method of this application embodiment can be executed by a server alone, by a terminal alone, or by both a terminal and a server. Furthermore, the medical image report generation method provided in this application embodiment can also be software running on a server. The server can be configured as a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed device. It can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the medical image report generation method, but is not limited to the above forms.

[0031] This application provides a method for generating medical image reports, a system for generating medical image reports, an electronic device, and a computer-readable storage medium. The specific embodiments are described below. First, the method for generating medical image reports in this application is described.

[0032] See Figure 1 A method for generating a medical image report according to one embodiment of this application includes: Step 101: The reasoning agent performs reasoning on the prompt words and real medical image reports to obtain multiple candidate reasoning paths; Step 102: The deductive agent performs report deduction on each candidate reasoning path and reference path report pair to obtain the deductive medical image report. The reference path report pair includes the reference reasoning path and the reference medical image report. Step 103: The verification agent filters candidate inference paths based on the deduced medical image report and the reference medical image report to obtain the retained inference path, and the reference medical image report corresponding to the retained inference path is used as the retained medical image report. Step 104: Construct triples based on the preserved inference path, the preserved medical image report, and the reference medical image corresponding to the preserved medical image report, and use the triples to train the medical image report generation model to obtain the model weights; Step 105: Based on the model weights, a medical reasoning agent is constructed. The medical reasoning agent is used to generate a target reasoning path and a target medical image report based on the target medical image.

[0033] The advantage of the embodiments described in steps 101 to 105 above is that, firstly, a reasoning agent is used to mine the hidden thought process of real medical image reports, i.e., candidate reasoning paths. Then, a deductive agent is used to reconstruct the mined thought process into a deductive medical image report conforming to medical standards through context learning of reference path report pairs. Next, a verification agent is used to filter the candidate reasoning paths, obtaining retained reasoning paths and filtering out unreasonable candidate reasoning paths. Then, a medical image report generation model is trained based on a triple constructed from the retained reasoning paths, reports, and images to obtain model weights. Finally, based on the model weights, a medical reasoning agent is constructed, which can be used to generate target reasoning paths and target medical image reports based on target medical images. Therefore, this application can significantly improve the quality of image reports and externalize the thought process in medical report generation to achieve verifiable report generation.

[0034] In step 101, a reasoning agent performs reasoning on the prompts and the actual medical image report to obtain multiple candidate reasoning paths. The reasoning agent, which can be constructed using DeepSeek-R1, is used to uncover the hidden thought processes within the report. Each candidate reasoning path represents the process of reasoning from the medical image to obtain the actual medical image report.

[0035] In one example, the prompt is: You will act as a medical reasoning assistant, outputting at least three independent reasoning paths based on real medical imaging reports. Each path includes: preconditions, key imaging clues, a set of potential diagnoses, exclusions, necessary additional information, and a breakdown of the rationale for the final diagnosis. Please ensure that each path is verifiable, independent, and avoids duplication.

[0036] In one example, a real medical imaging report may include: chest imaging reports (chest X-ray, chest CT, thoracoscopic images); abdominal and pelvic imaging reports (liver, gallbladder, pancreas, spleen, kidneys, urinary system, pelvic organs); brain / nervous system imaging reports (MRI brain / spinal cord, CT head); and musculoskeletal system imaging reports (XR / CT / MRI bones and joints, musculoskeletal system).

[0037] In one embodiment, prompts and real medical image reports are input into the reasoning agent, which then outputs multiple reasoning paths as the basis for subsequent thought chain selection and optimization of downstream models. The process of obtaining candidate reasoning paths can be described as follows: , in, As a prompt word, For the i-th real medical image report, It is a reasoning intelligent agent. Let M be the m-th candidate inference path corresponding to the i-th real medical impact report. The total number of corresponding candidate inference paths. Each candidate inference path can be represented as: , to express The underlying step-by-step reasoning process.

[0038] In step 102, a deductive agent performs report deduction on each candidate inference path and reference path report pair to obtain a deductive medical image report. The reference path report pair includes a reference inference path and a reference medical image report. The deductive agent can be built based on DeepSeek-V3.

[0039] In one embodiment, reference is made to Figure 2 Step 102 may include: Step 201: Obtain report style guidance keywords; Step 202: The deductive agent generates a report based on the candidate inference path, the reference path report pair, and the report style guidance prompts, thus obtaining the deductive medical image report.

[0040] For example, for each candidate inference path Using a small sample set Generate deductive medical image reports The report deduction process is defined as follows: , in, for The corresponding derivational medical imaging report, each example is paired with Both are verified reference path report pairs. For the j-th reference reasoning path, Let J be the j-th reference medical imaging report, and J represent the total number of reference pathway report pairs. These are prompts for guiding report style.

[0041] The advantage of the embodiments of steps 201 to 202 described above is that the generated deductive medical image reports can conform to standard clinical reporting norms in style and wording.

[0042] In step 103, the verification agent filters candidate inference paths based on the deduced medical image report and the reference medical image report to obtain the retained inference path, and the reference medical image report corresponding to the retained inference path is taken as the retained medical image report. The verification agent can be constructed based on RaTE.

[0043] In one embodiment, reference is made to Figure 3 Step 103 may include: Step 301: Entity extraction is performed on the interpreted medical image report by the verification agent to obtain the first entity set, and entity extraction is performed on the reference medical image report to obtain the second entity set; Step 302: Based on the correspondence between the first entity in the first entity set and the second entity in the second entity set, calculate the entity semantic consistency score; Step 303: The candidate reasoning paths are filtered based on the entity semantic consistency score to obtain the retained reasoning paths.

[0044] Specifically, each medical imaging report will be interpreted Corresponding real medical image reports Comparison to obtain entity semantic consistency scores ,in measure and Entity-level consistency between them: , in, To verify the intelligent agent, These are the prompt words. The final retained candidate inference paths are determined by selecting the candidate inference path with the highest entity consistency score and applying a threshold criterion: , in, This is the number of the candidate inference path with the highest entity consistency score. The candidate inference path with the highest entity consistency score. The entity consistency score of the candidate inference path with the highest entity consistency score. (Threshold not reached.) The candidate inference paths will be resampled and re-evaluated for up to K rounds. If no valid candidate inference path is obtained after K rounds, the instance will be removed from the retention set. The retention set can be represented as: ,in, It is the number of instances that generate valid candidate inference paths within K rounds.

[0045] The advantage of the embodiments of steps 301 to 303 described above is that they can filter out low-quality candidate inference paths.

[0046] In step 104, a triplet is constructed based on the retained inference path, the retained medical image report, and the reference medical image corresponding to the retained medical image report. The triplet is then used to train the medical image report generation model to obtain the model weights.

[0047] In one embodiment, reference is made to Figure 4 Step 104 may include: Step 401: Link the reasoning path in the triple and the medical image report to obtain the conditional generation target sequence; Step 402: The images in the triples are reasoned through the medical image report generation model to obtain the prediction reasoning path. A predicted medical image report is generated based on the images in the triples and the prediction reasoning path. The prediction reasoning path and the predicted medical image report are linked to obtain the conditional generation prediction sequence. Step 403: Perform domain comparison between the conditionally generated target sequence and the conditionally generated predicted sequence to obtain the domain comparison loss function, and adjust the model parameters of the medical image report generation model according to the domain comparison loss function to obtain the model weights.

[0048] Specifically, to adapt the general agent to the medical field and facilitate the generation of clinically coherent and accurate reports, this application uses paired path-report data constructed through the aforementioned multi-agent approach for domain alignment. The resulting retention set is... For each reserved instance ,Will Its corresponding reference medical image Pairing is performed to form training triples. During training, a mini-batch is sampled. And based on the inference path in the triples and medical image reports, a conditionally generated target sequence is constructed: ,in, : The element-wise output typically corresponds to reasoning steps, evidence points, diagnostic labels, and necessary connecting text. This is achieved by maximizing the given... hour The model is optimized using conditional likelihood estimation. The mini-batch domain alignment loss function is: Where θ represents the trainable parameters, and t represents the current generated parameter. The t-th output position, This represents the output at time step t. P represents a previously generated partial sequence. Represents a given reference medical image Previously generated partial sequences In addition to possible time step information The conditional probability distribution.

[0049] The advantage of the above embodiments is that they enable domain alignment, which helps improve the accuracy of medical image report generation.

[0050] In step 105, a medical reasoning agent is constructed based on the model weights. The medical reasoning agent is used to generate a target reasoning path and a target medical image report based on the target medical image.

[0051] In one embodiment, reference is made to Figure 5 Step 105 may include: Step 501: Initialize the general agent based on the model weights; Step 502: Obtain image report pairs, which include sample medical images and sample medical image reports; Step 503: Use the old strategy to reason about the sample medical images to obtain the output results, which include the output reasoning path and the output medical image report. Step 504: Perform causal alignment on the general agent based on the sample medical image report, the old strategy, and the output results to obtain the causal alignment loss function; Step 505: Update the general agent according to the alignment loss function to obtain the medical reasoning agent.

[0052] In step 501, a general agent is initialized using the model weights obtained after training the medical image report generation model based on triples. This agent has the ability to generate reports based on images.

[0053] In step 502, the sample medical image and sample medical image report in the image report pair are paired and can be obtained from a database or public dataset.

[0054] In step 503, the old strategy is used to infer the sample medical images to obtain the output results. The output results include the output inference path and the output medical image report. For example, for each input... ,in Represents sample medical images, This indicates a prompt instruction, replacing the old strategy. The output obtained from the inference is sampled from a set of G outputs. Each output It includes output reasoning path and output medical image report.

[0055] In one embodiment, reference is made to Figure 6 Step 504 may include: Step 601: Calculate the reward for the output result based on the sample medical image report to obtain the scalar reward of the output result; Step 602: Standardize the scalar rewards of multiple outputs to obtain the advantage estimate for each output; Step 603: Based on the parameters of the general agent, perform policy formulation to obtain the current policy, and calculate the ratio of the current policy to the old policy to obtain the policy ratio; Step 604: Based on the advantage estimate and policy ratio, calculate the truncated agent objective function; Step 605: Calculate the KL divergence based on the current strategy and the old strategy to obtain the strategy divergence term; Step 606: The agent objective function and the policy divergence term are fused to obtain the causal alignment loss function.

[0056] In step 601, for each sampled output Calculate scalar reward , which is a weighted sum of complementary components.

[0057] In one embodiment, step 601 may include: performing delimiter detection on the output inference path and the output medical image report in the output result to obtain a format reward; performing a similarity assessment on the output medical image report based on the sample medical image report to obtain a similarity reward; performing a longest common subsequence assessment on the output medical image report based on the sample medical image report to obtain an information rate coverage reward; and fusing the format reward, similarity reward, and information rate coverage reward to obtain a scalar reward.

[0058] In one example, the scalar reward can be represented as: , in, This indicates a similarity reward (such as a BLEU-4 reward). This indicates information rate coverage bonus (such as ROUGE-L bonus). This represents the format reward, a binary indicator used to check whether the output uses a predefined special delimiter to separate the reasoning process and the final report. The output is 1 if both parts are correctly separated, and 0 otherwise. (Coefficient) , and Used to measure the contribution weight of the three reward items.

[0059] In step 602, relative advantages within the sampling group can be constructed by standardizing the rewards within the sampling group. For each sampling output The advantage estimate is calculated as follows: , in, This represents the advantage estimate. This indicates a scalar reward.

[0060] In steps 603 to 606, for each output The strategy ratio is defined as: ,in The current policy is parameterized by θ. After the PPO-style update, the truncated agent objective function is defined as: , in These are the clipping parameters. The causal alignment loss function is: , in, Denotes KL divergence, The strength of KL regularization is controlled. This objective function enhances the causal consistency of the inference chain, improves the verifiability of reports, and enables the general agent to progressively learn to become a medical reasoning agent, generating coherent and clinically relevant reasoning behavior.

[0061] In summary, the hyperparameters involved in the MACoTR framework in the experiment are as follows: the number of candidate inference paths generated M=4, and a small sample set. The number of examples J=5, threshold τ=0.7, maximum number of polls K=3, mini-batch domain alignment B=16, coefficients λFormat, λ1, and λ2 are 1, 0.8, and 0.2 respectively, and the pruning parameters are... =0.2, regularization parameter β=0.04.

[0062] This application achieves the following effects through the above implementation methods: 1) Without increasing manual stepwise reasoning annotation, it automatically constructs high-quality reasoning supervision data consistent with the reference report through a closed-loop mechanism of a ternary reasoning agent checker (including a reasoning agent, a deductive agent, and a verification agent), reducing illusion and inconsistent reasoning from the source; 2) Through progressive training of domain alignment and causal alignment, it explicitly strengthens the causal coherence between the reasoning chain and the final conclusion, enabling the generated report to have traceable and verifiable reasoning basis, improving clinical credibility and usability; 3) It outperforms SoTA models such as R2GenGPT, PromptMRG, and CMN in both report generation datasets; 4) The method is universal, requiring only paired datasets to use the medical image report generation method provided in this application.

[0063] Please see Figure 7 This application also provides a medical image report generation system that can implement the above-described medical image report generation method. Figure 7 This is a block diagram of the module structure of a medical image report generation system provided in an embodiment of this application. The system includes: The reasoning agent module 701 is used to reason about prompt words and real medical image reports through the reasoning agent to obtain multiple candidate reasoning paths. Each candidate reasoning path represents the process of reasoning from medical images to obtain real medical image reports. The deductive agent module 702 is used to perform report deduction on each candidate reasoning path and path report pair through the deductive agent to obtain a deductive medical image report. The path report pair includes a reference reasoning path and a reference medical image report. The verification agent module 703 is used to filter candidate inference paths based on the deduced medical image report and the reference medical image report, obtain the retained inference path, and use the reference medical image report corresponding to the retained inference path as the retained medical image report. The domain alignment module 704 is used to construct triples based on the preserved inference path, the preserved medical image report, and the reference medical image corresponding to the preserved medical image report, and to train the medical image report generation model using the triples to obtain the model weights; The causal alignment module 705 is used to construct a medical reasoning agent based on model weights. The medical reasoning agent is used to generate a target reasoning path and a target medical image report based on the target medical image.

[0064] It should be noted that the specific implementation of this medical image report generation system is basically the same as the specific implementation of the above-mentioned medical image report generation method, and will not be repeated here.

[0065] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned medical image report generation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0066] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the medical image report generation method of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0067] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described medical image report generation method.

[0068] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0069] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0070] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0071] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0072] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0073] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0074] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0075] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating medical image reports, characterized in that, The method includes: The reasoning agent infers from the prompt words and real medical image reports to obtain multiple candidate reasoning paths. Each candidate reasoning path represents the process of reasoning from medical images to obtain the real medical image report. By performing report deduction on each candidate reasoning path and reference path report pair by a deductive agent, a deductive medical image report is obtained. The reference path report pair includes a reference reasoning path and a reference medical image report. The verification agent filters the candidate inference paths based on the deduced medical image report and the reference medical image report to obtain the retained inference path, and the reference medical image report corresponding to the retained inference path is used as the retained medical image report. Based on the preserved inference path, the preserved medical image report, and the reference medical image corresponding to the preserved medical image report, a triplet is constructed, and the medical image report generation model is trained using the triplet to obtain the model weights; Based on the model weights, a medical reasoning agent is constructed, which is used to generate a target reasoning path and a target medical image report based on the target medical image.

2. The method according to claim 1, characterized in that, The medical reasoning agent constructed based on the model weights includes: Initialize the general agent based on the model weights; Obtain image report pairs, wherein the image report pairs include sample medical images and sample medical image reports; The old strategy is used to infer the sample medical images to obtain the output results, which include the output inference path and the output medical image report. Based on the sample medical image report, the old strategy, and the output results, the general agent is causally aligned to obtain the causal alignment loss function. The general agent is updated according to the alignment loss function to obtain the medical reasoning agent.

3. The method according to claim 2, characterized in that, The step of performing causal alignment on the general agent based on the sample medical image report, the old strategy, and the output result to obtain the causal alignment loss function includes: The output result is calculated based on the sample medical image report to obtain a scalar reward for the output result; The advantage estimate for each output is obtained by standardizing the scalar reward of the multiple outputs. Based on the parameters of the general agent, a policy is formulated to obtain the current policy, and the ratio of the current policy to the old policy is calculated to obtain the policy ratio. Based on the advantage estimate and the policy ratio, the truncated agent objective function is calculated; The KL divergence is calculated based on the current strategy and the old strategy to obtain the strategy divergence term; The causal alignment loss function is obtained by fusing the agent objective function and the policy divergence term.

4. The method according to claim 3, characterized in that, The step of calculating a reward for the output result based on the sample medical image report to obtain a scalar reward for the output result includes: Separator detection is performed on the output inference path and the output medical image report in the output results to obtain a format reward; The output medical image report is evaluated for similarity based on the sample medical image report, and a similarity reward is obtained; Based on the sample medical image reports, the longest common subsequence of the output medical image reports is evaluated to obtain an information rate coverage bonus. The scalar reward is obtained by fusing the format reward, the similarity reward, and the information rate coverage reward.

5. The method according to any one of claims 1 to 4, characterized in that, The step of performing report deduction on each candidate reasoning path and reference path report pair by a deductive agent to obtain a deductive medical image report includes: Get report style guidance keywords; The deductive agent generates a report based on the candidate inference path, the reference path report pair, and the report style guidance prompts, thereby obtaining the deductive medical image report.

6. The method according to any one of claims 1 to 4, characterized in that, The step of filtering candidate inference paths by the verification agent based on the deduced medical image report and the reference medical image report to obtain retained inference paths includes: The verification agent performs entity extraction on the deduced medical image report to obtain a first entity set, and performs entity extraction on the reference medical image report to obtain a second entity set. Based on the correspondence between the first entity in the first entity set and the second entity in the second entity set, the entity semantic consistency score is calculated. The candidate reasoning paths are filtered based on the entity semantic consistency score to obtain the retained reasoning paths.

7. The method according to any one of claims 1 to 4, characterized in that, The process of training the medical image report generation model using the triples to obtain model weights includes: The reasoning path in the triples is linked with the medical image report to obtain the conditional generation target sequence; The medical image report generation model infers the images in the triplet to obtain a predictive inference path, generates a predictive medical image report based on the images in the triplet and the predictive inference path, and links the predictive inference path and the predictive medical image report to obtain a conditional generation prediction sequence. The target sequence and the predicted sequence generated under the conditions are compared in a domain to obtain a domain comparison loss function. The model parameters of the medical image report generation model are then adjusted based on the domain comparison loss function to obtain the model weights.

8. A medical image report generation system, characterized in that, The system includes: The reasoning agent module is used to reason about prompt words and real medical image reports through the reasoning agent to obtain multiple candidate reasoning paths. Each candidate reasoning path represents the process of reasoning from medical images to obtain the real medical image report. The deductive agent module is used to perform report deduction on each candidate reasoning path and path report pair to obtain a deductive medical image report. The path report pair includes a reference reasoning path and a reference medical image report. The verification agent module is used to filter the candidate inference paths based on the deduced medical image report and the reference medical image report, obtain the retained inference path, and use the reference medical image report corresponding to the retained inference path as the retained medical image report; The domain alignment module is used to construct triples based on the preserved inference path, the preserved medical image report, and the reference medical image corresponding to the preserved medical image report, and to train the medical image report generation model using the triples to obtain model weights; The causal alignment module is used to construct a medical reasoning agent based on the model weights. The medical reasoning agent is used to generate a target reasoning path and a target medical image report based on the target medical image.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images