Reinforced learning post-training method and device for image report generation model

Abstract

The invention discloses a reinforcement learning post-training method and device for an image report generation model. The method comprises the following steps: acquiring a training sample containing a medical image, a corresponding text instruction and reference information; for any training sample, controlling the to-be-trained model to repeatedly generate N candidate output sequences to form a candidate output group under the condition that the medical image and the text instruction are kept unchanged; a reward value used for reinforcement learning optimization is calculated for each candidate output sequence, and the reward value is a combination of multiple rewards and at least comprises a content evaluation reward, a space evaluation reward, a semantic logic reward and a format reward; and determining a relative updating weight in the group based on the reward value corresponding to each candidate output sequence in the candidate output group, optimizing and updating model parameters by adopting a group relative strategy, and finishing post-training. According to the method, more distinguishable feedback signals are provided through intra-group relative optimization and combined rewards, illusion and front-back contradiction are restrained, report generation consistency is improved, and the problem of reward sparseness is relieved.

Description

Technical Field

[0001] This application relates to the field of medical artificial intelligence technology, and in particular to a reinforcement learning post-training method and apparatus for an image report generation model. Background Technology

[0002] Medical imaging examinations (such as X-rays and CT scans) are crucial for clinical diagnosis and treatment. Radiologists typically need to write structured or semi-structured imaging reports based on the imaging findings to describe lesion characteristics, provide imaging impressions, and support clinical decision-making. With the development of deep learning and multimodal large-scale models, technologies for automatically generating imaging reports based on image input are gradually being applied to scenarios such as report writing assistance, quality control prompts, and primary healthcare support, aiming to improve report generation efficiency and consistency while ensuring medical accuracy.

[0003] In existing technologies, image report generation typically employs a visual encoder and text decoder or a multimodal large model architecture. Supervised learning (such as maximum likelihood training and cross-entropy loss) is used to help the model fit the report text of the training samples. For performance evaluation, text similarity metrics such as BLEU, ROUGE, and CIDEr, or rule / model-based disease consistency metrics, are commonly used. However, image reports are highly specialized and subject to strong factual constraints. Simple text similarity is insufficient to accurately reflect the accuracy of key clinical facts, and the model may still exhibit hallucinatory descriptions, missed diagnoses / misdiagnoses, errors in left / right or anatomical descriptions, and internal inconsistencies within the report. To reduce hallucinations and enhance the constraint on clinical facts, some approaches introduce reinforcement learning post-training (such as PPO, preference / reward-based optimization, etc.), updating the policy model by scoring the generated results. However, these methods often rely on outcome-level rewards (such as overall similarity or whether the final diagnosis is correct), resulting in sparse and coarse-grained reward signals. This makes it difficult to effectively distinguish between errors at the disease type, left / right side, and more detailed anatomical locations, leading to insignificant differences in scores between samples and slow learning progress during training. Meanwhile, many existing solutions lack lesion localization constraints that are strongly tied to imaging evidence, making it difficult for models to generate conclusions based on image regions. In addition, there is a relative lack of evaluation mechanisms for the consistency between the model's reasoning process (thinking steps) and the final report, as well as for the conformity with medical common sense and the clarity of expression. This makes it difficult to effectively punish and correct problems such as seemingly reasonable reasoning but inconsistent conclusions, or correct conclusions but ambiguous / contradictory reasoning.

[0004] Therefore, in the reinforcement learning post-training of image report generation, the sparse reward signal and the difficulty in measuring fine-grained, dense medical facts, the insufficient constraints of lesion localization and imaging evidence, and the difficulty in ensuring the consistency and logical reliability of the reasoning process and the final report have become urgent technical problems that need to be solved. Summary of the Invention

[0005] This application provides a reinforcement learning post-training method and apparatus for an image report generation model, aiming to solve the problems in the existing technology of reinforcement learning post-training for image report generation, such as sparse reward signals and difficulty in fine-grained, dense medical fact measurement, insufficient lesion localization and image evidence constraints, and difficulty in ensuring the consistency and logical reliability of the reasoning process and the final report.

[0006] In a first aspect, a reinforcement learning post-training method for an image report generation model, the method comprising: Multiple training samples are acquired, each training sample including a medical image, and text instructions and reference information corresponding to the medical image, wherein the reference information is used to evaluate candidate output sequences; For any of the training samples, the medical image and the text instruction are used as input. While keeping the medical image and the text instruction unchanged, the image report generation model to be trained is controlled to repeatedly generate N candidate output sequences, where N is a positive integer greater than or equal to 2. The candidate output sequences are text sequences generated by the image report generation model to be trained for the medical image and the text instruction, and each candidate output sequence includes image report text. The N candidate output sequences constitute a candidate output group. For each candidate output sequence in the candidate output group, a reward value for reinforcement learning optimization is calculated. The reward value is a combination of multiple rewards, including a content evaluation reward based on disease-related content in the candidate output sequence, a spatial evaluation reward based on lesion spatial location-related content in the candidate output sequence, a semantic logic reward based on the consistency between the inference content of the candidate output sequence and the image report text and a preset semantic rule, and a format reward for constraining the candidate output sequence to meet a preset output structure. Based on the reward value corresponding to each candidate output sequence within the candidate output group, the relative update weight within the group for each candidate output sequence is determined, and the parameters of the image report generation model to be trained are updated using a group relative strategy optimization, thus completing the reinforcement learning training.

[0007] Optionally, in the above scheme, the reference information includes the image report text corresponding to the medical image, disease annotation information, and / or lesion spatial region annotation information.

[0008] Optionally, in the above scheme, the candidate output sequence further includes inference content, which is used to characterize the generation basis and inference process of the image report text, and the inference content includes disease-related analysis information and / or coordinate information of the lesion spatial region; The reasoning content and the image report text are distinguished from each other in the candidate output sequence by a preset separator.

[0009] Optionally, in the above scheme, before the reinforcement learning post-training, the method further includes: Multiple cold start training samples are selected from the plurality of training samples; For any of the aforementioned cold start training samples, a target output sequence for supervised training is constructed based on the medical images, text instructions, and reference information of the cold start training samples. The target output sequence includes inference content and image report text. The inference content is determined based on disease annotation information and / or lesion spatial region annotation information in the reference information. The target output sequence is used to supervise the training of the image report generation model to be trained.

[0010] Optionally, in the above scheme, the method further includes: The disease extraction model is used to extract disease entities and their corresponding location descriptions from the image report text of the candidate output sequence to obtain the extraction results. The extraction results are matched with the disease labeling information to obtain the matching results; Based on the matching results, the content evaluation reward is determined.

[0011] Optionally, in the above scheme, matching the extraction result with the disease labeling information to obtain a matching result includes: The extraction results are matched with the disease labeling information based on the disease category to obtain the first-level matching results; The extraction results are matched with the disease labeling information based on the disease category and left / right side information to obtain the second-level matching results; The extraction results are matched with the disease annotation information based on the disease category, left and right side information and fine-grained anatomical location information to obtain the third-level matching results; The process of determining the content evaluation reward based on the matching results includes: The first-level matching result, the second-level matching result, and the third-level matching result are weighted and aggregated according to a preset level weight to obtain a level aggregation result; The hierarchical aggregation results corresponding to different disease entities are weighted and aggregated according to preset disease weights to determine the content evaluation reward.

[0012] Optionally, in the above scheme, the method further includes: The spatial region information of the predicted lesion is parsed from the inference content of the candidate output sequence, and the spatial region information of the predicted lesion includes the coordinate information of the spatial region of the lesion. The predicted lesion spatial region information is matched with the lesion spatial region annotation information to obtain the spatial matching result; Based on the spatial matching results, the spatial evaluation reward is calculated according to the overlap index; wherein, the overlap index includes the intersection-union ratio index.

[0013] Optionally, in the above scheme, the method further includes: The reasoning content of the candidate output sequence and the image report text are input into the semantic evaluation model according to preset semantic rules, and the semantic evaluation model outputs a semantic score. The semantic score is determined as the semantic logic reward.

[0014] Optionally, in the above scheme, the method further includes: Determine whether the candidate output sequence satisfies a preset output structure; wherein, the preset output structure includes inference content and image report text, and the inference content and the image report text are distinguished by a preset separator; The reward format is determined based on the judgment result.

[0015] Secondly, a reinforcement learning post-training device for an image report generation model, the device comprising: The sample acquisition module is used to acquire multiple training samples, each of which includes a medical image, as well as text instructions and reference information corresponding to the medical image. The reference information is used to evaluate candidate output sequences. A candidate generation module is configured to, for any training sample, take the medical image and the text instruction as input, and, while keeping the medical image and the text instruction unchanged, control the image report generation model to be trained to repeatedly generate N candidate output sequences, where N is a positive integer greater than or equal to 2; wherein, the candidate output sequence is a text sequence generated by the image report generation model to be trained for the medical image and the text instruction, and the candidate output sequence includes image report text; the N candidate output sequences constitute a candidate output group; The reward calculation module is used to calculate the reward value for reinforcement learning optimization for each candidate output sequence in the candidate output group. The reward value is a combination of multiple rewards, including content evaluation reward based on disease-related content in the candidate output sequence, spatial evaluation reward based on lesion spatial location-related content in the candidate output sequence, semantic logic reward based on the consistency between the inference content of the candidate output sequence and the image report text and the preset semantic rules, and format reward for constraining the candidate output sequence to meet the preset output structure. The weight determination module is used to determine the relative update weight within the group for each candidate output sequence based on the reward value corresponding to each candidate output sequence within the candidate output group. The strategy optimization module is used to update the parameters of the image report generation model to be trained by using group relative strategy optimization, and to complete the training after reinforcement learning.

[0016] Compared with the prior art, this application has at least the following beneficial effects: Based on further analysis and research of existing technical problems, this application recognizes that existing technologies, in the reinforcement learning post-training stage of image report generation, suffer from sparse reward signals, difficulty in fine-grained and dense medical fact measurement, insufficient constraints on lesion localization and imaging evidence, and difficulty in ensuring the consistency and logical reliability of the reasoning process and the final report. This application addresses these issues by repeatedly generating at least two candidate output sequences for any training sample, while keeping the medical image and text instructions unchanged, thus enabling the simultaneous generation of multiple different report candidates under the same input. For each candidate output sequence, a combined reward value consisting of content evaluation reward, spatial evaluation reward, semantic logic reward, and format reward is calculated. This incorporates information related to disease content consistency, lesion spatial location consistency, consistency between reasoning content and report text, and the standardization of the output structure in image report generation. The common mapping is used to obtain numerical feedback that can be used for optimization. Based on this, the proposed scheme determines the relative update weight within the candidate output group based on the reward value of each candidate output sequence within the candidate output group, and adopts a group relative strategy to optimize the parameter update of the model to be trained. This allows the model parameter update to be driven by the relative merits of candidate outputs under the same input, rather than relying solely on the coarse-grained evaluation signal of a single output. Therefore, a relatively dense and distinguishable optimization basis can be continuously obtained during the training process, and the model can simultaneously satisfy constraints such as disease-related content, lesion spatial location, consistency between inference and report, and structural norms when generating reports. This suppresses hallucinatory descriptions, inconsistencies, and logical confusion in report generation, and reduces the training progress difficulties caused by the sparsity of evaluation signals. Thus, it can address the problems of sparse reward signals, difficulty in fine-grained constraints, and insufficient consistency and localization constraints in existing technologies. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the reinforcement learning post-training method for an image report generation model provided in one embodiment of this application; Figure 2 A flowchart illustrating a reinforcement learning post-training method for image report generation based on fine-grained dense rewards, provided as an embodiment of this application; Figure 3 This is a schematic diagram of a multi-level dense reward system for disease identification, provided as an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] Existing large-scale medical imaging models commonly suffer from numerous problems when generating image reports, including severe model hallucinations, logical inconsistencies, and contradictions, significantly hindering the widespread application and promotion of image-assisted report generation in clinical practice. Research indicates that post-training large-scale models trained using supervised fine-tuning (SFT) through reinforcement learning can effectively reduce model hallucinations. Traditional reinforcement learning post-training methods, such as proximal policy optimization (PPO), require training multiple models simultaneously, resulting in a complex process and high resource consumption. In contrast, group relative policy optimization (GRPO) requires fewer resources and has higher training efficiency, demonstrating good performance in tasks such as medical imaging disease diagnosis, image-based visual question answering, and image lesion localization. However, post-training for image report generation tasks is even more complex. Medical imaging covers a wide range of diseases, and some symptoms are not prominent in images. Large-scale models themselves have poor report generation capabilities, resulting in sparse reward signals. Furthermore, due to the diverse writing styles of image reports, there are instances of "different images for the same disease" and "same images for different diseases," making it extremely difficult to accurately assess the description of image features and the corresponding diseases to calculate rewards. To address these issues, this embodiment proposes a reinforcement learning post-training method for image report generation based on fine-grained dense rewards. It innovatively introduces multi-level dense rewards for disease identification, disease lesion localization, and semantic logic rewards for reasoning, greatly enhancing the density of rewards during the post-training phase of image report generation and significantly improving the usability of the model's generated image reports.

[0020] There are three main paradigms for existing reinforcement learning post-training methods: Proximal Policy Optimization (PPO) maximizes expected reward through gradient ascent. PPO requires training a complete reward model to calculate the reward signal, which is then used to update the policy model. PPO training is stable but inefficient, requiring the simultaneous learning of a policy model, a reference model, a reward model, and a value model.

[0021] Direct Preference Optimization (DPO) simplifies the overall training process compared to PPO. It optimizes the policy model directly by comparing "good" and "bad" samples, without the need to train a reward model. However, the model's exploration space is relatively limited.

[0022] Group Relative Policy Optimization (GRPO) directly generates multiple candidate responses for the same input and optimizes them by ranking the relative scores of different responses within the group. It eliminates the need for reward and value models and also increases the explorable space of the policy model.

[0023] While GRPO can directly estimate the advantage function through intra-group relative reward ranking, significantly reducing post-training computational resource consumption and dependence on high-quality labeled data, its training stability is inferior to PPO, making it prone to entropy collapse and getting trapped in local optima. GRPO's advantage estimation heavily relies on intra-group relative comparisons. When the base model itself is poorly performed or the task is complex, it can easily lead to poor performance in generated responses within the same group, with no significant difference between samples within the group, resulting in an almost zero advantage function. This causes the policy model to converge very slowly, or even fail to update. Furthermore, most post-training methods, in order to reduce computational resource consumption, adopt an outcome-based reward model (ORM), lacking more fine-grained scoring of reasoning steps. This further sparses the reward signal, making reinforcement learning post-training for complex tasks such as image report generation extremely difficult.

[0024] To address the aforementioned problems, in one embodiment, such as Figure 1 As shown, a reinforcement learning post-training method for an image report generation model is provided, comprising the following steps: Multiple training samples are acquired, each training sample including a medical image, and text instructions and reference information corresponding to the medical image, wherein the reference information is used to evaluate candidate output sequences; For any of the training samples, the medical image and the text instruction are used as input. While keeping the medical image and the text instruction unchanged, the image report generation model to be trained is controlled to repeatedly generate N candidate output sequences, where N is a positive integer greater than or equal to 2. The candidate output sequences are text sequences generated by the image report generation model to be trained for the medical image and the text instruction, and each candidate output sequence includes image report text. The N candidate output sequences constitute a candidate output group. For each candidate output sequence in the candidate output group, a reward value for reinforcement learning optimization is calculated. The reward value is a combination of multiple rewards, including a content evaluation reward based on disease-related content in the candidate output sequence, a spatial evaluation reward based on lesion spatial location-related content in the candidate output sequence, a semantic logic reward based on the consistency between the inference content of the candidate output sequence and the image report text and a preset semantic rule, and a format reward for constraining the candidate output sequence to meet a preset output structure. Based on the reward value corresponding to each candidate output sequence within the candidate output group, the relative update weight within the group for each candidate output sequence is determined, and the parameters of the image report generation model to be trained are updated using a group relative strategy optimization, thus completing the reinforcement learning training.

[0025] In one implementation, the process of acquiring multiple training samples includes: reading medical image files (DICOM, PNG, JPG, etc.) from PACS / RIS or offline datasets, and performing preprocessing operations on the images. The preprocessing may include window width and level adjustment, pixel value normalization, resolution resampling, cropping / filling, selecting representative slices, or slice sampling of CT sequences. Simultaneously, text instructions and reference information corresponding to the image are read. The text instructions may be fixed template instructions or real instructions within the dataset, examples including "Generate what you see and what you get" and "Please output an imaging report and provide conclusions." The reference information is external comparison information corresponding to the image, used for subsequent reward calculations.

[0026] In one implementation, the image report generation model to be trained is a multimodal generation model, including a visual feature extraction subnetwork and a text generation subnetwork. The visual feature extraction subnetwork is used to encode medical images into visual feature sequences or feature maps; the text generation subnetwork is used to autoregressively generate text sequences given visual features and text instructions. The model can output a single report text or multiple text sequences; this embodiment requires that at least the candidate output sequences contain image report text.

[0027] In one implementation, for any training sample, while keeping the medical image and text instructions unchanged, the model is controlled to repeatedly generate N candidate output sequences. Repeated generation can be achieved through "random sampling decoding," such as changing the random seed, enabling temperature sampling, or top-k / top-p sampling, so that the same input yields multiple different candidate outputs. During generation, a maximum generation length, a terminator, and a set of disallowed tokens can be set. After generation, the N candidate output sequences are cached as a candidate output group for use in reward calculation and relative weight update calculation within the group.

[0028] In one implementation, a reward value is calculated for each candidate output sequence in the candidate output group. The reward value is a combination of multiple rewards, including at least the following four categories: Content evaluation reward: used to reflect the degree of relevance of disease-related content in candidate output sequences; Spatial evaluation reward: used to reflect the degree of conformity of the spatial location-related content of lesions in the candidate output sequence; Semantic logic reward: used to reflect the consistency between the reasoning content of the candidate output sequence and the image report text, as well as the degree of satisfaction with the preset semantic rules; Format reward: Used to reflect whether the candidate output sequence meets the preset output structure.

[0029] In implementation, the reward value can be expressed as a linear weighted sum or as a gated fusion. For example, if the format reward fails the judgment, the remaining rewards can be decayed or set to zero. Alternatively, each reward component can be pruned, normalized, or segmented to stabilize the training.

[0030] In one implementation, relative update weights within a candidate output group are determined based on the reward values ​​of each candidate output sequence within that group. These relative update weights can be achieved by: sorting the reward values ​​within the group and mapping them to relative weights; calculating the difference based on the group mean as the relative weight; or standardizing the group rewards to obtain a relative advantage, which is then mapped to the update weight. Subsequently, a group-based relative strategy is used to optimize the update of the model parameters: the generation probability of the candidate output sequences, the relative update weights within the group, and optional regularization constraints (e.g., constraints on the difference in distribution with the reference strategy) together constitute the optimization objective; gradient updates are performed on the model parameters, and this process is repeated for multiple training steps to complete the reinforcement learning post-training.

[0031] This embodiment uses a process of "generating candidate output groups from the same input, calculating combined rewards, updating weights relative to each group, and optimizing updates based on relative strategies within the group" to drive model updates based on relative merits within each group. It also utilizes multi-source rewards from content, space, semantics, and format to comprehensively constrain the output, thereby improving the optimization targeting and stability of the image report generation task during the post-training process.

[0032] In this embodiment, the reference information includes the image report text corresponding to the medical image, disease annotation information, and / or lesion spatial region annotation information.

[0033] This embodiment uses at least one of the report text, disease annotation information, and lesion spatial region annotation information as reference information, which can provide a comparison basis for the subsequent calculation of different reward components, and enable the training process to be implemented under various data annotation completeness levels.

[0034] In this embodiment, the candidate output sequence further includes inference content, which is used to characterize the generation basis and inference process of the image report text. The inference content includes disease-related analysis information and / or coordinate information of the lesion spatial region. The reasoning content and the image report text are distinguished from each other in the candidate output sequence by a preset separator.

[0035] In one implementation, the candidate output sequence includes inference content in addition to the image report text. The inference content characterizes the basis for generating the image report text and the reasoning process, and may include descriptions of abnormal signs, analysis and judgment of disease categories, descriptions of lateral and anatomical location, and spatial coordinate information of candidate lesions. The inference content can be a natural language description or may contain structured field fragments for parsing.

[0036] In one implementation, the inference content and the image report text are distinguished by a preset delimiter. The delimiter can be a specific string, a special token sequence, or a template field, for example, " <think> …< / think> <answer> …< / answer> "or "Inference: ...###Report: ...". This format can be used to constrain the model output at the generation end; at the parsing end, the start and end positions of the inference and report segments can be located using separator markers.

[0037] In one implementation, the "disease-related analysis information" of the reasoning content can be organized by item, for example, outputting a description of "abnormal manifestation - location - judgment" for each disease; the "coordinate information" of the reasoning content can be output in a uniform format, which is convenient to extract using regular expressions or parsers, and then used as input data for spatial evaluation rewards.

[0038] This embodiment introduces reasoning content that is distinguishable from the report text into the candidate output sequence, enabling the reward calculation to have "analyzable reasoning and coordinate carriers," thereby supporting the constraint and optimization of model output by semantic logic rewards and spatial evaluation rewards.

[0039] In this embodiment, before the reinforcement learning post-training, the method further includes: Multiple cold start training samples are selected from the plurality of training samples; For any of the aforementioned cold start training samples, a target output sequence for supervised training is constructed based on the medical images, text instructions, and reference information of the cold start training samples. The target output sequence includes inference content and image report text. The inference content is determined based on disease annotation information and / or lesion spatial region annotation information in the reference information. The target output sequence is used to supervise the training of the image report generation model to be trained.

[0040] In one implementation, cold-start supervised training is performed before reinforcement learning training. First, multiple cold-start training samples are selected from a pool of training samples. This can be done through random sampling, balanced sampling by disease category, or prioritizing samples with complete annotations (both disease and spatial annotations). The number of cold-start samples can be a subset of the training set, and the specific size can be determined by engineering resources and model convergence.

[0041] In one implementation, for any cold-start training sample, a target output sequence for supervised training is constructed based on its medical image, text instructions, and reference information. The target output sequence includes inference content and image report text. The image report text can be taken from the image report text in the reference information, or its standardized version (such as terminology standardization and paragraph templates). The inference content is determined based on the disease annotation information and / or lesion spatial region annotation information in the reference information: for example, the disease annotation information is converted into inference entries of "disease category, side and location", and the lesion spatial region annotation information is converted into coordinate entries associated with the disease category, and then spliced ​​together in a preset order to form an inference segment; when both types of annotations exist at the same time, the disease entries can be associated with the corresponding coordinate entries for output.

[0042] In one implementation, the target output sequence is used to perform supervised training on the model to be trained. Supervised training can employ standard sequence-supervised learning: using medical images and text instructions as input, and the target output sequence as the supervised target for teacher forcing training; during training, different loss weights can be set for the inference and report segments, or training can be conducted in stages (first learning the format and fields, then learning the complete inference and report text). After the cold start training is completed, the reinforcement learning post-training process begins.

[0043] This embodiment constructs a supervised target containing reasoning content based on reference information and performs cold start training, enabling the model to have the basic format and content expression ability to output "reasoning segment + report segment" before entering reinforcement learning training, reducing the difficulty of reward calculation caused by unparseable output in the later training stage.

[0044] In this embodiment, the method further includes: The disease extraction model is used to extract disease entities and their corresponding location descriptions from the image report text of the candidate output sequence to obtain the extraction results. The extraction results are matched with the disease labeling information to obtain the matching results; Based on the matching results, the content evaluation reward is determined.

[0045] In one implementation, a disease extraction model is used to parse image report text in candidate output sequences into structured disease entities and location descriptions. The disease extraction model can be implemented using a lightweight language model or a sequence labeling model, and the output may include fields such as: disease entity name (normalized category), laterality (left / right / bilateral / uncertain), and anatomical location (lung lobe, lung segment, rib number, etc.). The model can be trained using labeled report corpora and can incorporate a terminology dictionary for synonym mapping.

[0046] When calculating content evaluation rewards, the image report text of the candidate output sequence is first input into the disease extraction model to obtain extraction results. These results can be represented as a set of multiple entries, each containing "disease entity + location description information." The extraction results are then matched with disease annotation information to obtain matching results. These matching results may include information such as whether each extracted entry matches an annotation entry, which annotation was matched, and the matching score. Finally, content evaluation rewards are determined based on the matching results, for example, by summing the matching scores, averaging by entry, or adding or subtracting points for missed / false alarms according to preset rules.

[0047] This embodiment transforms free-text report content into an alignable and quantifiable structured evaluation by using a computational path of "report text, disease extraction, and matching with annotations to form rewards," thereby facilitating the stable calculation of content evaluation rewards and their use for reinforcement learning optimization.

[0048] In this embodiment, matching the extraction result with the disease labeling information to obtain a matching result includes: The extraction results are matched with the disease labeling information based on the disease category to obtain the first-level matching results; The extraction results are matched with the disease labeling information based on the disease category and left / right side information to obtain the second-level matching results; The extraction results are matched with the disease annotation information based on the disease category, left and right side information and fine-grained anatomical location information to obtain the third-level matching results; The process of determining the content evaluation reward based on the matching results includes: The first-level matching result, the second-level matching result, and the third-level matching result are weighted and aggregated according to a preset level weight to obtain a level aggregation result; The hierarchical aggregation results corresponding to different disease entities are weighted and aggregated according to preset disease weights to determine the content evaluation reward.

[0049] In one implementation, a hierarchical matching mechanism is introduced into the matching process to determine the consistency between the extracted results and the disease annotation information at different granularities. The first-level matching is based solely on the disease category field; the second-level matching adds left / right lateral fields to the first-level matching; and the third-level matching adds fine-grained anatomical location fields to the second-level matching. In implementation, the disease category, laterality, and location names can first be standardized and mapped (e.g., unified to a disease dictionary and an anatomical hierarchy dictionary), and then field consistency or similarity checks can be performed hierarchically.

[0050] In one implementation, when determining the content evaluation reward, the three-tiered matching results are weighted and aggregated according to preset tier weights to obtain a tiered aggregation result. The tier weights can reflect the importance of different tiered matching and are configurable. Subsequently, the tiered aggregation results corresponding to different disease entities are weighted and aggregated according to preset disease weights to obtain the final content evaluation reward; the disease weights can be configured according to disease category, disease risk level, clinical attention, or data statistics frequency.

[0051] In the implementation of the project, different deduction strategies can be set for missed reports, false reports, side errors, and location errors, and the deduction can be integrated into the calculation process of hierarchical aggregation or disease aggregation, so that the content evaluation reward can distinguish different types of errors.

[0052] This embodiment subdivides the matching results into multi-level matching of disease category / side / fine-grained location, and aggregates them using hierarchical weights and disease weights. This allows the content evaluation reward to generate differentiated scores for cases that are "partially correct but have minor errors", thereby providing more fine-grained optimization signals for reinforcement learning training.

[0053] In this embodiment, the method further includes: The spatial region information of the predicted lesion is parsed from the inference content of the candidate output sequence, and the spatial region information of the predicted lesion includes the coordinate information of the spatial region of the lesion. The predicted lesion spatial region information is matched with the lesion spatial region annotation information to obtain the spatial matching result; Based on the spatial matching results, the spatial evaluation reward is calculated according to the overlap index; wherein, the overlap index includes the intersection-union ratio index.

[0054] In one implementation, the spatial evaluation reward is calculated based on the predicted lesion spatial region information in the inference content. First, the predicted lesion spatial region information is parsed from the inference content of the candidate output sequence. This information includes at least the coordinates of the lesion spatial regions. The parsing method can be regular expression extraction, template-based field parsing, or JSON parsing based on structured output. If the inference content contains multiple lesion regions, multiple coordinate entries can be obtained through parsing.

[0055] The predicted lesion spatial region information is then matched with the lesion spatial region annotation information to obtain the spatial matching result. The matching method may include: grouping and matching according to the consistency of disease category; or matching according to spatial proximity, centroid distance, or maximum overlap when category information is not available; when there are many-to-many candidates, the "maximum matching" strategy or greedy matching strategy can be used to determine the matching pair.

[0056] This embodiment parses the lesion coordinates from the inference content and matches them with the labeled area, and then uses the overlap index such as IoU to form a spatial evaluation reward. This can establish a quantifiable alignment relationship between the model output and the spatial evidence of the image, thereby prompting the model to pay attention to lesion localization related information in post-training.

[0057] In this embodiment, the method further includes: The reasoning content of the candidate output sequence and the image report text are input into the semantic evaluation model according to preset semantic rules, and the semantic evaluation model outputs a semantic score. The semantic score is determined as the semantic logic reward.

[0058] In one implementation, predefined semantic rules are used to constrain the semantic consistency and logical rationality between the inference content and the image report text. Predefined semantic rules may include at least: whether explicit conclusions in the inference content are reflected in the report text; whether key conclusions in the report text can be supported by corresponding evidence in the inference content; whether there are conflicting descriptions between the inference and the report; and whether there are any violations of medical common sense or incomplete / vague expressions. Predefined semantic rules can be written as a fixed set of rules or as prompt word templates for the semantic evaluation model to execute.

[0059] In one implementation, the inference content of the candidate output sequence and the image report text are input into a semantic evaluation model according to preset semantic rules, and the semantic evaluation model outputs a semantic score. The semantic evaluation model can be a plain text large language model or a dedicated evaluation model; the input format can be a combination of "rules + inference content + report text" prompts, and the output semantic score is a scalar within a preset numerical range, or multiple sub-scores can be output first and then aggregated into a total score. To reduce score fluctuations, the prompt word template can be fixed, the scoring scale can be fixed, and the output score can be range-clipping or normalized.

[0060] The semantic score is determined as the semantic logical reward and is included as part of the combined reward in the calculation of the relative update weight within the group.

[0061] This embodiment introduces a semantic evaluation model based on preset semantic rules to output a semantic score, enabling the training process to quantitatively constrain the consistency and logical rationality of reasoning to report, thereby improving the consistency level of candidate outputs between the reasoning chain and the final report.

[0062] In this embodiment, the method further includes: Determine whether the candidate output sequence satisfies a preset output structure; wherein, the preset output structure includes inference content and image report text, and the inference content and the image report text are distinguished by a preset separator; The reward format is determined based on the judgment result.

[0063] In one implementation, a preset output structure is used to ensure that the candidate output sequence has a parsable, uniform format. The preset output structure includes at least two parts: inference content and image report text, and requires that the two parts be distinguished by a preset separator. The preset separator can be a fixed string; the preset output structure may further specify the order of appearance of the inference and report segments, the segment headings, and whether necessary fields are complete.

[0064] In one implementation, determining whether a candidate output sequence meets a preset output structure can be achieved through a rule parser: checking if a separator marker exists; checking if both the inference segment and the report segment are not empty; checking if the inference segment contains a specified field pattern (such as an optional coordinate field), etc. If the candidate output sequence meets the preset output structure, it is deemed to pass; otherwise, it is deemed to fail.

[0065] In one implementation, the format reward is determined based on the judgment result: a first format score is assigned when the judgment passes, and a second format score is assigned when the judgment fails; alternatively, multiple score levels can be assigned according to the degree of structural matching, for example, different scores are assigned for missing separator markers, missing inference segments, and missing report segments. The format reward is used as part of the combined reward for subsequent calculation of relative update weights within the group.

[0066] This embodiment improves the parsability and structural consistency of the output sequence by determining whether the candidate output sequence meets the preset output structure of "inference content, report text and separator mark" and setting format rewards accordingly, thus providing a stable input format for content extraction, coordinate parsing and semantic consistency evaluation.

[0067] This embodiment proposes a reinforcement learning post-training method for image report generation based on fine-grained dense rewards. It innovatively introduces multi-level dense rewards for disease identification, disease lesion localization, and semantic logic rewards for reasoning, which greatly enhances the reward density in the post-training stage of image report generation, increases the reward gap of intra-group responses, thereby increasing the advantage function, accelerating the training efficiency of reinforcement learning, and significantly improving the availability of the model's generated image reports.

[0068] The image report generation reinforcement learning post-training method proposed in this embodiment is as follows: Figure 2 As shown, the overall reinforcement learning paradigm is based on GRPO group relative policy optimization.

[0069] Suppose there exists an image report dataset. ,in This represents the input image. This indicates the corresponding text instruction. This describes the reasoning steps involved in generating the image report. This indicates the actual location of the lesion detection frame. This describes a realistic image report. The goal is to train and optimize a large multimodal model that, given an input image and text instructions, generates a corresponding image report.

[0070] in, It is the output image report. This represents the policy model (a multimodal large model with objective optimization). This embodiment considers the reinforcement learning post-training method and does not discuss the full-link training (pre-training, fine-tuning) of the medical image multimodal large model in depth. Therefore, it is assumed that the initial weights of the policy model are the medical image multimodal large model fine-tuned by the image report.

[0071] In this embodiment, the cold start is similar to Deepseek-R1. To provide a high-quality starting point for post-RL training and improve the speed of GRPO training, this embodiment first uses image report generation data with chain-of-thought (CoT) steps to supervise and fine-tune the initial model. Through cold-start supervised fine-tuning, on the one hand, the policy model's output can be aligned with the template of thinking before answering, obtaining higher format rewards in the early stages of training, allowing the model to focus on learning the more important reward of report accuracy. On the other hand, it can reduce the possibility of the policy model randomly outputting answers during the GRPO sampling process, avoiding low rewards for most candidate results within the group, which would slow down model training convergence.

[0072]

[0073] The loss function for objective optimization is shown in the above equation, where Indicates the generation of an output image report. For each token in the sequence, T represents the total sequence length. This phase uses only one-tenth of the total CoT data for cold start training.

[0074] Because image report generation styles are relatively free and diverse, traditional character-matching-based accuracy rewards are insufficient to effectively distinguish between high and low quality reports, and may even lead to reward sparsity or reward hacking. Therefore, in the post-GRPO training phase, in addition to basic format rewards, this embodiment proposes three reward calculation methods to improve the overall density of the reward signal.

[0075] In this embodiment, the multi-level dense reward for disease identification is as follows: the essence of image report generation is disease identification and diagnosis, as well as accurate localization of lesions. While some existing medical image VQA post-training methods consider this by designing multiple-choice questions to reward correct options, the probability of a large model simultaneously predicting and identifying all diseases and their locations in an image is low, especially given the limited diagnostic capabilities of the base model itself. This is particularly true for identifying disease locations, which requires a more refined anatomical understanding. For some non-visually significant diseases such as rib fractures and small pulmonary nodules, even after 8-16 rounds of sampling, the model cannot accurately identify the corresponding disease and its specific location. Therefore, it is more common for the model to only predict a portion of the answers. For example, if the true labels are a right 8th rib fracture and a left middle pulmonary nodule, the model may only sample a right 7th rib fracture and a left lower pulmonary nodule. Traditional character matching reward designs will not reward such candidates, inevitably leading to extremely sparse reward signals, which is detrimental to estimating the dominance function within a group. Ultimately, from a practical standpoint, a model that can identify the disease and provide partial solutions is better than one that completely misidentifies the disease; there should be a clear difference between the two.

[0076] Furthermore, in actual clinical diagnosis, the level of attention paid to different disease types in imaging reports varies. Doctors often focus more on diseases with higher clinical value and greater harm, such as lung lesions (which may indicate tumors) and pneumonia. In principle, they cannot tolerate any missed or misdiagnosed cases of these diseases, while they are relatively more lenient towards diseases with lower clinical value. Therefore, the reward function should assign different weights to different disease types. For example, if the true disease list includes pneumonia, old lesions, and enlarged cardiac silhouette, and candidate sample 1 predicts pneumonia and old lesions, while candidate sample 2 predicts old lesions and enlarged cardiac silhouette, according to the previous reward calculation method, both would be considered partial hits and might both receive a 2 / 3 score. However, in reality, the result of sample 1 is more clinically acceptable than that of sample 2.

[0077] Based on two observations and considerations from the above embodiments, this embodiment proposes a multi-level dense reward system for disease identification, such as... Figure 3 As shown, firstly, a Qwen2.5-0.5B pure language model is trained through fine-tuning to extract diseases and their locations from prediction reports. Based on the extracted diseases and their locations, three levels of matching scores can be calculated.

[0078]

[0079] in, This represents the reward score obtained for disease type s. It's a reward score that only matches disease types; you only need to predict the corresponding disease type to receive the reward. =1, The reward score for disease identification is based on primary location; to receive the reward, one must simultaneously predict the corresponding disease and the left and right sides where the disease occurs. =1, This is a more refined secondary-level disease identification reward score. In addition to correctly predicting the disease and its left / right orientation, you must also correctly identify the corresponding lung region (left upper lobe, left lower lobe, etc.) or (left 7th rib, left 8th rib, etc.) to receive the corresponding reward. =1. And , and These represent the weights corresponding to different levels of disease identification rewards. This design ensures that the model can still receive rewards even when its sampled predictions partially hit the correct answer, increasing the within-group reward variance.

[0080]

[0081] Next, different weights are assigned to each disease based on its clinical importance, and reward scores are allocated to the corresponding disease types. Weighted summation is performed to obtain the final disease identification reward signal. .

[0082] This embodiment also includes a disease lesion localization bonus: Although this embodiment introduces a discrimination reward for predicting disease location in the multi-level dense reward calculation for disease identification, that is still a coarse-grained lesion localization reward. A more fine-grained lesion localization should output the corresponding lesion detection box in the original image. Most existing medical image reinforcement learning post-training methods define lesion localization and disease identification as two independent tasks, designing separate reward functions and training them separately. This is especially true in image report generation, where rarely is the training enhancement of the model's lesion localization ability considered. This embodiment, however, believes that by designing a corresponding lesion localization reward and incorporating it into the post-training of the image report generation task, it can, to some extent, guide the multimodal large model to "see" lesions in the image. Therefore, an additional disease lesion localization reward is designed and introduced during the GRPO training process.

[0083] Specifically, in the reasoning process before generating the image report, the large model first generates a differential diagnosis list, listing all possible diseases based on a general observation of the image features. Then, it confirms or excludes each disease in the differential diagnosis list. In this embodiment, the actual bounding box coordinates of the corresponding diseases are additionally added during these reasoning steps (obtained through manual annotation). This data is used for SFT cold start. In this way, the large model also predicts the location information of the corresponding diseases and outputs the bounding box coordinates during the reasoning process.

[0084] A reward signal can be calculated based on the coordinates of the actual labeled detection boxes and the coordinates of the detection boxes predicted by the large model during the inference process. First, the detection box labels (i.e., disease types) are matched. For two sets of detection boxes with the same disease type, matching is performed again, and the pair with the largest Intersection over Union (IoU) is selected as the final matching result. The average IoU of all matched detection boxes is calculated as the final reward score, as shown in the following formula:

[0085] in, and These represent a pair of ground truth bounding boxes and a predicted bounding box that match each other.

[0086] This embodiment also includes a reasoning semantic logic reward: The multi-level dense reward system for disease identification and the disease lesion localization reward system proposed in the above design can respectively target... <answer>Parts and <think>Some output content is subject to accuracy-based reward constraints. However, these constraints are limited to the accuracy of their respective outputs. They fail to effectively penalize semantic ambiguities in the reasoning process or contradictions between the generated image report and the reasoning process. This is a point often overlooked by most image report generation training methods. If left unchecked, inconsistencies or even contradictions between the reasoning process and the final report can occur, thus affecting the final quality of the generated report.

[0087] To address this, this embodiment introduces a reasoning semantic logic reward, essentially scoring through a large model. First, a set of semantic evaluation rules is pre-defined and used as input prompts to the pure language large model. Simultaneously, the reasoning steps and image reports generated by the strategy model are input, allowing the large model to score the generated content according to the rules. Specific prompts are: "You are a radiology imaging expert. Please evaluate the input imaging report and the reasoning steps used to generate the report. If there are no obvious semantic problems in the imaging report and reasoning steps, you can directly output a full score of 100. For each semantic or factual error, points will be deducted according to the rules until all points are deducted, at which point 0 points will be output. Imaging report: {}; Reasoning steps: {}; Rules: 30 points will be deducted for each instance of content that clearly violates medical common sense in the reasoning steps, such as "The right costophrenic angle is sharp, and the diaphragm surface is smooth, therefore it is inferred that pleural effusion may occur"; 10 points will be deducted for each instance of ambiguous content in the reasoning steps, such as "Patchy, linear, and band-like high-density shadows are visible in the left lung field"; 20 points will be deducted for each disease diagnosed in the reasoning steps but not mentioned in the generated imaging report; 20 points will be deducted for each disease not inferred in the reasoning steps but mentioned in the generated imaging report; Example..." Based on the three reward function designs mentioned above—multi-level dense reward for disease identification, reward for disease lesion localization, and semantic logic reward for inference—along with conventional format rewards, all reward signals for GRPO post-training are constituted. These carefully designed reward calculation methods can significantly improve the density of the sampling rewards in the policy model for image report generation tasks, accelerate GRPO training, and enhance the final report generation performance.

[0088] This embodiment addresses the common problem of poor performance in existing medical image report generation base models. These models struggle to sample accurate, stable, and dense reward signals during the post-training phase of reinforcement learning, resulting in limited post-training gain and suboptimal image report quality. From a clinical perspective, this embodiment proposes three reward calculation methods covering various sampling and prediction outputs, establishing a multi-level, fine-grained reward evaluation system. This improves intra-group reward density and enhances the training speed and convergence performance of the GRPO image report generation task.

[0089] The best post-training methods for reinforcement learning in chest X-ray report generation are ChestX-Reasoner[1] and CX-Mind[2]. ChestX-Reasoner[1] extracts disease entities from the generated content through an external large model and calculates the cross-union ratio between the predicted disease entities and the ground truth to obtain the reward signal, ignoring the accurate judgment of lesion location. Although the CX-Mind[2] method introduces the lesion localization task to train the model together, it treats image report generation and lesion localization as two independent tasks for training. This embodiment believes that by designing corresponding lesion localization rewards and adding them to the post-training of the image report generation task, it can guide the multimodal large model to "see" the lesions in the image to a certain extent, thereby generating higher quality image reports. Therefore, the detection box output for lesion location is added in the reasoning thinking step, and corresponding multi-level dense rewards for disease recognition and lesion localization rewards are designed to establish a coarse-to-fine lesion localization reward system, guiding the model to pay more attention to image information.

[0090] On the other hand, although both ChestX-Reasoner[1] and CX-Mind[2] introduce process rewards for the reasoning steps, they lack consideration for possible ambiguities and contradictions in the reasoning steps, which to some extent affects the quality of the final report. Based on this, this embodiment proposes a reasoning semantic logic reward. By predefining a comprehensive set of semantic evaluation rules, it prompts the external large model to score the generated reasoning steps, and thereby generates reward scores to supervise model training, improve the overall consistency of reasoning logic, and reduce factual errors.

[0091] In one embodiment, a reinforcement learning post-training apparatus for an image report generation model is provided, the apparatus comprising: The sample acquisition module is used to acquire multiple training samples, each of which includes a medical image, as well as text instructions and reference information corresponding to the medical image. The reference information is used to evaluate candidate output sequences. A candidate generation module is configured to, for any training sample, take the medical image and the text instruction as input, and, while keeping the medical image and the text instruction unchanged, control the image report generation model to be trained to repeatedly generate N candidate output sequences, where N is a positive integer greater than or equal to 2; wherein, the candidate output sequence is a text sequence generated by the image report generation model to be trained for the medical image and the text instruction, and the candidate output sequence includes image report text; the N candidate output sequences constitute a candidate output group; The reward calculation module is used to calculate the reward value for reinforcement learning optimization for each candidate output sequence in the candidate output group. The reward value is a combination of multiple rewards, including content evaluation reward based on disease-related content in the candidate output sequence, spatial evaluation reward based on lesion spatial location-related content in the candidate output sequence, semantic logic reward based on the consistency between the inference content of the candidate output sequence and the image report text and the preset semantic rules, and format reward for constraining the candidate output sequence to meet the preset output structure. The weight determination module is used to determine the relative update weight within the group for each candidate output sequence based on the reward value corresponding to each candidate output sequence within the candidate output group. The strategy optimization module is used to update the parameters of the image report generation model to be trained by using group relative strategy optimization, and to complete the training after reinforcement learning.

[0092] The specific implementation details of each module can be found in the above description of the limitations of the reinforcement learning post-training method for the image report generation model, and will not be repeated here.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.< / think> < / answer>

Claims

1. A reinforcement learning post-training method for an image report generation model, characterized in that, The method includes: Multiple training samples are acquired, each training sample including a medical image, and text instructions and reference information corresponding to the medical image, wherein the reference information is used to evaluate candidate output sequences; For any of the training samples, the medical image and the text instruction are used as input. While keeping the medical image and the text instruction unchanged, the image report generation model to be trained is controlled to repeatedly generate N candidate output sequences, where N is a positive integer greater than or equal to 2. The candidate output sequences are text sequences generated by the image report generation model to be trained for the medical image and the text instruction, and each candidate output sequence includes image report text. The N candidate output sequences constitute a candidate output group. For each candidate output sequence in the candidate output group, a reward value for reinforcement learning optimization is calculated. The reward value is a combination of multiple rewards, including a content evaluation reward based on disease-related content in the candidate output sequence, a spatial evaluation reward based on lesion spatial location-related content in the candidate output sequence, a semantic logic reward based on the consistency between the inference content of the candidate output sequence and the image report text and a preset semantic rule, and a format reward for constraining the candidate output sequence to meet a preset output structure. Based on the reward value corresponding to each candidate output sequence within the candidate output group, the relative update weight within the group for each candidate output sequence is determined, and the parameters of the image report generation model to be trained are updated using a group relative strategy optimization, thus completing the reinforcement learning training.

2. The method according to claim 1, characterized in that, The reference information includes the image report text corresponding to the medical image, disease annotation information, and / or lesion spatial region annotation information.

3. The method according to claim 1, characterized in that, The candidate output sequence also includes inference content, which is used to characterize the generation basis and inference process of the image report text. The inference content includes disease-related analysis information and / or coordinate information of the lesion spatial region. The reasoning content and the image report text are distinguished from each other in the candidate output sequence by a preset separator.

4. The method according to claim 2, characterized in that, Prior to the reinforcement learning training, the method further includes: Multiple cold start training samples are selected from the multiple training samples; For any of the aforementioned cold start training samples, a target output sequence for supervised training is constructed based on the medical images, text instructions, and reference information of the cold start training samples. The target output sequence includes inference content and image report text. The inference content is determined based on disease annotation information and / or lesion spatial region annotation information in the reference information. The target output sequence is used to supervise the training of the image report generation model to be trained.

5. The method according to claim 2, characterized in that, The method further includes: The disease extraction model is used to extract disease entities and their corresponding location descriptions from the image report text of the candidate output sequence to obtain the extraction results. The extraction results are matched with the disease labeling information to obtain the matching results; Based on the matching results, the content evaluation reward is determined.

6. The method according to claim 5, characterized in that, The step of matching the extraction result with the disease labeling information to obtain the matching result includes: The extraction results are matched with the disease labeling information based on the disease category to obtain the first-level matching results; The extraction results are matched with the disease labeling information based on the disease category and left / right side information to obtain the second-level matching results; The extraction results are matched with the disease annotation information based on the disease category, left and right side information and fine-grained anatomical location information to obtain the third-level matching results; The process of determining the content evaluation reward based on the matching results includes: The first-level matching result, the second-level matching result, and the third-level matching result are weighted and aggregated according to a preset level weight to obtain a level aggregation result; The hierarchical aggregation results corresponding to different disease entities are weighted and aggregated according to preset disease weights to determine the content evaluation reward.

7. The method according to claim 3, characterized in that, The method further includes: The spatial region information of the predicted lesion is parsed from the inference content of the candidate output sequence, and the spatial region information of the predicted lesion includes the coordinate information of the spatial region of the lesion. The predicted lesion spatial region information is matched with the lesion spatial region annotation information to obtain the spatial matching result; Based on the spatial matching results, the spatial evaluation reward is calculated according to the overlap index; wherein, the overlap index includes the intersection-union ratio index.

8. The method according to claim 3, characterized in that, The method further includes: The reasoning content of the candidate output sequence and the image report text are input into the semantic evaluation model according to preset semantic rules, and the semantic evaluation model outputs a semantic score. The semantic score is determined as the semantic logic reward.

9. The method according to claim 1, characterized in that, The method further includes: Determine whether the candidate output sequence satisfies a preset output structure; wherein, the preset output structure includes inference content and image report text, and the inference content and the image report text are distinguished by a preset separator. The reward format is determined based on the judgment result.

10. A reinforcement learning post-training device for an image report generation model, characterized in that, The device includes: The sample acquisition module is used to acquire multiple training samples, each training sample including a medical image, as well as text instructions and reference information corresponding to the medical image, the reference information being used to evaluate candidate output sequences; A candidate generation module is configured to, for any training sample, take the medical image and the text instruction as input, and, while keeping the medical image and the text instruction unchanged, control the image report generation model to be trained to repeatedly generate N candidate output sequences, where N is a positive integer greater than or equal to 2; wherein, the candidate output sequence is a text sequence generated by the image report generation model to be trained for the medical image and the text instruction, and the candidate output sequence includes image report text; the N candidate output sequences constitute a candidate output group; The reward calculation module is used to calculate the reward value for reinforcement learning optimization for each candidate output sequence in the candidate output group. The reward value is a combination of multiple rewards, including content evaluation reward based on disease-related content in the candidate output sequence, spatial evaluation reward based on lesion spatial location-related content in the candidate output sequence, semantic logic reward based on the consistency between the inference content of the candidate output sequence and the image report text and the preset semantic rules, and format reward for constraining the candidate output sequence to meet the preset output structure. The weight determination module is used to determine the relative update weight within the group for each candidate output sequence based on the reward value corresponding to each candidate output sequence within the candidate output group. The strategy optimization module is used to update the parameters of the image report generation model to be trained by using group relative strategy optimization, and to complete the training after reinforcement learning.

Images