Medical report generation method and system based on multi-modal large language model

Abstract

The invention relates to a medical report generation method and system based on a multi-modal large language model, and the method comprises the steps: collecting historical cases in advance, each historical case comprising a medical image and a medical report, obtaining corresponding embedded vectors, and storing the embedded vectors in a vector database; according to an input medical image of a new case, generating an embedded vector of an image mode, obtaining a historical case with the highest similarity as a cue word, inputting the medical image of the new case and the cue word into the visual language large model, and obtaining a generation report; calculating the cosine similarity between the generated report and the input medical image and the cosine similarity between the generated report and the medical report of the historical case so as to carry out threshold judgment; and performing optimization training on the visual language large model through a reinforcement learning mechanism based on expert knowledge according to the generated report. Compared with the prior art, clinical knowledge can be deeply fused, doctor diagnosis behaviors are simulated, and medical accuracy is emphasized in the optimization process to realize automatic report generation.

Description

Technical Field

[0001] This invention relates to the field of computer-aided diagnostic technology, and in particular to a method and system for generating medical reports based on a multimodal large language model. Background Technology

[0002] With the rapid growth of medical imaging data and the increasing workload of radiologists, automated medical report generation systems have become crucial tools for improving diagnostic efficiency and consistency. In recent years, Vision-Language Models (VLMs) have made significant progress in medical image analysis, enabling cross-modal reasoning from images to text. However, existing report generation systems still have several key limitations: First, the existing system fails to integrate the core logic of clinical diagnosis. When interpreting complex images, especially blurred or rare lesions, radiologists will instinctively refer to similar historical cases to aid decision-making. However, the current method treats each imaging examination as an isolated data point and does not build a contextual reference mechanism. This not only deviates from the actual diagnostic process of doctors, but also makes the model detached from the experience and knowledge that guides expert practice, and cannot simulate the clinical sequential reasoning required for accurate diagnosis. Secondly, radiologists review and revise their initial assessments when writing medical reports, but the existing system lacks quality monitoring and self-correction mechanisms, making it unable to address semantic drift or diagnostic inconsistencies. Third, the optimization goals of existing systems are out of touch with clinical needs. Mainstream methods rely on language-oriented reward functions such as BLEU and ROUGE. These indicators are weakly correlated with the anatomical accuracy and diagnostic consistency required for medical reports. This leads to the model prioritizing language fluency at the expense of the medical accuracy required for patient diagnosis and treatment, resulting in reports that are linguistically fluent but clinically distorted.

[0003] Therefore, there is an urgent need for an automated report generation framework that can deeply integrate clinical knowledge, simulate physician diagnostic behavior, and emphasize medical accuracy during the optimization process. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a medical report generation method and system based on a multimodal large language model. By simulating the diagnostic cognition and learning process of radiologists, it improves the clinical effectiveness and language quality of automatically generated medical reports, and assists radiologists in improving work efficiency and diagnostic accuracy.

[0005] The objective of this invention can be achieved through the following technical solutions: A method for generating medical reports based on a multimodal large language model includes the following steps: Historical cases are collected in advance, each including medical images and medical reports. The embedding vectors of each medical image and medical report are obtained and stored in the vector database. Based on the medical images of the new case, an embedding vector of the image modality is generated, and the similarity between the embedding vector of the image modality and each vector in the vector database is calculated. The historical case with the highest similarity is obtained as a prompt word. The medical images of the new case and the prompt word are input into a pre-trained visual language model to generate a report. The cosine similarity between the generated report and the input medical image and the medical report of the historical case with the highest similarity is calculated to make a threshold judgment. When the cosine similarity is lower than the corresponding threshold, a reflection and optimization process of the visual language large model is carried out. Based on the generated report, the large visual language model is optimized and trained using a reinforcement learning mechanism based on expert knowledge.

[0006] Furthermore, the expression for calculating the similarity between the embedding vector of the image modality and each vector in the vector database is as follows: In the formula, For the image modality embedding vector of the medical images of new cases, For the first in the vector database Embedded vectors of historical cases, Embedded vector and Cosine similarity between them.

[0007] Furthermore, during the threshold determination process, the cosine similarity between the generated report and the input medical image is calculated, as well as the cosine similarity between the generated report and the medical report of the historical case with the highest similarity, and then compared with the corresponding thresholds.

[0008] Furthermore, in the threshold judgment process, a dual threshold judgment mechanism is used for threshold comparison. The dual threshold judgment mechanism uses a static threshold in the early stage of training the visual language large model and a dynamic threshold in the later stage of training the visual language large model. The static threshold is determined based on the 3 / 4 quantile of the semantic similarity distribution of all samples in the training set of the visual language large model; the dynamic threshold is dynamically adjusted based on the 3 / 4 quantile of the semantic similarity distribution of each batch of training samples.

[0009] Furthermore, the reflection and optimization process of the aforementioned large visual language model includes: Generate an adaptive prompt containing image-report alignment warnings or report-report alignment suggestions. Input this adaptive prompt along with the medical images of the new case and the historical case with the highest similarity into the visual language big model, re-obtain the generated report, and perform iterative iterations until the calculated cosine similarity of the obtained generated report reaches the corresponding threshold or meets the corresponding number of iterations.

[0010] Furthermore, the expression for the reward function used in the expert knowledge-based reinforcement learning mechanism is as follows: In the formula, For mixed reward values, These are the weighting coefficients. The clinical effectiveness reward is calculated by extracting the relationship between clinical entities and diagnosis based on the medical reports of the currently generated report and the corresponding historical cases with the highest similarity, and calculating the precision, recall and F1 score of clinical entity recognition and diagnostic relationship modeling. This is a language quality metric, obtained by calculating the BLEU, METEOR, and ROUGE values ​​of the currently generated report.

[0011] Furthermore, the expert knowledge-based reinforcement learning mechanism calculates the in-group reward for each sample to determine the advantage value of each sample, thereby constructing the objective function. The expression for calculating the dominance value of each sample is: In the formula, For the sample In the t Advantage value at time step For the sample Mixed rewards, The average reward within the group. The standard deviation of the group's rewards. It is a constant.

[0012] Furthermore, the expression for the objective function is: In the formula, For strategy parameters, The objective function value, The total number of samples in each group. For the total time step, Limit the ratio to Within the range, The importance sampling ratio, For KL divergence weights, This is the old strategy. For the new strategy, As expected.

[0013] Furthermore, the expert-knowledge-based reinforcement learning mechanism retains only samples with high reward values ​​and significant variance for policy updates, and the corresponding samples satisfy the following: , In the formula, This is the variance threshold; The expert knowledge-based reinforcement learning mechanism for The function is decoupled and trimmed; the corresponding expression is: In the formula, Strict tailoring for low-end products. For a high-end, relaxed fit; The gradient update expression for the reinforcement learning mechanism based on expert knowledge is as follows: In the formula, For strategy The objective function value under the value.

[0014] This invention also provides a medical report generation system that implements the medical report generation method based on a multimodal large language model as described above, comprising: A visual encoder is used to extract embedding vectors from medical images of historical cases; A language decoder is used to extract embedding vectors from medical reports of historical cases; The data storage layer is used to store medical images and reports of historical cases and their corresponding embedding vectors; The preprocessing layer is used to clean, standardize, and enhance the medical images and reports stored in the data storage layer. The core functional layer is used to generate an image modality embedding vector based on the medical images of the new case, calculate the similarity between the image modality embedding vector and each vector in the vector database, obtain the historical case with the highest similarity as a prompt word, input the medical image of the new case and the prompt word into a pre-trained visual language model to generate a report; calculate the cosine similarity between the generated report and the input medical image and the medical report of the historical case with the highest similarity to perform a threshold judgment, and when the cosine similarity is lower than the corresponding threshold, perform a reflective optimization process for the visual language model; and optimize and train the visual language model based on the generated report through a reinforcement learning mechanism based on expert knowledge. The inference layer is used to achieve end-to-end generation from the input of new cases to the final generated report.

[0015] Compared with the prior art, the present invention has the following advantages: (1) The case retrieval mechanism proposed in this invention constructs a cross-modal clinical case retrieval system by simulating the diagnostic habits of radiologists in referring to historical cases. It stores medical images, medical reports and corresponding embedding vectors of historical cases. For new input cases, it retrieves the most similar historical cases through similarity matching and inputs them as part of the prompt words into the visual language big model to guide the model to perform contextualized reasoning. (2) The semantic reflection mechanism proposed in this invention monitors the semantic consistency of the generated report through static and dynamic dual thresholds. Different thresholds are used for judgment at different training stages of the model. The static threshold in the early stage of model training focuses on capturing early semantic shifts, while the dynamic threshold in the later stage of model training adapts to the quality screening needs after the model performance is improved. It can effectively correct problems such as relationship mismatch and diagnostic contradictions in the report.

[0016] (3) The reinforcement learning mechanism based on expert knowledge proposed in this invention constructs a hybrid reward function to realize reinforcement learning. The reward function adopts a hybrid design of clinical effectiveness and language quality. With expert knowledge as the core, it guides the model to generate reports that meet clinical standards, which can effectively balance clinical accuracy and language fluency and guide the model to optimize in the direction of meeting clinical standards. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a medical report generation method based on a multimodal large language model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram showing the comparison results of the various models provided in the embodiments of the present invention on key indicators; Figure 3 This is a radar chart comparing key indicators of various models provided in an embodiment of the present invention. Figure 4 This is a qualitative example diagram provided in an embodiment of the present invention to compare the report generation of the present invention with that of LLaVA-Rad. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] Example 1 like Figure 1 As shown, this embodiment provides a medical report generation method based on a multimodal large language model, including the following steps: Cross-modal case retrieval S1: Historical cases are pre-collected, each including medical images and medical reports. Embedding vectors for each medical image and medical report are obtained and stored in a vector database. Based on the medical images of the new case, an embedding vector for the image modality is generated, and the similarity between the embedding vector of the image modality and each vector in the vector database is calculated. The historical case with the highest similarity is selected as a prompt word. The medical images of the new case and the prompt word are input into a pre-trained visual language model to generate a report. Iterative Semantic Reflection S2: Calculate the cosine similarity between the generated report and the input medical image and the medical report of the historical case with the highest similarity, in order to make a threshold judgment, and when the cosine similarity is lower than the corresponding threshold, perform a reflection and optimization process of the visual language large model. Expert knowledge-guided reinforcement learning S3: Based on the generated report, the visual language large model is optimized and trained through an expert knowledge-based reinforcement learning mechanism.

[0022] Essentially, the above method consists of three parts: case retrieval, semantic reflection, and reward learning. Case retrieval uses the MedImageInsight (MI2) model to encode the current case and image-report pairs from the historical case database into a unified semantic space. It then utilizes the FAISS database for millisecond-level approximate nearest neighbor search, retrieving the top-1 most relevant historical cases in real time. This technique provides the model with interpretable clinical prior knowledge context. Semantic reflection also uses MI2 embeddings to calculate the cosine similarity between the generated text and the input image-text, serving as a semantic consistency quantification indicator. The RadGraph-XL model is an expert-level structured evaluator capable of parsing reports and accurately assessing the F1 score and relational accuracy of clinical entities. The reward learning mechanism deeply integrates this model into the reward function, fusing its output clinical effectiveness score with linguistic indicators to construct a multi-objective reward. An improved grouping relative policy optimization algorithm is employed, and variance-aware dynamic sampling technology is introduced, prioritizing high-reward, high-variance sample groups for policy updates to ensure efficient and stable training that is strictly aligned with clinical standards.

[0023] Specifically, step S1 is used to simulate the diagnostic habits of radiologists referring to historical cases and construct a cross-modal clinical case retrieval system. The expression for calculating the similarity between the embedding vector of the image modality and each vector in the vector database is as follows: In the formula, For the image modality embedding vector of the medical images of new cases, For the first in the vector database Embedded vectors of historical cases, Embedded vector and Cosine similarity between them.

[0024] In this embodiment, the specific process of step S1 includes: First, the MI2 embedding model is used to embed medical images. (such as ultrasound submodalities, X-ray images) and corresponding medical reports Mapping to a 768-dimensional semantic space yields embedding vectors for both the image and text modalities of the case. A case embedding library is constructed based on the FAISS library, storing images and reports along with their corresponding embedding vectors in the vector database, enabling efficient similarity matching. For a new input case, an embedding vector for the image modality is generated. Then, the similarity between this vector and all vectors in the embedding library is calculated using cosine similarity. .

[0025] System retrieval and The top-k most similar samples (selected in this example) Historical case studies. Finally, the retrieved historical case studies, including images and reports, are input into the model as part of the prompts to guide the model in contextualized reasoning.

[0026] Step S2 is used to simulate the process of radiologists self-verifying and iteratively optimizing diagnostic conclusions, thereby achieving dynamic monitoring and correction of report quality.

[0027] In the threshold determination process, the cosine similarity between the generated report and the input medical image is calculated, as well as the cosine similarity between the generated report and the medical report of the historical case with the highest similarity, and then compared with the corresponding thresholds.

[0028] Preferably, in the threshold judgment process, a dual threshold judgment mechanism is used for threshold comparison. The dual threshold judgment mechanism uses a static threshold in the early stage of training the visual language large model and a dynamic threshold in the later stage of training the visual language large model. The static threshold is determined based on the 3 / 4 quantile of the semantic similarity distribution of all samples in the training set of the large visual language model; the dynamic threshold is dynamically adjusted based on the 3 / 4 quantile of the semantic similarity distribution of each batch of training samples.

[0029] The preferred reflexive optimization process for the visual language large model includes: Generate an adaptive prompt containing image-report alignment warnings or report-report alignment suggestions. Input this adaptive prompt along with the medical images of the new case and the historical case with the highest similarity into the visual language big model, re-obtain the generated report, and perform iterative iterations until the calculated cosine similarity of the obtained generated report reaches the corresponding threshold or meets the corresponding number of iterations.

[0030] In this embodiment, the specific process of step S2 includes: Report generated based on MI2 embedding model With input image Reference Report Cosine similarity: in Each represents the generated report. With input image Reference Report The embedding vector.

[0031] To adapt to the model's performance at different training stages, the system employs a dual threshold judgment mechanism: static threshold. Used in the early stages of model training, based on the training set. The 3rd quartile of the semantic similarity distribution of all samples Determined to capture early semantic shifts; dynamic thresholding Used in the later stages of training, based on each batch of training samples. 3 / 4 quantile of semantic similarity distribution Dynamically adjust to adapt to the quality screening requirements after the model performance is improved: , When the semantic similarity of the generated report falls below the current threshold, the system automatically triggers a reflective optimization process: it generates adaptive prompts including "image-report alignment warning" and "report-report alignment hint," inputs these prompts along with the original input image and retrieved case into the model, and regenerates the report until the similarity reaches the threshold or the number of iterations does not exceed a certain limit. Next (selected in this embodiment) This ensures that reports maintain clinical consistency while remaining linguistically coherent.

[0032] Step S3 is used to simulate the learning process of radiologists aligning collective professional knowledge through peer communication and case discussions, and to build a reinforcement learning reward mechanism based on expert knowledge.

[0033] In this embodiment, the specific process of step S3 includes: The reward function employs a hybrid design that combines "clinical effectiveness" and "language quality": In the formula, Weighting coefficients (selected in this invention) This is used to balance clinical accuracy and linguistic fluency. The clinical effectiveness reward is calculated by the RadGraph-XL model, which extracts data to generate reports. Reference Report The clinical entity and diagnostic relationship are analyzed, and the precision, recall, and F1 score of entity recognition and relationship modeling are calculated as clinical effectiveness scores, directly reflecting the alignment between the report and radiology expertise; language quality indicators... It includes BLEU (assessing n-gram precision, reflecting the consistency of technical terminology), METEOR (assessing synonym processing and contextual ordering, such as synonym matching of "homogeneous echo" and "consistent echo"), and ROUGE (assessing the recall of diagnostic elements to ensure report completeness).

[0034] The reinforcement learning process employs a reconstructed group relative policy optimization algorithm and introduces a clip-higher mechanism for dynamic adversarial policy optimization to balance exploration and exploitation and prevent policy entropy collapse. The algorithm replaces the traditional value function with within-group reward standardization, directly calculating relative advantage based on Monte Carlo sampling, thus improving training stability and efficiency. When calculating the advantage function, for each group… For a batch consisting of 10 samples (from the same query suggestion), calculate the value of each sample. Advantages : in, For the sample Mixed rewards , The average reward within the group. The standard deviation of the group's rewards. It is a minimal constant to prevent division by zero. Objective function: in, The importance sampling ratio, Limit the ratio to Within the range, The new strategy is constrained by the KL divergence weights. Compared to the old strategy The distribution differences.

[0035] To avoid overfitting to common cases, the dynamic sampling mechanism of differential perception screening only retains samples with high reward values ​​and significant variance for policy updates. ,in A variance threshold is used to ensure that gradient signals originate from diverse clinical scenarios. To address the gradient blocking problem of traditional PPO on high-end reward signals, decoupled pruning of DAPO is introduced: in Strict cutting at the low end to control stability. High-end, lenient pruning allows high-reward samples to contribute to the gradient, encouraging the exploration of rare cases (in this invention). During gradient updates: This design ensures that high-reward samples can still propagate gradients, avoiding "gradient starvation," while low-reward samples are pruned to prevent unstable updates.

[0036] Example 2 This embodiment provides a medical report generation system that implements the medical report generation method based on a multimodal large language model as described in Embodiment 1, including: A visual encoder is used to extract embedding vectors from medical images of historical cases; A language decoder is used to extract embedding vectors from medical reports of historical cases; The data storage layer is used to store medical images and reports of historical cases and their corresponding embedding vectors; The preprocessing layer is used to clean, standardize, and enhance the medical images and reports stored in the data storage layer. The core functional layer is used to generate image modality embedding vectors based on the medical images of new cases, calculate the similarity between the image modality embedding vector and each vector in the vector database, select the historical case with the highest similarity as a prompt word, input the medical images of the new case and the prompt word into a pre-trained visual language model to generate a report; calculate the cosine similarity between the generated report and the input medical images and the medical reports of the historical case with the highest similarity to perform threshold judgment, and perform a reflective optimization process for the visual language model when the cosine similarity is lower than the corresponding threshold; and optimize and train the visual language model based on the generated report through a reinforcement learning mechanism based on expert knowledge. The inference layer is used to achieve end-to-end generation from the input of new cases to the final generated report.

[0037] The medical report generation system described above is described in detail below: I. System Overall Architecture This medical report generation system uses a multimodal large language model as its core architecture, specifically employing the Qwen2.5-VL-7B model as its basic backbone network. This model comprises two core components: a visual encoder and a language decoder. The visual encoder uses an improved VisionTransformer structure, with 12 layers employing a window attention mechanism and 4 layers employing a fully self-attention mechanism to capture global spatial context. Spatial location encoding is achieved through 2D RoPE, and the input image is segmented into 14×14 image blocks for feature extraction. The language decoder is initialized based on the Qwen-2.5 large language model, adapts to multimodal input through 1D RoPE, and optimizes computational efficiency using RMSnorm normalization and SwiGLU activation function. The system is divided into a data storage layer, a preprocessing layer, a core function layer, and an inference layer: the data storage layer uses the FAISS vector database to store cross-modal embeddings and related information of historical cases; the preprocessing layer is responsible for cleaning, standardizing, and enhancing medical images and report texts; the core function layer integrates a cross-modal case retrieval engine, an iterative semantic reflection module, and an expert knowledge-guided reinforcement learning module; and the inference layer realizes end-to-end generation from input cases to the final report.

[0038] II. Overall Process of Model Training and Optimization Considering the problems of large parameter scale, high memory consumption, and high training cost associated with full fine-tuning of multimodal large language models, this stage adopts the LoRA low-rank adaptation efficient fine-tuning strategy. This strategy significantly reduces parameter update volume and memory consumption while ensuring fine-tuning effectiveness, thus adapting to the model training and deployment needs in clinical scenarios. This invention employs a two-stage training strategy of "Supervised Fine-tuning (SFT) - Reinforcement Learning (RL)," the specific process of which is as follows: (I) First Stage: Supervised Fine-Tuning (SFT). 1. Data Selection: 1000 representative cases were selected from the dataset by clustering, covering various imaging manifestations and diagnostic scenarios. Supervised training was conducted to enable the model to initially grasp the writing standards of medical reports and basic diagnostic logic. 2. Training Configuration: LoRA low-rank matrices were introduced only for parameter fine-tuning of the linear layers in the Qwen2.5-VL-7B model. The pre-training parameters of the backbone model were not updated. The rank of the low-rank matrix was set to 8, the LoRA scaling factor to 32, and mixed precision training with bfloat16 was adopted. The initial learning rate was set to 1e-4, and a learning rate warm-up strategy was adopted, with a warm-up ratio of 0.05. 3. Model Saving: Validation was performed every 500 rounds. The BLEU-4 and RadGraph metrics were evaluated using the validation set. The model with the best performance on the validation set was saved as the result of the SFT stage and used for initialization in the subsequent RL stage.

[0039] (II) Second Stage: Reinforcement Learning (RL). During model initialization, the optimal model weights from the SFT stage are loaded as initial parameters for the policy network. During training, a case retrieval-semantic reflection-reward learning framework is introduced, as detailed below. All models were tested using Python and implemented using the PyTorch framework. The server was configured with an NVIDIA A100 TENSORCORE GPU and 48GB of RAM. The AdamW optimizer was used.

[0040] III. Specific Implementation of the Cross-Modal Case Retrieval Module The cross-modal case retrieval module aims to simulate the diagnostic habits of radiologists in referring to historical cases, and achieves efficient matching of images and text by constructing a unified semantic space.

[0041] The MedImageInsight (MI2) model was used as a cross-modal embedding tool. This model supports a unified semantic mapping between medical images and text. For an input medical image I, an image embedding is generated after visual branch processing. For the input report text R, text embeddings are generated through text branching. This enables the representation of images and text in a unified semantic space.

[0042] A case embedding library is built based on the FAISS database. Images and reports, along with their corresponding embedding vectors, are stored together in the vector database, enabling efficient similarity matching. For a new input case, an embedding vector for the image modality is generated. Then, the similarity between this vector and all vectors in the embedding library is calculated using cosine similarity. : in Indicates the first in the dataset Embedded vectors of individual cases, system retrieval and The top-k most similar samples (selected in this example) Historical case data is formatted as natural language prompts and embedded into the input template of the Qwen2.5-VL model. An example template is "Refer to similar historical cases: {retrieve case information}. Generate a medical report that meets clinical standards based on the current input image".

[0043] IV. Specific Implementation of the Iterative Semantic Reflection Module This module simulates the process of radiologists self-verifying and iteratively optimizing diagnostic conclusions, enabling dynamic monitoring and correction of report quality. Reports are generated based on the MI2 embedding model. With input image Reference Report Cosine similarity: in Each represents the generated report. With input image Reference Report The embedding vector. To adapt to the model's performance at different training stages, the system employs a dual thresholding mechanism: static threshold... Used in the early stages of model training, based on the training set. The 3rd quartile of the semantic similarity distribution of all samples Determined to capture early semantic shifts; dynamic thresholding Used in the later stages of training, based on each batch of training samples. 3 / 4 quantile of semantic similarity distribution Dynamically adjust to adapt to the quality screening requirements after the model performance is improved: , When the semantic similarity of the generated report falls below the current threshold, the system automatically triggers a reflective optimization process: it generates adaptive prompts including "image-report alignment warning" and "report-report alignment hint," inputs these prompts along with the original input image and retrieved case into the model, and regenerates the report until the similarity reaches the threshold or the number of iterations does not exceed a certain limit. Next (selected in this embodiment) This ensures that reports maintain clinical consistency while remaining linguistically coherent.

[0044] pass Figure 2 Tables 1 and 2 can verify the role of the iterative semantic reflection module. Figure 2 The results show that in the global training step, the average reward of the model without the semantic reflection module decreased by 3.4% compared to the full model, and the convergence stability was significantly reduced. Combined with Table 1, the BLEU-4 index of the model without the reflection module is 0.289, which is lower than the 0.316 of the full model. In Table 2, its RadGraph index is 0.398, which is lower than the 0.414 of the full model. This confirms that the reflection module effectively maintains the consistency of reported clinical data, reduces performance loss during training, and improves the stability of model output quality.

[0045] Table 1 Table 2 V. Expert-Guided Reinforcement Learning This module simulates the learning process of radiologists aligning their collective professional knowledge through peer communication and case discussions, constructing a reinforcement learning reward mechanism based on expert knowledge. The reward function adopts a hybrid design of "clinical effectiveness + language quality": in Weighting coefficients (selected in this embodiment) This is used to balance clinical accuracy and linguistic fluency. The clinical effectiveness reward is calculated by the RadGraph-XL model, which extracts data to generate reports. Reference Report The clinical entity and diagnostic relationship are analyzed, and the precision, recall, and F1 score of entity recognition and relationship modeling are calculated as clinical effectiveness scores, directly reflecting the alignment between the report and radiology expertise; language quality indicators... It includes BLEU (assessing n-gram precision, reflecting the consistency of technical terminology), METEOR (assessing synonym processing and contextual ordering, such as synonym matching of "homogeneous echo" and "consistent echo"), and ROUGE (assessing the recall of diagnostic elements to ensure report completeness).

[0046] The reinforcement learning process employs a reconstructed group relative policy optimization algorithm and introduces a clip-higher mechanism for dynamic adversarial policy optimization to balance exploration and exploitation and prevent policy entropy collapse. The algorithm replaces the traditional value function with within-group reward standardization, directly calculating relative advantage based on Monte Carlo sampling, thus improving training stability and efficiency. When calculating the advantage function, for each group… For a batch consisting of 10 samples (from the same query suggestion), calculate the value of each sample. Advantages : in, For the sample Mixed rewards , The average reward within the group. The standard deviation of the group's rewards. It is a minimal constant to prevent division by zero. Objective function: in, The importance sampling ratio, Limit the ratio to Within the range, The new strategy is constrained by the KL divergence weights. Compared to the old strategy The distribution differences.

[0047] To avoid overfitting to common cases, the dynamic sampling mechanism of differential perception screening only retains samples with high reward values ​​and significant variance for policy updates. ,in A variance threshold is used to ensure that gradient signals originate from diverse clinical scenarios. To address the gradient blocking problem of traditional PPO on high-end reward signals, decoupled pruning of DAPO is introduced: in Strict cutting at the low end to control stability. High-end, lenient pruning allows high-reward samples to contribute to the gradient, encouraging the exploration of rare cases (in this invention). During gradient updates: This design ensures that high-reward samples can still propagate gradients, avoiding "gradient starvation," while low-reward samples are pruned to prevent unstable updates.

[0048] pass Figure 2 Tables 1 and 2 can verify the effectiveness of the expert knowledge-guided reinforcement learning module. Figure 2The results show that the convergence threshold of the model without the reward module is 20.5% lower than that of the full model, and the mean reward increases slowly during training. In Table 1, the METEOR index of the model without the reward module is 0.370, which is lower than that of the full model (0.460). In Table 2, its ROUGE-L index is 0.438, which is lower than that of the full model (0.484). The full model, on the other hand, uses a hybrid reward function of "clinical effectiveness + language quality", which not only converges faster but also achieves the best results in both clinical effectiveness and language quality indicators. This confirms that the reward mechanism can effectively balance clinical accuracy and language fluency and guide the model to optimize in a direction that meets clinical standards.

[0049] This embodiment further demonstrates... Figure 3 Figure 4 Tables 1 and 2 verify the overall performance of the method of the present invention. Figure 2 The comparison results of various models on key indicators show that the present invention significantly outperforms traditional memory-driven transformer models, benchmark-based models, and other multimodal large models in terms of BLEU-4 (0.316), ROUGE-L (0.484), RadGraph (0.414), etc. The RadGraph indicator is 8.40% higher than LLaVA-Rad, which confirms the comprehensive superiority of the method of the present invention. Figure 4 For qualitative case comparison, it is shown that the report generated by the model proposed in this invention is more consistent with the reference report in terms of anatomical entity description and diagnostic relationship, while the comparison model has omissions in diagnostic information or expression deviations. Combined with the fact that the indicators of each ablation module in Tables 1 and 2 are lower than the whole model, it further confirms the necessity of the synergistic effect of the three modules of cross-modal case retrieval, iterative semantic reflection, and expert knowledge-guided reinforcement learning, which together ensure the clinical effectiveness and language quality of the report generated by the method of this invention.

[0050] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for generating medical reports based on a multimodal large language model, characterized in that, Includes the following steps: Historical cases are collected in advance, each including medical images and medical reports. The embedding vectors of each medical image and medical report are obtained and stored in the vector database. Based on the medical images of the new case, an embedding vector of the image modality is generated, and the similarity between the embedding vector of the image modality and each vector in the vector database is calculated. The historical case with the highest similarity is obtained as a prompt word. The medical images of the new case and the prompt word are input into a pre-trained visual language model to generate a report. The cosine similarity between the generated report and the input medical image and the medical report of the historical case with the highest similarity is calculated to make a threshold judgment. When the cosine similarity is lower than the corresponding threshold, a reflection and optimization process of the visual language large model is carried out. Based on the generated report, the large visual language model is optimized and trained using a reinforcement learning mechanism based on expert knowledge.

2. The medical report generation method based on a multimodal large language model according to claim 1, characterized in that, The expression for calculating the similarity between the embedding vector of the image modality and each vector in the vector database is as follows: In the formula, For the image modality embedding vector of the medical images of new cases, For the first in the vector database Embedded vectors of historical cases, Embedded vector and Cosine similarity between them.

3. The medical report generation method based on a multimodal large language model according to claim 1, characterized in that, During the threshold determination process, the cosine similarity between the generated report and the input medical image is calculated, as well as the cosine similarity between the generated report and the medical report of the historical case with the highest similarity, and then compared with the corresponding thresholds.

4. The medical report generation method based on a multimodal large language model according to claim 1, characterized in that, In the threshold judgment process, a dual threshold judgment mechanism is used to compare thresholds. The dual threshold judgment mechanism uses a static threshold in the early stage of training the visual language large model and a dynamic threshold in the later stage of training the visual language large model. The static threshold is determined based on the 3 / 4 quantile of the semantic similarity distribution of all samples in the training set of the visual language large model; the dynamic threshold is dynamically adjusted based on the 3 / 4 quantile of the semantic similarity distribution of each batch of training samples.

5. The medical report generation method based on a multimodal large language model according to claim 1, characterized in that, The reflection and optimization process of the large visual language model includes: Generate an adaptive prompt containing image-report alignment warnings or report-report alignment suggestions. Input this adaptive prompt along with the medical images of the new case and the historical case with the highest similarity into the visual language big model, re-obtain the generated report, and perform iterative iterations until the calculated cosine similarity of the obtained generated report reaches the corresponding threshold or meets the corresponding number of iterations.

6. The medical report generation method based on a multimodal large language model according to claim 1, characterized in that, The expression for the reward function used in the expert knowledge-based reinforcement learning mechanism is as follows: In the formula, For mixed reward values, These are the weighting coefficients. The clinical effectiveness reward is calculated by extracting the relationship between clinical entities and diagnosis based on the medical reports of the currently generated report and the corresponding historical cases with the highest similarity, and calculating the precision, recall and F1 score of clinical entity recognition and diagnostic relationship modeling. This is a language quality metric, obtained by calculating the BLEU, METEOR, and ROUGE values ​​of the currently generated report.

7. The medical report generation method based on a multimodal large language model according to claim 1, characterized in that, The expert-knowledge-based reinforcement learning mechanism calculates the in-group reward for each sample to determine the advantage value of each sample, thereby constructing an objective function. The expression for calculating the dominance value of each sample is: In the formula, For the sample In the t Advantage value at time step For the sample Mixed rewards, The average reward within the group. The standard deviation of the group's rewards. It is a constant.

8. The medical report generation method based on a multimodal large language model according to claim 7, characterized in that, The expression for the objective function is: In the formula, For strategy parameters, The objective function value, The total number of samples in each group. For the total time step, Limit the ratio to Within the range, The importance sampling ratio, For KL divergence weights, This is the old strategy. For the new strategy, As expected.

9. A medical report generation method based on a multimodal large language model according to claim 7, characterized in that, The expert-knowledge-based reinforcement learning mechanism retains only samples with high reward values ​​and significant variance for policy updates, and the corresponding samples satisfy the following: ， In the formula, This is the variance threshold; The expert knowledge-based reinforcement learning mechanism for The function is decoupled and trimmed; the corresponding expression is: In the formula, Strict tailoring for low-end products. For a high-end, relaxed fit; The gradient update expression for the reinforcement learning mechanism based on expert knowledge is as follows: In the formula, For strategy The objective function value under the value.

10. A medical report generation system that implements the medical report generation method based on a multimodal large language model as described in any one of claims 1-9, characterized in that, include: A visual encoder is used to extract embedding vectors from medical images of historical cases; A language decoder is used to extract embedding vectors from medical reports of historical cases; The data storage layer is used to store medical images and reports of historical cases and their corresponding embedding vectors; The preprocessing layer is used to clean, standardize, and enhance the medical images and reports stored in the data storage layer. The core functional layer is used to generate an image modality embedding vector based on the medical images of the new case, calculate the similarity between the image modality embedding vector and each vector in the vector database, obtain the historical case with the highest similarity as a prompt word, input the medical images of the new case and the prompt word into a pre-trained visual language large model, and generate a report. The cosine similarity between the generated report and the input medical image and the medical report of the historical case with the highest similarity is calculated to make a threshold judgment. When the cosine similarity is lower than the corresponding threshold, a reflection and optimization process of the visual language large model is carried out. Based on the generated report, the large visual language model is optimized and trained using a reinforcement learning mechanism based on expert knowledge. The inference layer is used to achieve end-to-end generation from the input of new cases to the final generated report.

Images