A medical report generation method based on counterfactual reasoning

Abstract

The application discloses a medical report generation method based on counterfactual reasoning, which comprises a given original medical organ image, inputting the original medical organ image into a large language model after modal conversion mapping of a visual encoder and a connector, obtaining an original probability distribution through output prediction of a current time word element, generating an attention distribution map from the original probability distribution, dividing the attention distribution map into a key visual information region and an irrelevant visual information region, applying Gaussian noise disturbance to the irrelevant visual information region to construct a counterfactual sample, obtaining a counterfactual probability distribution through the same method, obtaining a contrast probability distribution through the difference between the original probability distribution and the counterfactual probability distribution, designing an adaptive rationality constraint algorithm, optimizing the distribution, enhancing the confidence of the model to the direct decision, and designing a counterfactual sample construction strategy and a counterfactual contrast decoding strategy, improving the attention degree of the model to the key visual region, inhibiting the attention to the irrelevant region, and effectively relieving the illusion of the medical report generation model.

Description

A Medical Report Generation Method Based on Counterfactual Reasoning Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a method for generating medical reports based on counterfactual reasoning. Background Technology

[0002] The goal of medical report generation is to analyze organs and potential lesions in medical images to generate accurate text reports to support clinical diagnosis and decision-making. Currently, the interpretation of medical images and the writing of reports typically rely on manual methods. While this approach provides highly personalized diagnostic results, it is inefficient, susceptible to subjective factors, and carries the potential risk of diagnostic errors, especially when processing large volumes of image data, significantly increasing the workload for physicians. To improve efficiency and reduce human error, automated medical report generation technology has emerged. This technology combines computer vision and natural language generation techniques to automatically analyze medical images and generate professional diagnostic reports, thereby supporting physicians and helping to improve the speed and accuracy of diagnoses.

[0003] With the rapid development of artificial intelligence (AI) technology, especially the emergence of deep learning and multimodal large models, medical report generation has ushered in new development opportunities. Multimodal large models can process image and text information simultaneously, overcoming the limitations of traditional unimodal methods. They can extract lesion information from images and transform it into linguistic expression, demonstrating enormous potential in medical report generation tasks. By combining medical images with clinical knowledge, these models can not only generate image descriptions but also generate diagnostic text based on multidimensional clinical data, providing more accurate and comprehensive medical reports.

[0004] However, effectively incorporating clinical diagnostic information (such as lesion information, disease status, and patient history) into the model remains a challenge in current research. In particular, the model often fails to adequately focus on key lesion areas during decoding. This is often due to significant data imbalance in real-world medical report datasets: descriptions of normal lesions predominate, while descriptions of abnormal lesions are relatively scarce. This inherent data distribution discrepancy inevitably introduces data bias in supervised learning paradigms, causing the model to tend to generate benign diagnostic results, leading to the so-called "diagnostic illusion." This "illusion" not only causes the generated report to deviate from the actual image content but can also pose significant risks in clinical applications, misleading doctors' diagnostic decisions.

[0005] Therefore, there is an urgent need for a method that can effectively solve the above-mentioned "illusion" problem in the task of generating medical report models. Summary of the Invention

[0006] (1) Technical problems to be solved

[0007] This invention provides a medical report generation method based on counterfactual reasoning, which aims to solve the "illusion" problem caused by imbalanced training data in existing medical report generation models.

[0008] (2) Technical solution

[0009] This invention discloses a method for generating medical reports based on counterfactual reasoning, comprising the following steps:

[0010] Step 1: For a given original medical organ image, the original visual text information is generated by sequentially passing through a visual encoder and a connector. The original visual text information is then input into a large language model to obtain the original probability distribution.

[0011] Step 2: Generate an attention distribution map by backpropagating the original probability distribution through gradient backpropagation and normalization. Divide the attention distribution map into key visual information regions and irrelevant visual information regions using a connected component analysis algorithm. Apply Gaussian noise perturbation to the key visual information regions to construct counterfactual samples.

[0012] Step 3: Replace the original medical organ image in Step 1 with the counterfactual sample, and then obtain the counterfactual probability distribution using the same generation method as in Step 1;

[0013] Step 4: Obtain the contrast probability distribution from the difference between the original probability distribution and the counterfactual probability distribution, and design an adaptive rationality constraint algorithm to optimize the contrast probability distribution and enhance the confidence of the large language model for direct decision-making.

[0014] Further, step 1 includes the following steps:

[0015] Step 101: For a given original medical organ image, it is converted into original visual sequence features by a visual encoder;

[0016] Step 102: The original visual sequence features are transformed from image modality to text space through the connector to obtain the original visual cue information;

[0017] Step 103: Input the original visual cue information into the large language model and generate the original descriptive text through autoregression. The original descriptive text represents the lexical tags predicted at the current time step.

[0018] Step 104: For each time step, obtain the original probability distribution based on the prediction output of the word at the current time step by the large language model.

[0019] Furthermore, step 2 includes the following steps:

[0020] Step 201: Perform gradient backpropagation on the original probability distribution to obtain the gradient weight of each visual feature, and normalize the gradient weight to obtain the attention distribution map.

[0021] Step 202: For the attention distribution map, set a threshold, take the visual features with attention values ​​higher than the threshold as the visual information that the model pays attention to, and use the connected component analysis algorithm to divide it into the key visual information region and the irrelevant visual information region.

[0022] Step 203: Add Gaussian noise perturbation to the key visual information region to construct the counterfactual sample.

[0023] Further, in step 202, the visual information is divided into key visual information regions and irrelevant visual information regions using a connected component analysis algorithm, including the following steps:

[0024] Step 20201: Divide the original image into multiple clusters based on its size using a connected component analysis algorithm;

[0025] Step 20202: Select the region with the largest area and the highest attention value within the cluster region as the key visual information region;

[0026] Step 20203: The region outside the key visual information region in the cluster region is regarded as the irrelevant visual information region.

[0027] Furthermore, in step 203, applying Gaussian noise perturbation to the key visual information region to construct counterfactual samples includes the following steps:

[0028] Step 20301: Based on the distribution of image blocks in the key visual information region according to the attention distribution map, match them with the organ detection boxes provided in the Chest ImaGenome dataset, and obtain the attention score of the region where each organ detection box is located;

[0029] Step 20302: Extract the organ detection boxes corresponding to the regions with high attention scores, and add Gaussian noise to the corresponding organ detection boxes to obtain counterfactual samples.

[0030] Furthermore, step 3 includes the following steps:

[0031] Step 301: For the counterfactual sample, it is processed by a visual encoder and converted into the counterfactual visual sequence features;

[0032] Step 302: The counterfactual visual sequence features are transformed from image mode to text space through the connector to obtain counterfactual visual cue information;

[0033] Step 303: Input the counterfactual visual cue information into the large language model, and generate counterfactual descriptive text through autoregression, wherein the counterfactual descriptive text represents the lexical tags predicted at the current time step;

[0034] Step 304: For each time step, based on the prediction output of the word at the current time step by the large language model, obtain the counterfactual probability distribution.

[0035] Furthermore, step 4 includes the following steps:

[0036] Step 401: Calculate the contrast probability distribution based on the difference between the original probability distribution and the counterfactual probability distribution;

[0037] Step 402: Based on the confidence level of the output distribution related to the original visual input, design adaptive rationality constraints for the lexical units corresponding to the contrast probability distribution, and filter the vocabulary through the adaptive rationality constraints to obtain an effective vocabulary.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] This paper proposes a medical report generation algorithm based on counterfactual reasoning, aiming to alleviate the "hallucination" phenomenon in medical report generation tasks. Under a training-free paradigm, the algorithm improves the model's focus on key visual regions and suppresses attention to irrelevant regions by designing counterfactual sample construction and counterfactual comparison decoding strategies, thereby mitigating the "hallucination" effect. Attached Figure Description

[0040] Figure 1 shows a flowchart of the method for implementing the present invention;

[0041] Figure 2 shows a flowchart of step 1 of the present invention; Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] To better understand the technical content of this embodiment, let's first introduce the explanation of the terms involved in this embodiment.

[0044] Counterfactual reasoning: refers to the process by which a model understands or generates different possible scenarios by speculating and simulating outcomes under different assumptions.

[0045] Visual encoder: In computer vision tasks, it refers to a neural network model used to extract meaningful features from raw image or video data, transforming high-dimensional input image data into low-dimensional feature representations.

[0046] Autoregressive: refers to a model that, when generating sequential data (such as text, time series, etc.), depends on previously generated elements when generating each new element.

[0047] Lexical tokens: These refer to the units after words, subwords, characters, etc., are segmented and encoded in text processing.

[0048] Backpropagation: An algorithm used to train neural networks. It calculates the gradient of the loss function with respect to the parameters of each layer in the network, and then uses the backpropagation algorithm to pass the gradient information back to each layer to update the network weights in order to minimize the loss function.

[0049] Gradient weights refer to the magnitude of the gradient of each parameter in the network during backpropagation, representing the degree of contribution of a specific visual feature to the prediction result.

[0050] Normalization: refers to the process of converting data into a standard range (such as between 0 and 1), which is usually used to improve the stability and efficiency of models.

[0051] Attention map: A visualization technique used to show which parts of the input data are most important to the final decision when a neural network makes predictions. In vision tasks, attention maps are typically used to represent the attention weight of each region in an image, highlighting regions that have a significant impact on the model's output through variations in color or brightness.

[0052] Gaussian noise, also known as normal noise, is a type of noise that follows a Gaussian distribution (or normal distribution). In fields such as image processing and signal processing, Gaussian noise is often used to simulate or test the robustness of a system to noise.

[0053] Hyperparameters are parameters that need to be manually set during the training of a machine learning model. These parameters are not optimized through the model's own learning process, but are defined before training.

[0054] This invention discloses a method for generating medical reports based on counterfactual reasoning. Figure 1 shows a flowchart of an embodiment of this invention, including the following steps:

[0055] Step 1: For a given raw medical organ image After passing through the visual encoder Processing and converting into raw visual sequence features The original visual sequence features The modality conversion mapping by the connector is converted into raw visual text information. Then, the original visual text information Input to large language model At each time step, by the large language model The original probability distribution is obtained by predicting the output of the word at the current time. ;

[0056] Step 2: Convert the original probability distribution Attention distribution map is generated through gradient backpropagation and normalization. Using connected component analysis algorithms to The attention distribution map Divided into key visual information areas and irrelevant visual information areas For the key visual information region Constructing counterfactual samples by applying Gaussian noise perturbation ;

[0057] Step 3: For the counterfactual sample Using the original probability distribution Using the same generation method, a counterfactual probability distribution is obtained. ;

[0058] Step 4: From the original probability distribution and counterfactual probability distribution The differences between them yielded a comparative probability distribution. An adaptive rationality constraint algorithm is designed to optimize the comparison probability distribution. Enhance large language models Confidence level regarding direct decision-making.

[0059] As shown in Figure 2, the specific analysis of step 1 is as follows:

[0060] Specifically, step 1 includes the following steps:

[0061] Step 101: For a given raw medical organ image After passing through the visual encoder Processing and converting into raw visual sequence features ,in, , Represents the space of real numbers. Indicates the length of visual sequence features. Indicates the size of the feature dimension. Indicates the first Local visual features encoded from image patches, ,in This represents a real matrix with 1 row and L columns;

[0062] Step 102: Transfer the original visual sequence features After modal conversion by the connector, the image modality is mapped to the text space to obtain the original visual cue information. ;

[0063] Step 103: Transfer the original visual cue information Input to large language model The original descriptive text is generated through autoregression. ,in Indicates the current time step Predicted lexical tags;

[0064] Step 104: For each time step, based on the large language model probability distribution ,in The large language model The original output value in the output layer, Normalize the given input to a large language model. The original output values ​​are converted into the original probability distribution.

[0065] The specific explanation of step 2 is as follows:

[0066] Specifically, step 2 includes the following specific steps:

[0067] Step 201: For the original probability distribution Gradient backpropagation is performed to obtain the gradient weights of each visual feature, and the gradient weights are normalized to obtain the attention distribution map. ,in, ;

[0068] Step 202: Analyze the attention distribution map Set threshold The attention distribution map Medium above the threshold Visual features as a large language model Visual information that the model focuses on The system was divided into key visual information regions using a connected component analysis algorithm. and irrelevant visual information areas ;

[0069] Step 203: For the key visual information region Apply Gaussian noise perturbation to construct counterfactual samples .

[0070] Specifically, step 202 includes the following specific steps:

[0071] Step 20201: For the key visual information region The original medical organ image I was divided into multiple clusters based on its size using a connected component analysis algorithm.

[0072] Step 20202: Select the region with the largest area and the highest attention value within the cluster region as the key visual information region. ;

[0073] Step 20203: Identify the key visual information regions within the cluster region. The area outside is considered as an area of ​​irrelevant visual information. .

[0074] Key visual information areas Representing a large language model The diagnosis of the relevant area revealed key visual information and areas of irrelevant visual information. Representing a large language model Due to spurious correlations in the training data, the system mistakenly focuses on other regions in the image when diagnosing lesions. The information in these regions is not important for the correct diagnosis and may even be misleading.

[0075] Specifically, step 203 includes the following specific steps:

[0076] Step 20301: Based on the attention distribution map In key visual information areas The image patch distribution is mapped to the organ detection boxes provided in the Chest ImaGenome dataset, and the attention score of the region where each organ detection box is located is obtained.

[0077] Step 20302: Extract the organ detection boxes corresponding to the high attention score regions mentioned above, and add Gaussian noise to the organ detection boxes in the high attention score regions to obtain counterfactual samples. ,in , For the mask matrix, This represents a noisy image block, where i and j represent the row and column indices of the original medical organ image I, respectively. When the image pixel... When the organ detection frame is in If the value is 1, Gaussian noise is added; otherwise, it is not added.

[0078] The specific explanation of step 3 is as follows:

[0079] To make large number language models During the decoding output process, more attention can be paid to the visual features of key regions, while ignoring the influence of spurious correlations caused by the training distribution. Predefined Gaussian noise is added to key regions to construct counterfactual samples. .

[0080] Specifically, step 3 includes the following steps:

[0081] Step 301: For the counterfactual sample After passing through the visual encoder Processing is converted into counterfactual visual sequence features ,in, , Indicates the length of visual sequence features. Indicates the size of the feature dimension. Indicates the first Local visual features encoded from image patches, where This represents a real matrix with 1 row and L columns;

[0082] Step 302: Transfer the counterfactual visual sequence features After modal conversion by the connector, the image modality is mapped to the text space to obtain counterfactual visual cues. ;

[0083] Step 303: Present the counterfactual visual cue information Input to large language model Counterfactual descriptive text is generated through autoregression. Counterfactual description text Indicates the current time step Predicted lexical tags;

[0084] Step 304: For each time step, based on the large language model The counterfactual probability distribution is obtained from the predicted output of the word at the current time. .

[0085] The specific explanation of step 4 is as follows:

[0086] Specifically, step 4 includes the following specific steps:

[0087] Step 401: Based on the original probability distribution and counterfactual probability distribution The difference between them was used to calculate the contrast probability distribution. Where α is a hyperparameter, a larger α value indicates that the difference between the two distributions is amplified, prompting the model to pay more attention to the expression consistent with the real visual information when generating text, and reduce the adoption of erroneous information;

[0088] Step 402: Based on the confidence level of the output distribution associated with the original visual input, for the contrast probability distribution Corresponding lexical units Design adaptive rationality constraints , ,in, It is the model's output vocabulary. It is any word element in the vocabulary list. Indicates the previous generation of the model Each word element, It is between Hyperparameters between Indicates according to and current visual input The vocabulary is filtered using adaptive rationality constraints to obtain an effective vocabulary. Specifically, only when the current word element... Predicted probability Greater than or equal to the maximum probability Only when the term is doubled will the word element be used. Incorporated middle. Indicates if not Effective vocabulary list In the middle, the large language model This terminology will not be generated.

[0089] Adaptive rationality constraints enable large language models It adjusts its behavior more precisely, penalizing only the error probability distribution caused by the model being affected by counterfactual sample inputs, without suppressing those outputs that are still reasonable.

[0090] The technical principle of the method of this invention is: to propose a medical report generation algorithm based on counterfactual reasoning, which forms counterfactual samples by weakening the disease facts of the original samples, thereby deepening the understanding of the disease conditions of the original samples and alleviating the "illusion" problem caused by data bias in the medical report generation task.

[0091] Specifically, a counterfactual sample construction strategy is first designed. This strategy utilizes the model's attention distribution map to distinguish between critical visual information regions essential for diagnosis and irrelevant visual information regions. Then, samples that contradict the diagnostic facts of the original samples are generated through targeted noise addition. For these counterfactual samples, a counterfactual contrast decoding strategy is proposed. This dynamically corrects the probability distribution generated during the decoding process of the original samples. This enhances the model's ability to perceive and rely on key visual regions while suppressing the harmful effects of irrelevant regions or prior biases, ultimately mitigating the "illusion."

[0092] In this invention, no additional model training or reliance on external pre-trained models is required, thus offering the advantages of cost-effectiveness.

[0093] The technical features or steps of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features or steps in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A method for generating medical reports based on counterfactual reasoning, characterized in that, Includes the following steps: Step 1: For a given original medical organ image, the image is sequentially processed by a visual encoder and connector to generate original visual text information. This original visual text information is then input into a large language model to obtain the original probability distribution. Step 2: The original probability distribution is processed through gradient backpropagation and normalization to generate an attention distribution map. A connected component analysis algorithm is used to divide the attention distribution map into key visual information regions and irrelevant visual information regions. Gaussian noise perturbation is applied to the key visual information regions to construct counterfactual samples. Step 3: The original medical organ image from Step 1 is replaced with the counterfactual samples. The same generation method as in Step 1 is then used to obtain the counterfactual probability distribution. Step 4: A contrast probability distribution is obtained from the difference between the original probability distribution and the counterfactual probability distribution. An adaptive rationality constraint algorithm is designed to optimize the contrast probability distribution and enhance the confidence of the large language model for direct decision-making.

2. The method for generating medical reports based on counterfactual reasoning according to claim 1, characterized in that, Step 1 includes the following steps: Step 101: For a given original medical organ image, it is processed by a visual encoder and converted into original visual sequence features; Step 102: The original visual sequence features are modally transformed by a connector to map from the image modality to the text space to obtain the original visual cue information; Step 103: The original visual cue information is input into a large language model, and original descriptive text is generated through autoregression, wherein the original descriptive text represents the lexical tokens predicted at the current time step; Step 104: For each time step, the original probability distribution is obtained based on the prediction output of the lexical tokens at the current time step by the large language model.

3. The method for generating medical reports based on counterfactual reasoning according to claim 2, characterized in that, Step 2 includes the following steps: Step 201: Perform gradient backpropagation on the original probability distribution to obtain the gradient weight of each visual feature, and normalize the gradient weight to obtain an attention distribution map; Step 202: Set a threshold for the attention distribution map, and take visual features with attention values ​​higher than the threshold as the visual information that the model pays attention to, and use a connected component analysis algorithm to divide the visual information into the key visual information region and the irrelevant visual information region; Step 203: Add Gaussian noise perturbation to the key visual information region to construct the counterfactual sample.

4. The method for generating medical reports based on counterfactual reasoning according to claim 3, characterized in that, In step 202, the visual information is divided into key visual information regions and irrelevant visual information regions using a connected component analysis algorithm, including the following steps: Step 20201: Divide the original image into multiple clusters based on its size using a connected component analysis algorithm; Step 20202: Select the region with the largest area and the highest attention value within the cluster region as the key visual information region; Step 20203: Select the region outside the key visual information region within the cluster region as the irrelevant visual information region.

5. The method for generating medical reports based on counterfactual reasoning according to claim 3, characterized in that, In step 203, applying Gaussian noise perturbation to the key visual information region to construct counterfactual samples includes the following steps: Step 20301: Based on the image patch distribution of the attention distribution map in the key visual information region, match it with the organ detection boxes provided in the ChestImaGenome dataset, and obtain the attention score of the region where each organ detection box is located; Step 20302: Extract the organ detection boxes corresponding to the regions with high attention scores, add Gaussian noise to the corresponding organ detection boxes, and obtain counterfactual samples.

6. The method for generating medical reports based on counterfactual reasoning according to claim 1, characterized in that, Step 3 includes the following steps: Step 301: For the counterfactual sample, it is processed by a visual encoder and converted into the counterfactual visual sequence feature; Step 302: The counterfactual visual sequence feature is modally transformed by a connector and mapped from the image modality to the text space to obtain counterfactual visual cue information; Step 303: The counterfactual visual cue information is input into a large language model, and counterfactual descriptive text is generated through autoregression, wherein the counterfactual descriptive text represents the lexical tokens predicted at the current time step; Step 304: For each time step, the counterfactual probability distribution is obtained based on the predicted output of the lexical tokens at the current time step by the large language model.

7. A method for generating medical reports based on counterfactual reasoning according to claim 6, characterized in that, Step 4 includes the following steps: Step 401: Calculate the contrast probability distribution based on the difference between the original probability distribution and the counterfactual probability distribution; Step 402: Based on the confidence level of the output distribution related to the original visual input, design adaptive rationality constraints for the lexicons corresponding to the contrast probability distribution, and filter the vocabulary through the adaptive rationality constraints to obtain an effective vocabulary.

Images