A medical report generation method based on multi-modal pathological detail memory retrieval enhancement
By constructing visual and textual memory banks and combining them with a multimodal pathological detail memory retrieval enhancement method using Hopfield networks, the problem of insufficient visual pattern memory in existing technologies is solved, enabling high-quality generation of medical imaging reports and improving the accuracy and clinical consistency of reports.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-12
AI Technical Summary
Existing methods for generating medical image reports lack a dual memory mechanism of visual pattern memory and textual knowledge memory, resulting in limited ability to identify lesion areas and insufficient accuracy and relevance of generated reports.
We construct visual and textual memory banks, and through a process of classification followed by retrieval, we utilize the Hopfield network to enhance multimodal pathological detail memory retrieval and generate medical reports.
It effectively reduces irrelevant search noise in report generation, improves clinical consistency and generation quality, and enhances the accuracy and relevance of reports.
Smart Images
Figure CN122201590A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing technology, specifically relating to a medical report generation method based on multimodal pathological detail memory retrieval enhancement. Background Technology
[0002] Automatic generation of medical image reports is an important interdisciplinary field connecting computer vision and natural language processing. It aims to build an intelligent system capable of simulating radiologists and automatically generating structured diagnostic reports from medical images. Compared to general image description tasks, this task is characterized by longer text lengths, stricter structural requirements, and a high density of technical terminology. Furthermore, lesions in medical images are often subtle, have low contrast, and are easily affected by complex anatomical structures, placing higher demands on the model's perception and descriptive capabilities.
[0003] Current mainstream methods mostly adopt the "encoding-retrieval enhancement-decoding" framework, which has made some progress, but still has significant shortcomings in mimicking the real clinical diagnostic cognitive process, mainly in the following two aspects:
[0004] First, existing methods largely focus on building and utilizing textual knowledge bases, aiding in generation by retrieving similar text from historical reports. This merely simulates the textual knowledge retrieval process in physician diagnosis, neglecting the equally important visual pattern memory mechanism. In actual diagnosis, physicians rely not only on textual memory of past reports but also on visual memory templates of typical disease imaging features—including the morphology and texture of lesions and other local visual patterns. Existing models often lack the ability to explicitly model and memorize local lesion areas, making it difficult to compare and focus on visual features, thus limiting the ability to identify lesion areas.
[0005] Second, existing retrieval enhancement methods typically perform cross-modal matching between the visual features of the entire image and textual memory. This "image feature directly linked to textual memory" approach lacks clear clinical semantic guidance and is essentially a blind similarity retrieval. It ignores the "identify first, refine later" cognitive logic followed by physicians in actual diagnosis: that is, first perform preliminary pathological identification and classification of the image to form a semantically clear clinical hypothesis, and then retrieve relevant professional knowledge and descriptive templates based on this hypothesis. Existing methods skip the intermediate steps of classification and characterization, attempting to directly map from low-level visual signals to high-level text, which can easily lead to retrieval deviating from the true pathological content, introducing irrelevant information, and thus affecting the accuracy and relevance of the generated report.
[0006] Therefore, how to construct a dual memory mechanism that simultaneously possesses visual pattern memory and textual knowledge memory, and how to achieve accurate and interpretable report generation through a process of classification followed by retrieval, has become a key issue that urgently needs to be addressed in this field. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention proposes a medical report generation method based on multimodal pathological detail memory retrieval enhancement. The method includes: acquiring a medical image to be processed and inputting it into a trained medical report generation model to obtain a medical report.
[0008] The training process for the medical report generation model includes:
[0009] S1: Obtain the image-text pair dataset and preprocess it to obtain the preprocessed image-text pair dataset;
[0010] S2: The image is processed using a pre-trained medical vision model to obtain local features of the image patch and the disease classification result of the image patch; text features are extracted from the text, and the disease classification result of the text is obtained.
[0011] S3: Construct a visual memory bank based on local features of image patches and disease classification results of image patches; construct a text memory bank based on text features and disease classification results of text.
[0012] S4: Enhance the local features of the image patch by retrieving and strengthening them based on the visual memory bank to obtain enhanced local features;
[0013] S5: Fill the prompt word template with the disease classification results of the image patch and encode the prompt word template to obtain the prompt template encoding features;
[0014] S6: Enhance the text prompt features by retrieving and strengthening the coding features of the prompt template based on the text memory;
[0015] S7: Fuse enhanced local features and enhanced text prompt features to obtain fused features; input the disease classification results of the image patch as prompt word tags and fused features into the decoder for processing to obtain the medical report;
[0016] S8: Calculate the total loss of the model and iteratively adjust the model parameters based on the total loss to obtain the trained medical report generation model.
[0017] Preferably, the training process for a medical vision model includes:
[0018] The image was divided into 196 non-overlapping image blocks, each with a size of 16×16 pixels;
[0019] The image patches are converted into a labeled representation through a linear projection layer;
[0020] The labeled representation is input into the Transformer model for visual feature extraction, resulting in local features of image patches and global features of the image.
[0021] The global features of the image are input into the classifier to obtain the disease classification result of the image;
[0022] The classification loss of the medical vision model is calculated based on the disease classification results of the images. The model parameters are then adjusted based on the classification loss to obtain a trained medical vision model.
[0023] Furthermore, the classification loss of the medical vision model is expressed as:
[0024]
[0025] in, This represents the classification loss of a medical vision model. Indicates the total number of diseases. Indicates the number of samples. Indicates the number of disease states. Indicates the first The sample at the th This disease belongs to a state The true label, The first prediction made by the medical vision model The sample at the th This disease belongs to a state The probability of.
[0026] Preferably, the process of constructing a visual memory bank includes:
[0027] Generate a heatmap of the image and calculate the average activation value of each image patch in the heatmap;
[0028] When the average activation value of an image patch is greater than a preset activation threshold, the local features of the image patch are added to the initial visual memory bank corresponding to the disease type of the image patch.
[0029] Calculate the mean feature center of local features of all image patches in the initial visual memory bank;
[0030] Select the k nearest neighbor samples of the mean feature center to construct a compact subset corresponding to the disease type;
[0031] By merging compact subsets of all disease types, the final visual memory bank is obtained.
[0032] Furthermore, the formula for calculating the average activation value of an image patch is:
[0033]
[0034] in, Represents image blocks The average activation value in Represents image blocks Total number of pixels in Representing a heat map The activation value at pixel (x,y).
[0035] Preferably, the process of constructing a text memory includes:
[0036] For each disease, an initial text memory is constructed based on the corresponding text features;
[0037] Calculate the mean feature center of all text features in the initial text memory;
[0038] Select the k nearest neighbor samples of the mean feature center to construct a compact subset corresponding to the disease type;
[0039] By merging compact subsets of all disease types, the final text memory is obtained.
[0040] Preferably, the method of enhancing the retrieval of local features of image patches based on the visual memory bank is represented as follows:
[0041]
[0042] in, This indicates enhancement of local features. Representing local features of an image patch, Represents the visual memory bank, This represents the Hopfield network.
[0043] Preferably, the retrieval enhancement representation based on the text memory for the coding features of the prompt template is as follows:
[0044]
[0045] in, This indicates enhanced text prompt features. Represents the Hopfield network. This indicates the encoding characteristics of the prompt template.
[0046] Preferably, the formula for calculating the total loss of the model is:
[0047]
[0048] in, This represents the total loss of the model. Indicates the report length. This represents the text token generated at time t. Indicates fusion characteristics, Indicates prompt word marker, This represents conditional probability.
[0049] The beneficial effects of this invention are as follows:
[0050] This invention effectively reduces pathology-irrelevant retrieval noise during the report generation model's search process, while improving clinical consistency and generation quality. Extensive experiments on the MIMIC-CXR dataset demonstrate that the proposed model achieves competitive performance, validating its effectiveness and practical value.
[0051] In terms of clinical value, this invention can effectively alleviate the workload of radiologists. By comprehensively covering various typical and atypical imaging manifestations, it provides diagnostic references for physicians with different experience levels, helping to promote the balanced allocation of medical resources. The generated standardized reports can be seamlessly integrated with hospital information systems, supporting an intelligent auxiliary closed loop from imaging diagnosis to clinical decision-making. This provides a key technological path for promoting the construction of highly reliable and interpretable AI-assisted diagnostic systems, and helps to achieve widespread access to high-quality medical resources. Attached Figure Description
[0052] Figure 1 This is a flowchart of the medical report generation method based on multimodal pathological detail memory retrieval enhancement in this invention;
[0053] Figure 2 This is a flowchart illustrating the construction process of the visual memory bank in this invention.
[0054] Figure 3 This invention generates a visual result chart for medical reports. Detailed Implementation
[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0056] This invention proposes a medical report generation method based on multimodal pathological detail memory retrieval enhancement, such as... Figure 1 As shown, the method includes the following:
[0057] The medical image to be processed is acquired and input into a trained medical report generation model to obtain a medical report.
[0058] The training process for the medical report generation model includes:
[0059] S1: Obtain the image-text pair dataset and preprocess it to obtain the preprocessed image-text pair dataset.
[0060] An image-text pair dataset was acquired, and the medical images in the dataset underwent standardized preprocessing. First, all images were uniformly downsampled to a resolution of 224×224 pixels to adapt to the input requirements of the pre-trained visual encoder. Simultaneously, the corresponding original medical report texts were cleaned: irrelevant punctuation was removed, medical terminology was standardized, abbreviations were standardized, and paragraphs irrelevant to the research (such as patient privacy information, duplicate descriptions, and non-diagnostic content) were deleted. The cleaned text retained the core clinical manifestations, imaging findings, and diagnostic conclusions, forming a structured medical description for subsequent multimodal alignment and generation tasks.
[0061] S2: Use a pre-trained medical vision model to process the image, obtain local features of the image patch and disease classification results of the image patch; extract text features from the text and obtain the disease classification results of the text.
[0062] The training process for a medical vision model (ViT model) includes:
[0063] The image was divided into 196 non-overlapping image blocks, each with a size of 16×16 pixels;
[0064] The image patches are converted into a labeled representation through a linear projection layer;
[0065] The labeled representation is input into the Transformer model for visual feature extraction, resulting in local features of image patches. and global features of the image Specifically, we define the visual feature extraction process as follows:
[0066]
[0067] in Indicates the number of image patches. Represents the feature dimension.
[0068] The global features of the image are input into the classifier to obtain the image disease classification results. Specifically, the classification head maps the extracted visual features F to the 14 disease categories defined in the MIMIC-CXR dataset.
[0069] The classification loss of the medical vision model is calculated based on the disease classification results of the images. The model parameters are then adjusted based on the classification loss to obtain a trained medical vision model.
[0070] The classification loss is defined as follows:
[0071]
[0072] in, This represents the classification loss of a medical vision model. Indicates the total number of diseases. Indicates the number of samples. Indicates the number of disease states. Indicates the first The sample at the th This disease belongs to a state The true label (one-hot encoding). The first prediction made by the medical vision model The sample at the th This disease belongs to a state The probability is calculated by summing and averaging the negative log probabilities of all tasks, all samples, and all classes, and is used to optimize the model's classification performance on multiple tasks.
[0073] The model was fine-tuned on the training set, with all labels derived from annotated medical reports. During this process, the weights of the visual backbone network remained frozen, and only the linear classification layers were fine-tuned.
[0074] The process involves extracting textual features from the text and obtaining disease classification results. Specifically, BioClinicalBERT, pre-trained on a large-scale biomedical corpus, is used as the text encoder to extract textual features from all radiology reports in the training set. These textual features are then classified according to the disease labels associated with each report to obtain the disease classification results for the text.
[0075] S3: Construct a visual memory bank based on the local features of image patches and the disease classification results of image patches; construct a text memory bank based on the text features and the disease classification results of text.
[0076] Building a visual memory bank:
[0077] To focus on regions most relevant to specific diseases, this invention employs high-resolution class activation mapping to highlight image patches that significantly contribute to the classification task. Compared to gradient class activation mapping, which often generates blurry and coarse heatmaps, high-resolution class activation mapping can generate heatmaps with more precise localization and clearer details, thereby accurately delineating key pathological regions.
[0078] like Figure 2 Specifically, for the input X-ray image I and each of its positive disease labels, a corresponding heatmap is generated. .
[0079] For heatmap Each image block Calculate the average activation value within this block region. :
[0080]
[0081] in, Represents image blocks The average activation value in Represents image blocks Total number of pixels in Representing a heat map The activation value at pixel (x,y).
[0082] Only when Exceeding the preset threshold At that time, image blocks Only local features of the image patch will be preserved and stored in the initial visual memory bank of the corresponding disease label:
[0083]
[0084] in. The visual memory bank representing disease d. Directly using the complete visual memory bank. Training and retrieval can lead to significant efficiency bottlenecks and resource challenges due to the large sample size. To address this issue, computation... The mean feature center of all samples :
[0085]
[0086] in, express The total number of samples in the sample, This represents the local features of an image patch of a single sample in the initial visual memory bank.
[0087] A compact subset is constructed by selecting the k nearest neighbors of the mean feature center. This selection process is applied to all disease categories, extracting a compact subset for each disease category. The final visual memory bank... It is formed by merging all these subsets.
[0088] Building a text memory:
[0089] To utilize the rich semantic information in medical reports, this invention constructs a text memory bank that runs parallel to the visual memory bank; specifically:
[0090] For each disease d, construct its corresponding initial text memory. Formalized, it is defined as the set of all text feature vectors belonging to this type of disease.
[0091] To control the size of the text memory, the same core sample selection strategy as the visual memory is adopted for all disease categories: calculate the mean feature center of all text features in the initial text memory; select the k nearest neighbor samples of the mean feature center to construct a compact subset of the corresponding disease type; merge the compact subsets of all disease types to obtain the final text memory. .
[0092] S4: Enhance the local features of the image patch by retrieving them from the visual memory bank to obtain enhanced local features.
[0093] Using the visual memory bank of pathological perception Local features of image patches Enhance retrieval based on Hopfield network:
[0094]
[0095] in, This represents enhanced local features, which are based on The enhanced representation of the memory bank retrieval. This retrieval enhancement mechanism effectively improves the accuracy of the generated reports and the relevance of the clinical descriptions by incorporating prior pathological knowledge.
[0096] S5: Fill the prompt word template with the disease classification results of the image patch and encode the prompt word template to obtain the prompt template encoding features.
[0097] During report generation, the model first identifies specific diseases and their states based on the classification output of the visual model, and then generates prompts according to pre-defined syntactic templates. For example, for the disease "lung opacity," if the model determines its state as "present," the corresponding prompt template will be filled with: "The lung opacity is present in the image." Similarly, for "pleural effusion," if the model determines it as "uncertain," it generates: "The pleural effusion is uncertain in the image." This templated prompt mechanism transforms the structured classification results into natural language descriptions, providing clear semantic guidance for subsequent text generation.
[0098] The filled-in cue word template is then encoded by BioClinicalBERT to obtain the cue template encoded features. .
[0099] S6: Enhance the retrieval based on the text memory prompt template encoding features to obtain enhanced text prompt features.
[0100] By leveraging a text memory database for pathological perception and employing a Hopfield network for retrieval enhancement, clinical semantic knowledge and typical expression patterns relevant to the current disease state are extracted, thereby strengthening the representational power of cue template encoding features. The definition is as follows:
[0101]
[0102] in, This refers to the enhanced text cue features obtained through Hopfield network retrieval, which are derived from a disease-perceived text memory. The system performs associative retrieval; the Hopfield network injects disease- and state-specific clinical semantics and typical expression patterns into the prompt representation, generating enhanced text prompt features, and providing more informative semantic guidance for subsequent report generation.
[0103] S7: Fuse enhanced local features and enhanced text prompt features to obtain fused features; input the disease classification results of the image patch as prompt word labels and fused features into the decoder for processing to obtain the medical report.
[0104] The fusion of enhanced local features and enhanced text cue features includes: Enhanced text cue features By combining with enhanced local visual features for retrieval The fusion, preferably, is achieved through a feedforward network using the SwishGLU activation function, ultimately forming a joint representation X, i.e., the fused feature.
[0105] The disease classification results of image patches are used as cue word tags and fusion features, which are then input into the decoder for processing, ultimately generating a coherent and professional diagnostic report, such as... Figure 3 As shown. The decoding process is represented as follows:
[0106]
[0107] in, This represents the text token generated at time t. Indicates fusion characteristics, Indicates prompt word marker,
[0108] S8: Calculate the total loss of the model and iteratively adjust the model parameters based on the total loss to obtain the trained medical report generation model.
[0109] The report generation process minimizes the total model loss, i.e., the language modeling loss. Optimize:
[0110]
[0111] in, This represents conditional probability.
[0112] The model is trained iteratively based on the total loss. When the loss function converges or reaches the preset number of iterations, the model parameters are maintained, and the data from the test set is input into the model. After meeting the preset indicators, the trained medical report generation model is obtained.
[0113] By acquiring the medical image to be processed and inputting it into a trained medical report generation model, a medical report corresponding to the medical image can be generated.
[0114] Evaluation of the present invention:
[0115] The present invention was compared with existing comparative methods and models through simulation, and the results are shown in Table 1.
[0116] Table 1. Comparison of metrics between the present invention and the comparative method on the MIMIC-CXR dataset.
[0117] As can be seen from Table 1, the present invention outperforms the comparative method in both CE and NLG indices, demonstrating the effectiveness of the present invention.
[0118] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A medical report generation method based on multimodal pathological detail memory retrieval enhancement, characterized in that, include: The medical image to be processed is acquired and input into a trained medical report generation model to obtain a medical report; The training process for the medical report generation model includes: S1: Obtain the image-text pair dataset and preprocess it to obtain the preprocessed image-text pair dataset; S2: The image is processed using a pre-trained medical vision model to obtain local features of the image patch and the disease classification result of the image patch; text features are extracted from the text, and the disease classification result of the text is obtained. S3: Construct a visual memory bank based on local features of image patches and disease classification results of image patches; construct a text memory bank based on text features and disease classification results of text. S4: Enhance the local features of the image patch by retrieving and strengthening them based on the visual memory bank to obtain enhanced local features; S5: Fill the prompt word template with the disease classification results of the image patch and encode the prompt word template to obtain the prompt template encoding features; S6: Enhance the text prompt features by retrieving and strengthening the coding features of the prompt template based on the text memory; S7: Fuse enhanced local features and enhanced text prompt features to obtain fused features; input the disease classification results of the image patch as prompt word tags and fused features into the decoder for processing to obtain the medical report; S8: Calculate the total loss of the model and iteratively adjust the model parameters based on the total loss to obtain the trained medical report generation model.
2. The medical report generation method based on multimodal pathological detail memory retrieval enhancement according to claim 1, characterized in that, The training process for medical vision models includes: The image was divided into 196 non-overlapping image blocks, each with a size of 16×16 pixels; The image patches are converted into a labeled representation through a linear projection layer; The labeled representation is input into the Transformer model for visual feature extraction, resulting in local features of image patches and global features of the image. The global features of the image are input into the classifier to obtain the disease classification result of the image; The classification loss of the medical vision model is calculated based on the disease classification results of the images. The model parameters are then adjusted based on the classification loss to obtain a trained medical vision model.
3. The medical report generation method based on multimodal pathological detail memory retrieval enhancement according to claim 2, characterized in that, The classification loss of a medical vision model is expressed as: ； in, This represents the classification loss of a medical vision model. Indicates the total number of diseases. Indicates the number of samples. Indicates the number of disease states. Indicates the first The sample at the th This disease belongs to a state The true label, The first prediction made by the medical vision model The sample at the th This disease belongs to a state The probability of.
4. The medical report generation method based on multimodal pathological detail memory retrieval enhancement according to claim 1, characterized in that, The process of building a visual memory bank includes: Generate a heatmap of the image and calculate the average activation value of each image patch in the heatmap; When the average activation value of an image patch is greater than a preset activation threshold, the local features of the image patch are added to the initial visual memory bank corresponding to the disease type of the image patch. Calculate the mean feature center of local features of all image patches in the initial visual memory bank; Select the k nearest neighbor samples of the mean feature center to construct a compact subset corresponding to the disease type; By merging compact subsets of all disease types, the final visual memory bank is obtained.
5. A medical report generation method based on multimodal pathological detail memory retrieval enhancement according to claim 4, characterized in that, The formula for calculating the average activation value of an image patch is: ； in, Represents image blocks The average activation value in Represents image blocks Total number of pixels in Representing a heat map The activation value at pixel (x,y).
6. The medical report generation method based on multimodal pathological detail memory retrieval enhancement according to claim 1, characterized in that, The process of building a text memory includes: For each disease, an initial text memory is constructed based on the corresponding text features; Calculate the mean feature center of all text features in the initial text memory; Select the k nearest neighbor samples of the mean feature center to construct a compact subset corresponding to the disease type; By merging compact subsets of all disease types, the final text memory is obtained.
7. The medical report generation method based on multimodal pathological detail memory retrieval enhancement according to claim 1, characterized in that, The enhancement of local features of image patches based on visual memory is represented as follows: ； in, This indicates enhancement of local features. Representing local features of an image patch, Represents the visual memory bank, This represents the Hopfield network.
8. The medical report generation method based on multimodal pathological detail memory retrieval enhancement according to claim 1, characterized in that, The retrieval enhancement based on the text memory's encoded features of the prompt template is represented as follows: ； in, This indicates enhanced text prompt features. Represents the Hopfield network. This indicates the encoding characteristics of the prompt template.
9. A medical report generation method based on multimodal pathological detail memory retrieval enhancement according to claim 1, characterized in that, The formula for calculating the total loss of the model is: ； in, This represents the total loss of the model. Indicates the report length. This represents the text token generated at time t. Indicates fusion features, Indicates prompt word marker, This represents conditional probability.