Similarity report prompt cross-modality augmented medical report generation method

Abstract

The application discloses a similarity report prompt cross-modal enhanced medical report automatic generation method, an image feature extraction module extracts multi-scale visual features, and a text feature coding module extracts text features; in a training stage, medical images have corresponding reports; in a test stage, for medical images used for testing, a dynamic prompt generation module selects similar cases, and text features of text reports corresponding to the similar cases are weighted and fused to output a dynamic prompt vector, and the dynamic prompt vector is used to replace the corresponding report; a cross-modal feature fusion module dynamically fuses image features and text features to output a fusion feature vector; after the fusion feature vector and a self-recurrence generated unfinished report are jointly input into a CMN, enhanced features are obtained; and a report generation module processes the enhanced features to output a natural language-based diagnosis report text. The application adopts a dynamic prompt mechanism and cross-modal attention, reduces irrelevant noise interference, and reduces the risk of misdiagnosis.

Description

Technical Field

[0001] This invention belongs to the fields of medical image processing and computer vision technology, and specifically relates to a method for automatically generating medical reports with cross-modal enhancement based on similarity report prompts. Background Technology

[0002] Interpreting radiological images (such as chest X-rays) and writing diagnostic reports are essential tasks in clinical practice, often requiring a significant amount of manual work. Therefore, the automatic generation of radiological reports with free text descriptions based on radiological images is widely used to alleviate the burden on radiologists while maintaining the quality of medical care. Recently, substantial progress has been made in research on automated radiology report generation models.

[0003] Most existing studies employ a traditional encoder-decoder architecture, using convolutional neural networks (CNNs) as encoders and recurrent (e.g., LSTM / GRU) or non-recurrent networks (e.g., Transformers) as decoders, following an image description paradigm. While existing methods have made significant progress in medical report generation tasks, they still have significant limitations in deeply exploring the fine-grained (local) semantic correspondence between radiological images and reports. This limitation stems primarily from two aspects: first, current supervised learning methods rely on paired annotations between images and the entire report, lacking region-level supervision signals, making it difficult to effectively learn the correspondence between local image regions and text descriptions; second, existing models mostly focus on overall semantic modeling in their structural design, lacking the ability to explicitly model the local alignment mechanisms between images and text, thus limiting the accurate description of subtle lesions. However, few studies have dedicated themselves to addressing this limitation.

[0004] For example, patent CN114743630B discloses a medical report generation method based on cross-modal contrastive learning. This technical solution relies on ViT to extract image features, which may be insufficient in capturing local details (such as small lesions). Contrastive learning requires a large amount of high-quality paired data, resulting in high data annotation costs. It also does not explicitly handle the temporal relationships between multiple images (such as changes in lesions in consecutive frames). Patent CN116913455A discloses a gastroscopy report generation device. This prior art requires the simultaneous input of multiple gastroscopy images and blood reports. In clinical practice, blood reports may be delayed, and model pre-training and fine-tuning are performed in stages, making the training process complex. It does not design differentiated feature fusion strategies for different lesion types (such as tumors / polyps). Patent CN119274734A discloses an automatic generation method for medical image diagnostic reports based on cross-modal global feature fusion Transformer. The sliding window encoder (SWE) and mesh recurrent network (MRN) in this solution increase computational complexity, lack adaptive learning, and do not fully utilize the hierarchical structure of medical images (such as multi-scale features of organs-lesions-cells). Patent CN119474442A discloses a method, system, device and medium for retrieving similar surgical images for minimally invasive surgery based on deep learning. The feature code mapping may lose detailed information, affecting the retrieval accuracy of small lesions. At the same time, it does not integrate the temporal information of the surgical video and only processes single frame images.

[0005] In summary, existing technologies have several key shortcomings:

[0006] 1. Insufficient coverage of rare cases: Traditional models rely on large-scale labeled data for training, but rare case samples (such as rare tumors and genetic diseases) are scarce in medical scenarios, leading to misjudgments, missed diagnoses, and vague descriptions when the model generates reports, and it cannot make full use of existing features and contextual information.

[0007] 2. Cross-modal data is fragmented. Most methods process images and text separately (such as classifying images first and then filling templates), resulting in weak semantic association between image features and text. The generated report may miss key lesion descriptions and produce contradictions between images and text.

[0008] 3. Insufficient feature extraction capability: Traditional encoders (such as ResNet) may lose minute features in medical images. Shallow networks have low sensitivity to minute lesions (such as early retinal diseases) and may also lead to poor feature redundancy. Deep networks have insufficient feature reusability due to gradient vanishing.

[0009] 4. Limited generation diversity: Template-based methods generate rigid content (such as fixed paragraph combinations), while end-to-end generation models (such as Transformer) may produce unreasonable diversity (such as terminology errors or logical contradictions). Summary of the Invention

[0010] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a method for automatically generating cross-modal enhanced medical reports with similarity report prompts.

[0011] Technical Solution: This invention provides a method for automatically generating cross-modal enhanced medical reports with similarity report prompts. An automatic generation network is constructed and trained, comprising an image feature extraction module, a text feature encoding module, a cross-modal feature fusion module, a dynamic prompt generation module, and a report generation module. Upon receiving the original medical image and the corresponding text report, the automatic generation network performs the following steps:

[0012] Step 1: The image feature extraction module uses DenseNet, a densely connected convolutional network, as the backbone network. For the received medical images, the image feature extraction module extracts multi-scale visual features and outputs high-level semantic image features.

[0013] Step 2: The text feature encoding module is based on the pre-trained multimodal model CLIP. Upon receiving a text report, the text encoder of the text feature encoding module performs word segmentation and feature transformation sequentially to obtain the corresponding text features. During the training phase, the medical images used for training already have corresponding reports. During the testing phase, the medical images used for testing do not have corresponding reports. The dynamic prompt generation module selects similar cases and weights and fuses the text features of the text reports corresponding to the similar cases to output a dynamic prompt vector, which is used to replace the corresponding report.

[0014] Step 3: Input the image features obtained in Step 1 and the text features obtained in Step 2 into the cross-modal feature fusion module. The cross-modal feature fusion module dynamically fuses and aligns the features of the two modalities by calculating attention weights and outputs the fused feature vector.

[0015] Step 4: Input the obtained fused feature vector (i.e., cross-modal features) and the incomplete report generated by autoregression into the CMN to obtain enhanced features;

[0016] Step 5: Input the enhanced features obtained in Step 4 into the report generation module. The report generation module is based on the Transformer decoder and outputs a diagnostic report text based on natural language. Only completed reports will be output. Incomplete reports will be automatically passed to CMN to assist in the generation of enhanced features.

[0017] In this invention, the unfinished report generated by autoregression is generated by CMN. Initially, only a start flag BOS is input. After entering the CMN module, the features are fed into the Transformer decoder based on the image features and the enhanced features of the start flag BOS. The Transformer decoder predicts the next character. In the next round, the start flag BOS and the first predicted character are fed into the CMN module together with the image features for processing, and then fed into the Transformer decoder to predict the next character. This process continues until the EOS flag is predicted, at which point the final text is output.

[0018] Furthermore, the image feature extraction module includes a DenseBlock and a TransitionLayer. The DenseBlock preserves the original input features (such as edges and textures) from being diluted by the deep network, while high-level features (such as semantic information) are directly fused with low-level details to form a multi-scale feature combination. The TransitionLayer suppresses channel expansion of the DenseBlock output (e.g., 256 input channels → 128 output channels) to reduce subsequent computation and memory usage. It gradually expands the receptive field to extract more global semantic information, conforming to the CNN hierarchical abstraction principle (high-resolution details → low-resolution semantics).

[0019] Here, a densely connected convolutional network, DenseNet, is used as the backbone network. It can preserve the detailed features of the early layers and perform better on small abnormalities such as lung nodules and cardiac shadow edges. Each layer can obtain composite information from both shallow and deep layers, which helps to capture the comprehensive semantics of local lesions and overall structure in the image.

[0020] Furthermore, the text feature encoding module is based on the pre-trained multimodal model CLIP, which includes a word embedding layer, a text encoder, and an image encoder. During the training phase, the text encoder extracts the text features of the text report corresponding to the medical image.

[0021] Because text prompts are used during the training phase, performance would degrade if text prompts were not used during the testing phase. Therefore, during the testing phase, the cosine similarity between the current test medical image and the medical images in the training data is calculated first. The K training medical images with the highest similarity are selected, i.e., the Top-K similar samples are selected. Then, the text features of the text reports corresponding to these K medical images are weighted and fused according to similarity to obtain the dynamic prompt vector.

[0022] Furthermore, obtain dynamic cue vectors. During the process, cosine similarity The calculation formula is as follows:

[0023] ;

[0024] In the above formula, It refers to the high-level semantic image features (dimension 1*D) of the test image. It refers to the high-level semantic image features (dimension 1*D) of the i-th medical image in the training data. This refers to the vector dot product operation; It refers to the L2 norm of a vector;

[0025] Obtain each cosine similarity Then, Top-K similar samples are selected, and text features are fused according to similarity weights to obtain dynamic prompt vectors. :

[0026] ;

[0027] ;

[0028] In the above formula, It refers to the text features (dimension 1*D) of the text report corresponding to the j-th similar training medical image; The normalized weight of the j-th text feature (calculated using Softmax) refers to the weight of the j-th text feature. It refers to the similarity score of the j-th similar training medical image.

[0029] To address the challenge of cross-modal alignment between images and text, the cross-modal feature fusion module is based on a CMN memory module, comprising an intermediate memory matrix, cross-modal attention units, and a feature fusion layer. The shared intermediate memory matrix (MemoryMatrix) serves as an explicit cross-modal semantic storage unit, used to store global image-text association knowledge. The cross-modal attention unit achieves dynamic alignment of visual and textual features through a query-key-value (Q-K-V) interaction mechanism, where the query vector Q comes from the input modality (image or text), and the key vector K and value vector V come from the memory matrix. The feature fusion layer fuses the residual-enhanced cross-modal features with the original modal features (concatenation + linear transformation), achieving deep interaction and information preservation of multi-source heterogeneous features. The calculation formula is as follows:

[0030] ;

[0031] In the above formula, Q (Query) refers to the query matrix, which is derived from image features; K (Key) and V (Value) refer to the key matrix and value matrix, respectively, both derived from text features. This refers to the dimension of the feature vector (e.g.) =512); It refers to an exponential function normalized by row.

[0032] A memory matrix is ​​used to store cross-modal information, and memory queries and responses are performed on image and text features using the memory matrix. During the memory query process, the most relevant memory vectors are extracted from the matrix, and their weights are calculated based on the input visual and text features. Then, the response is generated by weighting the memory vectors from the query.

[0033] Furthermore, the calculation steps for the intermediate memory matrix are as follows:

[0034] Step (1): First, perform a memory lookup by inputting image features. Text sequence and memory matrix ;

[0035] Will and Transform the vectors in the memory matrix into a unified space:

[0036] , , ;

[0037] in A learnable linear projection matrix is ​​used to map visual and textual features to a unified query space. Represents the memory key vector. Represents the visual query vector. Represents a text query vector. It refers to the shared intermediate memory matrix. Represents image region features, This refers to text word embedding;

[0038] Then calculate the similarity (inner product) between each query (image or text) and the memory vector:

[0039] ;

[0040] ;

[0041] It refers to visual similarity. This refers to text similarity. This refers to the visual query vector. This refers to the text query vector. This refers to the memory key vector, where d represents the feature dimension; the top K most relevant memory matrices are selected based on distance (Top-K), and finally, the similarity is softmax normalized and used as the attention weight. , ;

[0042] Step (2): The memory response first performs a linear transformation on the selected memory vector. The transformation formula is: ;

[0043] Then, the memory vector is weighted and summed using the weights obtained earlier, as shown in the following formula:

[0044] , ;

[0045] Finally, the response vector and The data are fed into the encoder and decoder respectively, and used in conjunction with the original features to generate a report.

[0046] Furthermore, the report generation module is based on the Transformer decoder. Specifically, the generated text sequence is input into the CMN memory module. The CMN module interacts with image features through the intermediate memory matrix to generate clinical summary features. These clinical summary features are then input into the Transformer decoder. In the cross-attention layer, image memory and clinical summary features are aligned. Relevant medical terms are retrieved through the intermediate memory matrix. The probability distribution of the next word is predicted based on the comprehensive context. The new word is added to the generated sequence, and the hidden state is updated to retain historical information. This process is repeated with the CMN memory module. Then, image memory is viewed through cross-modal attention (cross-attention), aligning image features and text features. Next, relevant medical terms are retrieved from the intermediate memory matrix, and the probability distribution of the next word is predicted based on the comprehensive context. Finally, the new word is added to the generated sequence, and the hidden state is updated to retain the generation history.

[0047] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0048] (1) Security: Existing technologies rely on static templates or single-modal data, which may lead to the leakage of sensitive information or incorrect diagnostic suggestions in the generated reports due to training data bias. In contrast, this invention adopts a dynamic prompting mechanism, extracting relevant text features only from similar cases, avoiding direct exposure of the original training data (such as patient privacy text), and also uses cross-modal attention to screen highly relevant features, reducing irrelevant noise interference and lowering the risk of misdiagnosis.

[0049] (2) Recognition (accuracy): Existing technologies struggle to capture the complex relationships between images and text, leading to a disconnect between the generated report content and the image semantics. In contrast, this invention achieves cross-modal dynamic fusion by dynamically aligning image regions and text keywords through an attention mechanism. During the fusion process, textual features from similar cases are introduced to enhance the clinical relevance of the generated report.

[0050] (3) Simplicity (ease of use): Existing technologies require manual design of templates or complex rules, resulting in high deployment and maintenance costs. In contrast, this invention only requires input of images and corresponding report text, eliminating the need for manual annotation of region-text alignment data.

[0051] (4) Universality (generalization ability): Existing models show a sharp decline in performance on new devices or rare disease images. However, the dynamic prompt generation module of this invention can generate reasonable reports even if there are few rare cases in the training set by searching for similar cases.

[0052] (5) Controllability: Reports generated by existing technologies are uncontrollable and may output descriptions that do not conform to clinical guidelines. In contrast, the present invention allows manual adjustment of the weight of similar cases (such as prioritizing reference to specific authoritative cases), and a medical terminology verification layer can be embedded in the decoding stage to avoid non-standard expressions.

[0053] (6) Scalability: Existing technologies are difficult to adapt to new modalities (such as pathological slides) or new report formats, while the image / text encoder in this invention can be replaced with any new model (such as switching ViT to an image encoder). Even if new case data is added, there is no need to retrain the entire model. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the overall structure of the present invention;

[0055] Figure 2 This is a schematic diagram of the overall process of the present invention;

[0056] Figure 3 This is a flowchart of the CMN module in this invention;

[0057] Figure 4 This is a schematic diagram of a medical image in the embodiment. Detailed Implementation

[0058] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0059] like Figure 1 and Figure 2 As shown, the similarity report prompt cross-modal enhanced automatic medical report generation method of the present invention constructs and trains an automatic generation network, which includes an image feature extraction module, a text feature encoding module, a cross-modal feature fusion module, a dynamic prompt generation module, and a report generation module; after receiving the original medical image and the corresponding text report, the automatic generation network performs the following steps:

[0060] Step 1: The image feature extraction module uses DenseNet, a densely connected convolutional network, as the backbone network. For the received medical images, the image feature extraction module extracts multi-scale visual features and outputs high-level semantic image features. The dense connection structure of DenseNet preserves the detailed features of early layers (such as nodule edges and microcalcifications), which is superior to the global block processing of ViT (CN114743630B). For example, for 2mm micronodules, the feature extraction error of DenseNet is reduced by 18% compared with ResNet.

[0061] Step 2: The text feature encoding module is based on the pre-trained multimodal model CLIP. Upon receiving the text report, the text encoder of the text feature encoding module performs word segmentation and feature transformation in sequence to obtain the corresponding text features.

[0062] During the training phase, corresponding reports are already available for the medical images used for training;

[0063] During the testing phase, a dynamic prompt generation module is used to select similar cases for the medical images used in the test, and the text features of the text reports corresponding to the similar cases are weighted and fused to output a dynamic prompt vector. The dynamic prompt vector is then used to replace the corresponding report.

[0064] Step 3: Input the image features obtained in Step 1 and the text features obtained in Step 2 into the cross-modal feature fusion module. The cross-modal feature fusion module dynamically fuses and aligns the features of the two modalities by calculating attention weights and outputs the fused feature vector.

[0065] Step 4: Input the fused feature vector obtained in Step 3 and the incomplete report generated by autoregression into the CMN to obtain the enhanced features;

[0066] Step 5: Input the enhanced features obtained in Step 4 into the report generation module. The report generation module is based on the Transformer decoder and outputs a diagnostic report text based on natural language.

[0067] The image feature extraction module of this embodiment includes a dense block (DenseBlock) and a transition layer (TransitionLayer). The dense block retains the original input features and forms a multi-scale feature combination. The transition layer (TransitionLayer) suppresses channel dilation of the output of the dense block (DenseBlock).

[0068] The text feature encoding module in this embodiment is based on the pre-trained multimodal model CLIP, which includes a word embedding layer, a text encoder, and an image encoder. During the training phase, the text encoder extracts the text features of the text reports corresponding to the medical images. During the testing phase, the cosine similarity between the current test medical image and the medical images in the training data is calculated first. The K training medical images with high similarity are selected, i.e., the Top-K similar samples are selected. Then, the text features of the text reports corresponding to these K medical images are weighted and fused according to similarity to obtain the dynamic prompt vector.

[0069] Obtain dynamic hint vectors During the process, cosine similarity The calculation formula is as follows:

[0070] ;

[0071] In the above formula, It refers to the high-level semantic image features (dimension 1*D) of the test image. It refers to the high-level semantic image features (dimension 1*D) of the i-th medical image in the training data. This refers to the vector dot product operation; It refers to the L2 norm of a vector;

[0072] Obtain each cosine similarity Then, Top-K similar samples are selected, and text features are fused according to similarity weights to obtain dynamic prompt vectors. :

[0073] ;

[0074] ;

[0075] In the above formula, It refers to the text features of the text report corresponding to the j-th similar training medical image; The normalized weight of the j-th text feature. It refers to the similarity score of the j-th similar training medical image.

[0076] like Figure 1 and Figure 3As shown, the cross-modal feature fusion module in this embodiment is based on the CMN memory module, including an intermediate memory matrix, a cross-modal attention unit, and a feature fusion layer. The shared intermediate memory matrix serves as an explicit cross-modal semantic storage unit, used to store global image-text association knowledge. The cross-modal attention unit achieves dynamic alignment of visual and textual features through a query-key-value (Q-K-V) interaction mechanism, where the query vector Q comes from the input modality (image or text), and the key vector K and value vector V come from the memory matrix. The feature fusion layer fuses the residual-enhanced cross-modal features with the original modal features (concatenation + linear transformation) to achieve deep interaction and information preservation of multi-source heterogeneous features. The calculation formula is as follows:

[0077] ;

[0078] In the above formula, Q refers to the query matrix, and K and V refer to the key matrix and value matrix, respectively. This refers to the dimension of the feature vector. It refers to an exponential function normalized by row.

[0079] The report generation module in this embodiment is based on the Transformer decoder and includes a self-attention layer and a cross-attention layer. The specific working method is as follows: First, the currently generated text and clinical summary features are input into the CMN memory module for interaction. Then, the image memory is viewed through cross-modal attention (i.e., cross-attention in the figure) to align the image features and text features. Next, relevant medical terms are retrieved from the intermediate memory matrix. The probability distribution of the next word is predicted based on the comprehensive context. Finally, the new word is added to the generation sequence, and the hidden state is updated to retain the generation history.

[0080] Cross-attention mechanisms enable information complementarity and fusion, as well as feature space alignment. Text features leverage image features (typically concrete, visual, and spatial) to enhance their understanding of visual scenes, while image features utilize text features to obtain higher-level semantic guidance and contextual information. When generating the next word, the cross-attention mechanism allows the model to dynamically focus on the region in the image most relevant to that word. This greatly helps the model generate more accurate and relevant descriptions. Text and image features typically come from different pre-trained models (such as BERT and ResNet) and exist in different feature spaces. Through the cross-attention mechanism, by calculating attention weights and weighted summation, it implicitly learns how to align and interact features from the two modalities within a shared semantic space. The weighted context vector can be viewed as a fused representation within a shared semantic space.

[0081] like Figure 3 As shown, the calculation steps for the intermediate memory matrix are as follows:

[0082] Step (1): First, perform a memory lookup by inputting image features. Text sequence And memory matrix M= ;Will and Transform the vectors in the memory matrix into a unified space:

[0083] , , ;

[0084] in A learnable linear projection matrix is ​​used to map visual and textual features to a unified query space. Represents the memory key vector. Represents the visual query vector. Represents a text query vector. It refers to the shared intermediate memory matrix. Represents image region features, This refers to text word embedding;

[0085] Then calculate the similarity (inner product) between each query (image or text) and the memory vector:

[0086] ;

[0087] ;

[0088] It refers to visual similarity. This refers to text similarity. This refers to the visual query vector. This refers to the text query vector. This refers to the memory key vector, where d represents the feature dimension; the top K most relevant memory matrices are selected based on distance (Top-K), and finally, the similarity is softmax normalized and used as the attention weight. , ;

[0089] Step (2): The memory response first performs a linear transformation on the selected memory vector. The transformation formula is: ;

[0090] Then, the memory vector is weighted and summed using the weights obtained earlier, as shown in the following formula:

[0091] , ;

[0092] Finally, the response vector and The data are fed into the encoder and decoder respectively, and used in conjunction with the original features to generate a report.

[0093] To verify the feasibility of this invention, this invention... Figure 4 The TCM images (sourced from the iu-xray dataset) were used to automatically generate a medical report. After processing, the final report was: "The cardiac silhouette and mediastinum size are within normal limits. There is no pulmonary edema. There is no focal consolidation. There are no XXXX of a pleural effusion. There is no evidence of pneumothorax."

[0094] To further verify the technical effect of the present invention, the technical solution of the present invention was compared with the existing model in an experiment, and the results are shown in Table 1.

[0095] Table 1

[0096]

[0097] As shown by the experimental data in Table 1, the present invention has a low sensitive information leakage rate, reduces network model training time, significantly improves BLEU-4 of images, significantly improves the accuracy of identification and description of rare diseases (such as pneumothorax and pericardial effusion), and the BLEU-1 index decreases by no more than 1.3%, maintaining good consistency.

[0098] In summary, this invention employs a dynamic prompt generation module to generate dynamic auxiliary text prompts. During the training phase, image features and corresponding text reports are saved. During the testing phase, Top-K similar training images are retrieved using cosine similarity, and their reports are fused to generate prompt text. For example, for an image of "atypical pulmonary nodules," descriptions such as "spiculation" and "lobulation" from similar cases are automatically fused to generate accurate prompts. The cross-modal memory network (CMN) of this invention achieves fine-grained alignment between images and text. By explicitly storing image-text associations (such as "cardiac shadow edge → myocardial hypertrophy") through a learnable memory matrix, it replaces the traditional co-attention mechanism and can capture more complex cross-modal patterns (such as the nonlinear association between lesion morphology and descriptive words).

Claims

1. A method for automatically generating cross-modal enhanced medical reports with similarity report prompts, characterized in that, An automatic generation network is constructed and trained, comprising an image feature extraction module, a text feature encoding module, a cross-modal feature fusion module, a dynamic prompt generation module, and a report generation module. Upon receiving the original medical image and the corresponding text report, the automatic generation network performs the following steps: Step 1: The image feature extraction module uses DenseNet, a densely connected convolutional network, as the backbone network. For the received medical images, the image feature extraction module extracts multi-scale visual features and outputs high-level semantic image features. Step 2: The text feature encoding module is based on the pre-trained multimodal model CLIP. Upon receiving the text report, the text encoder of the text feature encoding module performs word segmentation and feature transformation in sequence to obtain the corresponding text features. During the training phase, corresponding reports are already available for the medical images used for training; During the testing phase, a dynamic prompt generation module is used to select similar cases for the medical images used in the test, and the text features of the text reports corresponding to the similar cases are weighted and fused to output a dynamic prompt vector. The dynamic prompt vector is then used to replace the corresponding report. Step 3: The image features obtained in Step 1 and the text features obtained in Step 2 are fed into the cross-modal feature fusion module. The cross-modal feature fusion module dynamically fuses and aligns the features of the two modalities by calculating attention weights, and outputs a fused feature vector. The cross-modal feature fusion module is based on the CMN memory module, which includes an intermediate memory matrix, cross-modal attention units, and a feature fusion layer. The calculation steps of the intermediate memory matrix are as follows: Step (1): First, perform a memory lookup by inputting image features. Text sequence and memory matrix ;Will and Transform the vectors in the memory matrix into a unified space: ， ， ； in A learnable linear projection matrix is ​​used to map visual and textual features to a unified query space. Represents the memory key vector. Represents the visual query vector. Represents a text query vector. It refers to the shared intermediate memory matrix. Represents image region features, This refers to text word embedding; Then calculate the similarity between each query and the memory vector: ； It refers to visual similarity. This refers to text similarity. This refers to the visual query vector. This refers to the text query vector. This refers to the memory key vector, where d represents the feature dimension; the top K most relevant memory matrices are selected based on distance, and finally, the similarity is softmax normalized and used as the attention weights. , ; Step (2): The memory response first performs a linear transformation on the selected memory vector. The transformation formula is: ； Then, the memory vector is weighted and summed using the weights obtained earlier, as shown in the following formula: ， ； Finally, the response vector and The data are fed into the encoder and decoder respectively, and used in conjunction with the original features to generate a report. Step 4: Input the obtained fused feature vector and the incomplete report generated by autoregression into the CMN to obtain enhanced features; Step 5: Input the enhanced features obtained in Step 4 into the report generation module. The report generation module is based on the Transformer decoder and outputs a diagnostic report text based on natural language.

2. The method for automatically generating cross-modal enhanced medical reports based on similarity report prompts according to claim 1, characterized in that, The image feature extraction module includes a dense block (DenseBlock) and a transition layer (TransitionLayer). The dense block retains the original input features and forms a multi-scale feature combination. The transition layer (TransitionLayer) suppresses channel dilation in the output of the dense block (DenseBlock).

3. The method for automatically generating cross-modal enhanced medical reports based on similarity report prompts according to claim 1, characterized in that, The text feature encoding module is based on the pre-trained multimodal model CLIP, which includes a word embedding layer, a text encoder, and an image encoder; During the training phase, the text encoder extracts text features from the text reports corresponding to the medical images; During the testing phase, the cosine similarity between the current test medical image and the medical images in the training data is first calculated. K training medical images with high similarity are selected, i.e., Top-K similar samples are selected. Then, the text features of the text reports corresponding to these K medical images are weighted and fused according to similarity to obtain dynamic prompt vectors.

4. The method for automatically generating cross-modal enhanced medical reports based on similarity report prompts according to claim 3, characterized in that, Obtain dynamic hint vectors During the process, cosine similarity The calculation formula is as follows: ； In the above formula, It refers to the high-level semantic image features of the test image. It refers to the high-level semantic image features of the i-th medical image in the training data; This refers to the vector dot product operation; It refers to the L2 norm of a vector; Obtain each cosine similarity Then, Top-K similar samples are selected, and text features are fused according to similarity weights to obtain dynamic prompt vectors. : ； In the above formula, It refers to the text features of the text report corresponding to the j-th similar training medical image; The normalized weight of the j-th text feature. It refers to the similarity score of the j-th similar training medical image.

5. The method for automatically generating cross-modal enhanced medical reports based on similarity report prompts according to claim 1, characterized in that, The cross-modal attention unit enables dynamic semantic alignment and information filtering between modalities; The feature concatenation layer achieves deep fusion and information preservation of multi-source heterogeneous features; the intermediate memory matrix stores cross-modal correlation information, maps image features and text features, and is calculated using the following formula: ； In the above formula, Q refers to the query matrix, and K and V refer to the key matrix and value matrix, respectively. This refers to the dimension of the feature vector, and Softmax() is an exponential function that normalizes by row.

6. The method for automatically generating cross-modal enhanced medical reports based on similarity report prompts according to claim 1, characterized in that, The report generation module is based on a Transformer decoder. Its specific operation is as follows: the currently generated text sequence is input into the CMN memory module. The CMN module interacts with image features through an intermediate memory matrix to generate enhanced features. These enhanced features are then input into the Transformer decoder. Image memory and clinical summary features are aligned in a cross-attention layer. Relevant medical terms are retrieved through the intermediate memory matrix. The probability distribution of the next word is predicted based on the comprehensive context. The new word is added to the generation sequence, and the hidden state is updated to retain historical information. This process is repeated with the CMN memory module. Then, image memory is viewed through cross-attention, aligning image features and text features. Next, relevant medical terms are retrieved from the intermediate memory matrix, and the probability distribution of the next word is predicted based on the comprehensive context. Finally, the new word is added to the generation sequence, and the hidden state is updated to retain the generation history.

Images