Radiology report generation method and system based on multi-modal attribute retrieval enhanced guidance, and storage medium

Abstract

The application discloses a kind of radiology report generation methods, systems and storage medium based on multi-modal attribute retrieval enhancement guide, the method includes the following steps: radiology image is collected and input visual encoder is extracted, obtains global visual feature vector;The similarity of visual feature in pre-defined medical knowledge base is calculated, and relevant text embedding expression is retrieved;Text embedding expression is input attribute generation module, and refined guide attribute feature is formed;Hybrid cross-modal interaction module is constructed, and enhanced multi-modal context representation is generated, input decoder, in combination with the text sequence that has been generated, the probability distribution of next word element is recursively predicted until complete radiology diagnosis report is generated.Adopt the technical solution, introduce cross-modal retrieval mechanism and explicit attribute guide, solve the technical problems that radiology image pathological feature is sparse and visual and text semantic alignment is not accurate in prior art, and intelligent generation radiology report.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of medical image processing, and relates to a method, system and storage medium for generating radiology reports based on enhanced multimodal attribute retrieval guidance. Background Technology

[0002] Radiological imaging diagnosis (such as chest X-ray, CT, MRI, etc.) is a crucial part of the clinical medical system, and radiological reports are the core documents that record imaging findings and provide diagnostic recommendations.

[0003] With the explosive growth of medical imaging data, radiologists face immense pressure in interpreting images and a heavy workload of paperwork. Therefore, automated radiology report generation (RGG) technology has emerged, aiming to use artificial intelligence algorithms to automatically extract features from medical images and generate structured descriptive text. This not only significantly improves clinical efficiency but also reduces the rate of missed or misdiagnosed cases due to fatigue, possessing significant social and economic value.

[0004] Early automated report generation methods largely borrowed from general image captioning techniques, primarily based on an encoder-decoder architecture. They typically used deep convolutional neural networks (CNNs) or visual transformers (ViTs) as encoders to extract image features, and then combined recurrent neural networks (RNNs) or transformers as decoders for text prediction.

[0005] However, there is a fundamental difference between generating reports in the medical field and describing landscapes in general fields. Existing technical solutions still face the following deep-seated challenges in actual clinical application:

[0006] First, there is the severe sparsity of pathological features and the limitations of visual perception. In radiological images, abnormal signs reflecting the condition (such as small nodules, minor pleural effusions, and blurred infiltrates) often occupy only a tiny pixel space and are visually highly similar to normal anatomical structures. Traditional models tend to extract global visual features, which can lead to "oversmoothing" during training—that is, overemphasizing the dominant normal anatomical areas while ignoring subtle lesion details with diagnostic value. This sparsity of pathological features often results in reports that tend to describe "no abnormalities," lacking the ability to capture key pathological information and thus having insufficient value for clinical decision-making.

[0007] Second, cross-modal semantic alignment (CMA) presents an extremely high challenge. Radiology report generation is essentially a mapping process from low-level visual pixels to high-level medical semantics. A significant "modal gap" exists between visual patterns in medical images and the technical terminology in reports. Existing end-to-end mapping models often lack explicit intermediate guidance, relying solely on statistical correlations of large-scale data, making it difficult to accurately learn the correspondence between specific visual features and specific medical attributes (such as "cardiac enlargement" or "mediastinal widening"). Furthermore, due to the highly specialized and logically rigorous nature of medical language, simple visual feature fusion often leads to logical fallacies or false positives for key attributes in the generated text.

[0008] Third, there is insufficient utilization of external prior knowledge and a lack of retrieval mechanisms. Radiologists do not rely solely on visual observation when writing reports; they combine long-term clinical experience with a wealth of professional knowledge. Most existing models are closed-loop, meaning they only rely on the currently input images for reasoning, lacking effective retrieval and reference to historical high-quality report databases. Although some studies have attempted to introduce retrieval mechanisms, they often remain at the level of simple template matching, failing to achieve in-depth refinement and attribute calibration of images and text within a shared semantic space. This lack of utilization of prior knowledge limits the model's ability to generate diagnostic conclusions with professional standards and breadth of knowledge.

[0009] Fourth, the simplistic nature of the interaction fusion mechanism. Existing technologies often employ simple concatenation or linear addition when fusing visual features with textual guidance, failing to consider the semantic differences between different modalities. The lack of a dynamically adjustable, learnable bias-based hybrid interaction mechanism makes it difficult for the model to spatially refocus on specific pathological attributes, resulting in reports that are unsatisfactory in both accuracy and granularity. Summary of the Invention

[0010] The purpose of this invention is to address the aforementioned problems in existing technologies by proposing a radiology report generation method, system, and storage medium based on enhanced multimodal attribute retrieval guidance.

[0011] To achieve the above objectives, the basic solution of this invention is: a radiology report generation method based on multimodal attribute retrieval enhancement guidance, comprising the following steps:

[0012] S1. Establish a visual encoder, acquire radiological images and input them into the visual encoder for feature extraction to obtain a global visual feature vector. ;

[0013] S2 utilizes the image-to-text retrieval (I2TR) mechanism to calculate the similarity of visual features in a predefined external medical knowledge base and retrieve relevant text embeddings.

[0014] S3 embeds the text into the Expression Input Attribute Generation (AGM) module, predicts the probability distribution of clinical attributes present in the image through a multi-label classification layer, and maps it to the semantic space to form refined guiding attribute features. ;

[0015] S4, construct a hybrid cross-modal interaction module, utilizing a cross-attention mechanism with learnable biases to integrate global visual features. With Refined Guiding Attributes Perform depth alignment to generate enhanced multimodal context representations. ;

[0016] S5 represents the multimodal context. The input decoder recursively predicts the probability distribution of the next word until a complete radiology diagnostic report is generated.

[0017] The working principle and beneficial effects of this basic solution are as follows: This technical solution explicitly extracts prior clinical attributes from the original images through an attribute generation module, and introduces an image-to-text retrieval mechanism to refine attribute accuracy from the knowledge base using a cross-modal shared space. A hybrid cross-modal interaction module with learnable biases is constructed to achieve deep alignment and feature fusion of visual features and text-guided attributes. This not only significantly enhances the model's ability to capture subtle pathological signs, but also greatly improves clinical effectiveness indicators and the quality of natural language generation, providing efficient and accurate technical support for the automated and intelligent generation of radiology reports.

[0018] The attribute generation module of this invention uses text features to accurately generate attributes, guiding the detailed report generation process and enabling the model to better understand the relationship between visual and text features. The image-to-text retrieval mechanism allows for the retrieval of relevant text information based on specific image features. The hybrid cross-modal interaction module enhances the interactive features between images and attributes, ensuring that the model effectively integrates visual information with textual attributes in a formatted manner, resulting in a more coherent and detailed radiology report.

[0019] Furthermore, radiological images are acquired and input into a visual encoder for feature extraction to obtain a global visual feature vector. The method is as follows:

[0020] The acquired raw radiological images are subjected to standardized preprocessing, including size adjustment and normalization;

[0021] Deep neural networks (ResNet-101 or Swin-Transformer) are used as visual encoders to extract multi-dimensional features from standardized preprocessed radiological images, obtain global contour information of radiological images, segment radiological images into multiple local regions, and extract local feature matrices containing fine textures and pathological information.

[0022] By using global average pooling, the local feature matrix (dispersed spatial features) is aggregated into a global visual feature vector representing the overall semantics of the image, as follows:

[0023] ,

[0024] Where 𝐿𝑁 represents layer normalization, , is the trainable parameter, and is the input image.

[0025] By preserving local texture information and global semantic information of images, the model's ability to perceive subtle lesions is enhanced, providing a more discriminative visual representation basis for subsequent cross-modal interactions.

[0026] Furthermore, step S2 utilizes the image-to-text retrieval (I2TR) mechanism to calculate the similarity of visual features in a predefined medical knowledge base, retrieving relevant text embeddings as follows:

[0027] An external medical knowledge base is pre-built, which stores several historical images and their corresponding authoritative diagnostic reports;

[0028] Using a pre-trained medical-specific text encoder, text from an external medical knowledge base is transformed into high-dimensional vectors;

[0029] A dedicated cross-modal projection layer is constructed to map global visual feature vectors and text features from an external medical knowledge base into the same "shared semantic space," in which images and text no longer have modal distinctions.

[0030] Calculate the global visual feature vector Embedded vectors of candidate radiology text reports from an external medical knowledge base cosine similarity :

[0031] ,

[0032] Cosine similarity retrieval is used to select the K reports most relevant to radiological images, achieving semantic alignment between images and text. The retrieved reports are then extracted using a pre-trained medical text encoder (such as CLIP text end) to generate text features for attribute detection. .

[0033] By introducing an external medical knowledge base, semantic alignment between visual features and text priors was achieved, significantly improving the model's prior cognitive ability for complex pathologies and making up for the limitations of traditional closed models.

[0034] Furthermore, in step S3, the text is embedded into the Expression Input Attribute Generation (AGM) module, which predicts the probability distribution of clinical attributes present in the image through a multi-label classification layer and maps it to the semantic space to form refined guiding attribute features. The specific method is as follows:

[0035] The most frequently retrieved report from external medical knowledge bases Each report contains a non-stop word as a potential attribute label, and for each report, there is a binary label vector C = { ,…, The actual tag is constructed if the attribute exists. = 1, where, For input text features ,Will Average pooling obtains text semantic features ;

[0036] Text semantic features High-order semantics are captured through a global semantic path and two layers of linear expansion and contraction structures. :

[0037] ,

[0038] in, , It is a linear layer. , It is a weight matrix. It is the GELU activation function;

[0039] Text semantic features Low-rank and fine-grained features are obtained through a local interaction path. :

[0040] ,

[0041] in, , It is a weight matrix;

[0042] Introducing a lightweight gating unit The gating unit dynamically adjusts the contribution of each path based on the input context and input features. Real-time calculation of scalar weights :

[0043] ,

[0044] Fusion features Obtained by weighted summation of the two-path outputs:

[0045] ,

[0046] in, These are the weight parameters, and ⊙ is the Hadamard product;

[0047] exist At this time, calibrated residual connections are used to bridge the dimensionality gap, projecting the original input 𝑥 onto the hidden dimension. And a learnable scaling factor 𝛾 is introduced:

[0048] ,

[0049] ,

[0050] in, These are weight parameters. Let represent the learnable bias vector, where is a trainable parameter initialized to 0.1; This refers to the features after bridging the dimensionality gap using calibrated residual connections. The final features are obtained by introducing a learnable scaling factor 𝛾; Text features Dimension It is the hidden layer feature dimension;

[0051] Features after fusion Finally, the attribute prediction head generates attribute prediction probabilities. Finally go through The activation function generates the final predicted attributes. :

[0052] ,

[0053] ,

[0054] in, , These are weight parameters. It is a linear layer;

[0055] Finally, the top R attributes with the highest probabilities are extracted as refined guiding attribute features. .

[0056] By coordinating global semantic paths and local interaction paths, and combining them with learnable gating mechanisms, we have achieved simultaneous modeling of macroscopic lesions and microscopic lesions, which has significantly improved the accuracy of attribute prediction and the ability to express guided features.

[0057] Furthermore, in step S4, a hybrid cross-modal interaction module is constructed, utilizing a cross-attention mechanism with learnable biases to integrate global visual features. With Refined Guiding Attributes Perform depth alignment to generate enhanced multimodal context representations. ,for:

[0058] ,

[0059] Among them, 𝑄 is the refined guide attribute output. , 𝐾 and 𝑉 originate from global medical image features , K and V are used in the formula for calculation; B is the bias matrix that is automatically optimized during training and is used to dynamically adjust the spatial and logical mapping relationship between local visual features and semantic attribute features. This is the scaling factor.

[0060] By using attribute features as queries and visual features as keys, and introducing a trainable bias matrix, a precise spatial mapping between visual regions and medical attributes is achieved, effectively solving the problems of "semantic misalignment" and "hallucination generation" in traditional models.

[0061] Furthermore, it also includes employing a multi-task loss function. The steps for joint optimization are as follows:

[0062] ,

[0063] in, To report the generated cross-entropy loss, Multi-label binary cross-entropy loss for attribute prediction:

[0064] ,

[0065] ,

[0066] in, It is a one-hot vector representing the label. The position in vocabulary V is 1, and the other positions are 0; It is a real label (0 or 1). It represents the probability of the predicted attribute existing; T is the length of the generated medical report, and t is the word at the t-th position in the medical report. It predicts the probability of the t-th word. is the number of Ground-Truth attribute labels in the input radiology report, and i is the predicted i-th attribute.

[0067] By simultaneously optimizing language modeling and attribute classification tasks, the model's ability to perceive key pathological attributes was enhanced, improving the performance of generated reports in terms of clinical consistency and diagnostic accuracy.

[0068] The present invention also provides a radiology report generation system based on the method described in the present invention, comprising a data acquisition unit, a visual encoder, an attribute prediction and retrieval unit, a hybrid cross-modal interaction module, and a Transformer decoder;

[0069] The data acquisition unit is used to acquire radiological images and transmit them to the visual encoder for feature extraction to obtain a global visual feature vector. ;

[0070] The attribute prediction and retrieval unit includes an I2TR module and an AGM module connected in sequence. The I2TR module uses a cross-modal projection layer to map global visual features to a semantic space shared with the medical knowledge base, retrieves the Top-K related reports through cosine similarity, and generates an initial text feature vector through a medical-specific text encoder.

[0071] The AGM module employs a dual-path heterogeneous processing architecture. It captures macroscopic anatomical features through a global semantic path composed of two layers of linear expansion-contraction structures, while simultaneously utilizing a local interaction path composed of low-rank transformations to mine semantic relationships between fine-grained pathological attributes. Context-aware gating units are used to dynamically weight and fuse the dual-path features. The attribute prediction head then generates the probability distribution of clinical attributes in the image and extracts high-probability attribute mappings to form refined guiding attribute features. ;

[0072] The hybrid cross-modal interaction module constructs a cross-attention mechanism with attribute features as the core navigation, refining and guiding attribute features. As a query vector, global visual features The mapping is done as keys and values, and a learnable spatial bias matrix B is introduced into the attention weight calculation to perform inductive bias correction. The hybrid cross-modal interaction module achieves accurate spatial recalibration and depth alignment of visual regions and medical attributes in multiple subspaces through multi-head parallel computation, generating multimodal context representations with explicit attribute enhancements. ;

[0073] The input of the Transformer decoder is connected to the output of the hybrid cross-modal interaction module. It employs an autoregressive architecture combined with the generated text sequence, utilizing cascaded masked multi-head self-attention and cross-attention layers. The clinical attribute feature region in the model is strongly constrained and guided to recursively predict the probability distribution of the next term until a radiology diagnostic report is generated.

[0074] This system utilizes its modules to obtain complete radiology diagnostic reports, possessing extremely high medical professional logic and accuracy, and can directly assist radiologists in completing their daily diagnostic and treatment work.

[0075] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to perform the method described in the present invention.

[0076] This ensures the portability and industrial application capability of the method, making it easy to implement the report generation method on different computing platforms and giving it broad application prospects. Attached Figure Description

[0077] Figure 1 This is a flowchart of the radiology report generation method based on multimodal attribute retrieval enhancement guided by the present invention;

[0078] Figure 2 This is a flowchart illustrating the process of forming refined guiding attribute features in the radiology report generation method based on multimodal attribute retrieval enhancement of the present invention;

[0079] Figure 3 This is a comparison chart between the report generated by the radiology report generation method based on the multimodal attribute retrieval enhancement guidance of this invention and the actual report. Detailed Implementation

[0080] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0081] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0082] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0083] This invention discloses a radiology report generation method based on multimodal attribute retrieval enhancement guidance. Addressing the core technical barriers of sparse pathological features and cross-modal semantic alignment misalignment in radiology images, it constructs a generation system (RA-MAG) that integrates retrieval enhancement and explicit attribute guidance. Through the attribute generation module (AGM), implicit visual features are transformed into structured clinical priors, and an image-to-text retrieval mechanism (I2TR) is used to introduce an external knowledge base for deep semantic calibration. Furthermore, relying on a hybrid cross-modal interaction module with learnable biases, it achieves accurate mapping between visual pixels and medical terminology, thereby fundamentally improving the diagnostic sensitivity, logical rigor, and professional quality of natural language generation in automated reports in complex clinical environments.

[0084] like Figure 1 As shown, the radiology report generation method based on multimodal attribute retrieval enhancement includes the following steps:

[0085] S1. Establish a visual encoder, acquire radiological images and input them into the visual encoder for feature extraction to obtain a global visual feature vector. ;

[0086] S2 utilizes the image-to-text retrieval (I2TR) mechanism to calculate the similarity of visual features in a predefined external medical knowledge base and retrieve relevant text embeddings.

[0087] S3 embeds the text into the Expression Input Attribute Generation (AGM) module, predicts the probability distribution of clinical attributes present in the image through a multi-label classification layer, and maps it to the semantic space to form refined guiding attribute features. ;

[0088] S4, construct a hybrid cross-modal interaction module, utilizing a cross-attention mechanism with learnable biases to integrate global visual features. With Refined Guiding Attributes Perform depth alignment to generate enhanced multimodal context representations. ;

[0089] S5 represents the multimodal context. The input decoder recursively predicts the probability distribution of the next word until a complete radiology diagnostic report is generated, such as... Figure 3 As shown.

[0090] The enhanced multimodal context representation generated in step S4 The mapping is to a radiological diagnostic report that conforms to clinical standards and is logically rigorous. Its core is built on a 2-layer deep Transformer decoding architecture. Each decoder layer realizes restricted text generation under the guidance of pathological prior through cascaded mask 8-head self-attention layer, cross-attention layer and feedforward neural network layer.

[0091] Process the generated word sequence using a pre-trained medical vocabulary embedding layer and sinusoidal positional encoding. Subsequently, in the cross-attention mechanism, this hidden state is used as the query vector to dynamically focus on... The corresponding clinical attribute feature regions are used to achieve strong constraint guidance, and the probability distribution of the next word is recursively predicted at each time step using a Softmax layer through a bundle search or kernel sampling algorithm until a complete diagnostic text is generated.

[0092] A multimodal feature matrix, incorporating visual details, retrieval priors, and explicit attribute guidance, is input into the Transformer decoder. The decoder employs an autoregressive approach, combining the generated text sequence with the enhanced fused features to generate the final radiology report word by word. To ensure report quality, a beam search algorithm is used for path optimization during the generation process. The final report not only conforms to natural language expression habits but also possesses extremely high medical professional logic and diagnostic accuracy, directly assisting radiologists in their daily diagnostic and treatment work.

[0093] In a preferred embodiment of the present invention, radiological images are acquired and input into a visual encoder for feature extraction to obtain a global visual feature vector. The method is as follows:

[0094] Standardized preprocessing is performed on the acquired raw radiological images (such as frontal and lateral views of chest X-ray films), including size adjustment and normalization.

[0095] High-performance deep neural networks (such as ResNet-101 or visual Transformers, such as SwinTransformer) are used as visual encoders (e.g., using a hierarchical Swin-Transformer or a deep residual network ResNet-101) to extract multi-dimensional (multi-directional) features from standardized preprocessed radiological images, obtaining global contour information of the radiological images. The radiological images are then segmented (automatically segmented using deep learning neural networks such as resNet101 or swin_transformer) into multiple local regions, and local feature matrices (computable matrix vectors) containing subtle textures and pathological information are extracted. The extracted features are then concatenated into a feature sequence to ensure the storage of comprehensive pathological spatial information.

[0096] By using global average pooling, the local feature matrix (dispersed spatial features) is aggregated into a global visual feature vector representing the overall semantics of the image, as follows:

[0097] ,

[0098] Where 𝐿𝑁 represents layer normalization, , Here, is the trainable parameter, and is the input image (i.e., the raw radiographic image). The extracted raw features are processed by linear layer projection and layer normalization to transform them into a visual representation of the model's hidden space dimension. .

[0099] In a preferred embodiment of the present invention, in order to overcome the high sparsity of pathological signs in radiographic images and the limitations of visual features in capturing subtle lesions, step S2 utilizes an image-to-text retrieval (I2TR) mechanism to calculate the similarity of visual features in a predefined medical knowledge base and retrieve relevant text embedding expressions.

[0100] Construct a joint embedding manifold space based on contrastive learning. A pre-trained medical multimodal model (such as the CLIP architecture fine-tuned with Bio-Clinical data) is used as the feature extraction benchmark. The visual encoder is one such model. Responsible for mapping image I to visual embedding Text encoder Then the candidate report texts in the knowledge base will be... Mapping to text embedding .

[0101] An external medical knowledge base is pre-built, which stores several historical images and their corresponding authoritative diagnostic reports;

[0102] Using a pre-trained medical-specific text encoder, text from an external medical knowledge base is transformed into high-dimensional vectors;

[0103] Construct a dedicated cross-modal projection layer (implemented using a neural network clip model) to map global visual feature vectors and text features from an external medical knowledge base into the same "shared semantic space," in which images and text no longer have modal distinctions.

[0104] Calculate the global visual feature vector Embedded vectors of candidate radiology text reports from an external medical knowledge base (using the training set of the dataset as candidate text reports). cosine similarity :

[0105] ,

[0106] The cosine similarity retrieval method selects the K reports most relevant to the radiological images, achieving semantic alignment between images and text (the neural network clip model uses contrastive learning for automatic alignment). The retrieved reports are then extracted using a pre-trained medical text encoder (deep learning neural networks such as ResNet101, Swin Transformer, and CLIP text-side) to generate text features for attribute detection. .

[0107] To mitigate potential noise or bias in individual reports, a non-maximum suppression approach is employed, selecting the top K candidate reports based on similarity. This step simulates the clinical thinking process of a senior radiologist when faced with complex images, referring to the diagnostic logic of similar historical cases. The retrieved text collection... The input is a pre-trained medical-specific Transformer encoder (such as Clinical-BERT). Through a self-attention mechanism, it can identify key tokens in the report describing the location, nature, and severity of lesions. Attention-weighted pooling is then used to fuse these K sets of features, eliminating redundant information and generating highly condensed textual prior features. .

[0108] In a preferred embodiment of the present invention, such as Figure 2As shown, step S3 embeds the text into the Expression Input Attribute Generation (AGM) module, predicts the probability distribution of clinical attributes present in the image through a multi-label classification layer, and maps it to the semantic space to form refined guiding attribute features. The specific method is as follows:

[0109] Based on a large-scale medical imaging report corpus, through rigorous statistical analysis and clinical value assessment, hundreds of core medical terms covering anatomical abnormalities and pathophysiological changes were selected, thereby constructing a standardized clinical attribute labeling space.

[0110] The most frequently retrieved report from external medical knowledge bases Each report contains a non-stop word as a potential attribute label, and for each report, there is a binary label vector C = { ,…, The actual tag is constructed if the attribute exists. = 1, where, For input text features ,Will Average pooling obtains text semantic features ;

[0111] By simulating the diagnostic logic of radiologists "from global assessment to local detailed scanning," the AGM module achieves structured reorganization of features. It abandons the traditional single-chain feature transformation mode and innovatively constructs a dual-path nonlinear architecture to achieve multi-dimensional coverage of medical semantics. The Global Semantic Path is dedicated to capturing the macroscopic anatomical logic and overall lesion trend in images.

[0112] A two-layer linear expansion-contraction (LEC) structure is employed. Features are mapped to a high-dimensional hidden space using an upscaling matrix to maximize feature separability. Subsequently, a non-linear activation function (such as GELU) is used in conjunction with the downscaling matrix for information compression. This allows for the processing of textual semantic features. High-order semantics are captured through a global semantic path and two layers of linear expansion and contraction structures. :

[0113] ,

[0114] in, , It is a linear layer. , It is a weight matrix, which is a learnable two-dimensional vector matrix; It is the GELU activation function; this design can effectively capture pathological signs involving large-scale spatial relationships, such as "enlarged cardiac silhouette" and "increased lung markings". The Local Interaction Path targets sparse and subtle abnormal signs commonly seen in radiology, such as "micronodules" and "local exudation". This path introduces a low-rank transformation mechanism.

[0115] By decomposing the weight matrix into a low-rank matrix product, the model can reduce parameter complexity while forcing features to focus on frequently occurring key local pathological features and suppressing background noise interference. This incorporates textual semantic features. Low-rank and fine-grained features are obtained through a local interaction path. :

[0116] ,

[0117] in, , It is a weight matrix, which is a learnable two-dimensional vector matrix;

[0118] To achieve dynamic optimization of global and local information, a lightweight gating unit is introduced. The gating unit dynamically adjusts the contribution of each path based on the input context and input features. Real-time calculation of scalar weights :

[0119] ,

[0120] Fusion features Obtained by weighted summation of the two-path outputs:

[0121] ,

[0122] in, is the weight parameter, and ⊙ is the Hadamard product; this mechanism ensures that when the image shows a large area of ​​lesion, the model automatically tends to global features, while when the image contains minor lesions, local interactive features dominate.

[0123] When processing features from different encoders, inconsistencies in dimensionality often exist. (Technological challenges)

[0124] exist The process employs calibrated residual connections to bridge the dimensionality gap, and uses a projection matrix to map the original input, projecting the original input 𝑥 onto the hidden dimensions. And a learnable scaling factor 𝛾 is introduced:

[0125] ,

[0126] ,

[0127] in, These are weight parameters. Let represent the learnable bias vector, where is a trainable parameter initialized to 0.1; This refers to the features after bridging the dimensionality gap using calibrated residual connections. The final features are obtained by introducing a learnable scaling factor 𝛾; Text features Dimension It is the hidden layer feature dimension;

[0128] Features after fusion Finally, the attribute prediction head generates attribute prediction probabilities. Finally go through The activation function generates the final predicted attributes. :

[0129] ,

[0130] ,

[0131] in, , These are weight parameters. It is a linear layer, and finally the top R attributes with the highest probabilities are extracted as refined guiding attribute features. ; It not only provides explicit guidance signals for report generation, but also serves as an intermediate result of computer-aided diagnosis (CAD) to feed back to clinicians, significantly improving the "white-box" nature and medical safety of deep learning models.

[0132] Finally, the top R attributes with the highest probabilities are extracted as refined guiding attribute features. .

[0133] The system abandons the single feature extraction mode and instead adopts an innovative dual-path heterogeneous processing framework: the global semantic path uses an alternating structure of multi-layer linear feature expansion and non-linear dimensionality reduction contraction to abstract high-order diagnostic logic flow in the macro dimension, thereby ensuring the model's accurate qualitative assessment of the overall lesion state of the image.

[0134] The local interaction path relies on advanced low-rank mapping technology to deeply explore the complex correlations and semantic symbiotic logic between different pathological attributes, enabling the system to keenly capture fine-grained interactions between tiny lesions, such as accurately identifying the causal co-occurrence between pneumonia infiltrates and exudates.

[0135] To achieve optimal fusion of two types of heterogeneous information, a lightweight intelligent gating unit with context awareness was introduced. This unit can automatically and dynamically adjust the contribution weights of global logic and local details in feature representation based on the real-time input image feature stream, and finally synthesize a highly refined and highly interpretable guided attribute feature stream. This forces the subsequent report generation process to focus on the key signs with the most clinical diagnostic value with unprecedented attention, and completely solves the problems of focus shift and semantic ambiguity that are prone to occur in the description of complex cases in traditional methods.

[0136] In a preferred embodiment of the present invention, in step S4, to overcome the limitations of traditional radiology report generation models, such as inaccurate alignment between visual representations and textual semantics and the easy loss of pathological details, a hybrid cross-modal interaction module is constructed. This module utilizes a cross-attention mechanism with learnable bias to integrate global visual features. With Refined Guiding Attributes Perform depth alignment to generate enhanced multimodal context representations. ,for:

[0137] ,

[0138] Among them, 𝑄 is the refined guide attribute output. , 𝐾 and 𝑉 originate from global medical image features , K and V are used in the formula for calculation; B is the bias matrix that is automatically optimized during training and is used to dynamically adjust the spatial and logical mapping relationship between local visual features and semantic attribute features. This is the scaling factor.

[0139] The core of the hybrid cross-modal interaction module is the cross-attention mechanism, which uses local features of the image as query signals and refined guiding attribute features as keys and values ​​for interaction.

[0140] Using clinical attributes as navigation signals, the HCMI module performs depth space recalibration and semantic alignment on the original visual features. It abandons the traditional blind search model that uses visual features as queries, and instead adopts an attribute-based feature-driven approach. A cross-attention architecture guided by the core.

[0141] The refined guiding attribute characteristics output in step S3 As the query vector Q. Because The explicit clinicopathological logic has been condensed through the AGM module, which, as Q, can actively "retrieve" matching visual evidence in the visual feature space, and extract the global visual features from step S1. The mapping is represented by key K and value V. By calculating the dot product similarity between Q and K, the model can identify which spatial regions in the image directly support the determination of specific attributes (such as "enhanced lung texture" or "blunting costophrenic angle").

[0142] To enhance the model's ability to capture structured features of medical images, HCMI innovatively introduces a learnable bias vector B in the attention score calculation. The introduction of the learnable bias B acts as an "inductive bias," which can adaptively redistribute the spatial weights of visual features, correct feature distortions caused by image noise or shooting angle shifts, and ensure that visual representations are semantically closely aligned with clinical attributes.

[0143] To capture the subtle manifestations of pathological features in different subspaces, HCMI employs multi-head attention for parallel computation. The system will... and The features are projected into eight independent subspaces, and the aforementioned recalibration calculations are performed independently in each subspace. The features output by each head are then concatenated and passed through a linear projection layer for dimensionality restoration. Layer normalization and residual connections are then performed sequentially to generate the final enhanced multimodal context representation. It is no longer an isolated visual matrix, but a structured semantic tensor that has been "filtered" and "enhanced" by clinical attributes.

[0144] In this embodiment, the HCMI module not only performs visual-text alignment at the semantic level, but also implicitly learns the anatomical constraints of medical images (e.g., cardiac attributes typically correspond to the central region of the image) through bias terms. This collaborative mechanism ensures that the subsequent decoder can generate report terms based on highly deterministic... The reasoning process significantly reduces the "hallucination" phenomenon in the generated results, improving the scientific rigor and accuracy of the diagnostic reports.

[0145] In a preferred embodiment of the present invention, to ensure the clinical accuracy and linguistic fluency of the generated results, the radiology report generation method further includes employing a multi-task loss function. The steps for joint optimization are as follows:

[0146] ,

[0147] in, To report the generated cross-entropy loss, Multi-label binary cross-entropy loss for attribute prediction:

[0148] ,

[0149] ,

[0150] in, It is a one-hot vector representing the label. The position in vocabulary V is 1, and the other positions are 0; It is a real label (0 or 1). It represents the probability of the predicted attribute existing; T is the length of the generated medical report, and t is the word at the t-th position in the medical report. It predicts the probability of the t-th word. is the number of Ground-Truth attribute labels in the input radiology report, and i is the predicted i-th attribute.

[0151] This allows the generated reports to not only cover the overall imaging overview, but also to provide accurate and medically logical explanations for subtle local lesions such as trace amounts of pleural effusion, significantly improving the technical effectiveness of automated reports in terms of clinical consistency and diagnostic reliability.

[0152] For example, different methods were applied to IU-Xray and MIMIC-CXR, and the comparison results are shown in Table 1:

[0153] Table 1 Comparison of Implementation Results

[0154]

[0155] Experimental results show that the embodiments of the present invention perform excellently on the IU-Xray and MIMIC-CXR datasets. On the IU-Xray dataset, as shown in Table 1, the present invention achieves BLEU-4 (0.223), METEOR (0.227), and ROUGE-L (0.432) scores, surpassing all competing methods.

[0156] Compared to reinforcement learning-based CMN+RL, the method of this invention improves BLEU-4 by 4.2% and ROUGE-L by 4.8%. Compared to knowledge-driven methods such as PPKED, the gains of this invention are 5.5% (BLEU-4), 3.7% (METEOR), and 5.6% (ROUGE-L).

[0157] On the larger MIMIC-CXR dataset, this invention maintained its competitive advantage, achieving scores of BLEU-4 (0.125) and ROUGE-L (0.296), with BLEU-4 exceeding XPRONET by 2.0% and M2KT by 2.2%. These results demonstrate that the proposed method and system significantly improve semantic consistency and clinical relevance, particularly in capturing subtle lesion descriptions crucial for diagnostic accuracy.

[0158] The present invention also provides a radiology report generation system based on the method described herein, comprising a data acquisition unit, a visual encoder, an attribute prediction and retrieval unit, a hybrid cross-modal interaction module, and a Transformer decoder.

[0159] The data acquisition unit is used to acquire radiological images and transmit them to the visual encoder for feature extraction to obtain a global visual feature vector. .

[0160] The attribute prediction and retrieval unit includes an I2TR module and an AGM module connected in sequence.

[0161] The I2TR module uses a cross-modal projection layer to map global visual features to a semantic space shared with the medical knowledge base, retrieves the top-K related reports through cosine similarity, and generates an initial text feature vector through a medical-specific text encoder.

[0162] The AGM module employs a dual-path heterogeneous processing architecture. It captures macroscopic anatomical features through a global semantic path composed of two layers of linear expansion-contraction structures, while simultaneously utilizing a local interaction path composed of low-rank transformations to mine semantic relationships between fine-grained pathological attributes. Context-aware gating units are used to dynamically weight and fuse the features from both paths. The attribute prediction head then generates the probability distribution of clinical attributes in the image and extracts high-probability attribute mappings to form refined guiding attribute features. ;

[0163] The Hybrid Cross-Modal Interaction (HCMI) module constructs a cross-attention mechanism with attribute features as the core navigation, refining and guiding attribute features. As a query vector, global visual features The mapping is done as keys and values, and a learnable spatial bias matrix B is introduced into the attention weight calculation to perform inductive bias correction. The hybrid cross-modal interaction module achieves accurate spatial recalibration and depth alignment of visual regions and medical attributes in multiple subspaces through multi-head parallel computation, generating multimodal context representations with explicit attribute enhancements. ;

[0164] The input of the Transformer decoder is connected to the output of the hybrid cross-modal interaction module. It employs an autoregressive architecture combined with the generated text sequence, utilizing cascaded masked multi-head self-attention and cross-attention layers. The clinical attribute feature region in the model is strongly constrained and guided to recursively predict the probability distribution of the next term until a radiology diagnostic report is generated.

[0165] The system can be pre-trained using large-scale medical image report corpora (such as IU-Xray or MIMIC-CXR) and uses natural language processing tools to extract high-frequency clinical attribute labels.

[0166] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to perform the method described in the present invention. For example, it can run on a deep learning platform equipped with high-performance computing resources (such as an NVIDIA 4090 GPU).

[0167] This invention aims to solve the scientific problem of sparse pathological signs and cross-modal alignment imbalance in radiological images by constructing a closed-loop architecture of "retrieval-attribute guidance-multimodal interaction".

[0168] The specific embodiments described herein are merely illustrative examples of the present invention. Those skilled in the art can make various modifications or additions to the described embodiments or use similar methods to substitute them, without departing from the technology of the present invention or exceeding the scope defined by the appended claims.

[0169] In the embodiments of this application, terms such as "fixed," "fixed connection," and "fixed connection" refer to common fixing methods in the prior art, such as welding, riveting, and screws. "Rotary connection" refers to common rotary connection methods in the prior art, such as hinges and bearing rotation. If electrical components are provided, the functions, control, and power supply methods of all electrical components are common technical means in the prior art. This application has not improved them and they are not within the protection scope of this application. Therefore, this application will not elaborate on them.

Claims

1. A radiology report generation method based on enhanced guidance of multimodal attribute retrieval, characterized in that, Includes the following steps: S1. Establish a visual encoder, acquire radiological images and input them into the visual encoder for feature extraction to obtain a global visual feature vector. ; S2 utilizes the image-to-text retrieval (I2TR) mechanism to calculate the similarity of visual features in a predefined external medical knowledge base and retrieve relevant text embeddings. S3 embeds the text into the Expression Input Attribute Generation (AGM) module, predicts the probability distribution of clinical attributes present in the image through a multi-label classification layer, and maps it to the semantic space to form refined guiding attribute features. ; S4, construct a hybrid cross-modal interaction module, utilizing a cross-attention mechanism with learnable biases to integrate global visual features. With Refined Guiding Attributes Perform depth alignment to generate enhanced multimodal context representations. ; S5 represents the multimodal context. The input decoder recursively predicts the probability distribution of the next word until a complete radiology diagnostic report is generated.

2. The radiology report generation method based on multimodal attribute retrieval enhancement guidance according to claim 1, characterized in that, Acquire radiological images and input them into a visual encoder for feature extraction to obtain a global visual feature vector. The method is as follows: The acquired raw radiological images are subjected to standardized preprocessing, including size adjustment and normalization; A deep neural network was used as a visual encoder to extract multi-dimensional features from standardized preprocessed radiological images, obtain global contour information of the radiological images, segment the radiological images into multiple local regions, and extract local feature matrices containing fine textures and pathological information. By using global average pooling, the local feature matrices are condensed into a single global visual feature vector representing the overall semantics of the image, as follows: ， Where 𝐿𝑁 represents layer normalization, , is the trainable parameter, and is the input image.

3. The radiology report generation method based on multimodal attribute retrieval enhancement guidance according to claim 1, characterized in that, Step S2 utilizes the image-to-text retrieval (I2TR) mechanism to calculate the similarity of visual features in a predefined medical knowledge base, and retrieves relevant text embeddings. An external medical knowledge base is pre-built, which stores several historical images and their corresponding authoritative diagnostic reports; Using a pre-trained medical-specific text encoder, text from an external medical knowledge base is transformed into high-dimensional vectors; A dedicated cross-modal projection layer is constructed to map global visual feature vectors and text features from an external medical knowledge base into the same "shared semantic space," in which images and text no longer have modal distinctions. Calculate the global visual feature vector Embedded vectors of candidate radiology text reports from an external medical knowledge base cosine similarity : ， Cosine similarity retrieval is used to select the K reports most relevant to radiological images, achieving semantic alignment between images and text. The retrieved reports are then extracted using a pre-trained medical text encoder to generate text features for attribute detection. .

4. The radiology report generation method based on multimodal attribute retrieval enhancement guidance according to claim 3, characterized in that, Step S3 embeds the text into the Expression Input Attribute Generation (AGM) module, predicts the probability distribution of clinical attributes present in the image through a multi-label classification layer, and maps it to the semantic space to form refined guiding attribute features. The specific method is as follows: The most frequently retrieved report from external medical knowledge bases Each report contains a non-stop word as a potential attribute label, and for each report, there is a binary label vector C = { ,…, The actual tag is constructed if the attribute exists. = 1, where, For input text features ,Will Average pooling obtains text semantic features ; Text semantic features High-order semantics are captured through a global semantic path and two layers of linear expansion and contraction structures. : ， in, , It is a linear layer. , It is a weight matrix. It is the GELU activation function; Text semantic features Low-rank and fine-grained features are obtained through a local interaction path. : ， in, , It is a weight matrix; Introducing a lightweight gating unit The gating unit dynamically adjusts the contribution of each path based on the input context and input features. Real-time calculation of scalar weights : ， Fusion features Obtained by weighted summation of the two-path outputs: ， in, These are the weight parameters, and ⊙ is the Hadamard product; exist At this time, calibrated residual connections are used to bridge the dimensionality gap, projecting the original input 𝑥 onto the hidden dimension. And a learnable scaling factor 𝛾 is introduced: ， ， in, These are weight parameters. Let represent the learnable bias vector, where is a trainable parameter initialized to 0.1; This refers to the features after bridging the dimensionality gap using calibrated residual connections. The final features are obtained by introducing a learnable scaling factor 𝛾; Text features Dimension It is the hidden layer feature dimension; Features after fusion Finally, the attribute prediction head generates attribute prediction probabilities. Finally go through The activation function generates the final predicted attributes. : ， ， in, , These are weight parameters. It is a linear layer; Finally, the top R attributes with the highest probabilities are extracted as refined guiding attribute features. .

5. The radiology report generation method based on multimodal attribute retrieval enhancement guidance according to claim 1, characterized in that, In step S4, a hybrid cross-modal interaction module is constructed, utilizing a cross-attention mechanism with learnable biases to integrate global visual features. With Refined Guiding Attributes Perform depth alignment to generate enhanced multimodal context representations. ,for: ， Among them, 𝑄 is the refined guide attribute output. , 𝐾 and 𝑉 originate from global medical image features , K and V are used in the formula for calculation; B is the bias matrix that is automatically optimized during training and is used to dynamically adjust the spatial and logical mapping relationship between local visual features and semantic attribute features. This is the scaling factor.

6. The radiology report generation method based on multimodal attribute retrieval enhancement guidance according to claim 1, characterized in that, It also includes using a multi-task loss function. The steps for joint optimization are as follows: ， in, To report the generated cross-entropy loss, Multi-label binary cross-entropy loss for attribute prediction: ， ， in, It is a one-hot vector representing the label. The position in vocabulary V is 1, and the other positions are 0; It is a real label (0 or 1). It represents the probability of the predicted attribute existing; T is the length of the generated medical report, and t is the word at the t-th position in the medical report. It predicts the probability of the t-th word. is the number of Ground-Truth attribute labels in the input radiology report, and i is the predicted i-th attribute.

7. A radiology report generation system based on the method of any one of claims 1-6, characterized in that, It includes a data acquisition unit, a visual encoder, an attribute prediction and retrieval unit, a hybrid cross-modal interaction module, and a Transformer decoder; The data acquisition unit is used to acquire radiological images and transmit them to the visual encoder for feature extraction to obtain a global visual feature vector. ; The attribute prediction and retrieval unit includes an I2TR module and an AGM module connected in sequence. The I2TR module uses a cross-modal projection layer to map global visual features to a semantic space shared with the medical knowledge base, retrieves the Top-K related reports through cosine similarity, and generates an initial text feature vector through a medical-specific text encoder. The AGM module employs a dual-path heterogeneous processing architecture. It captures macroscopic anatomical features through a global semantic path composed of two layers of linear expansion-contraction structures, while simultaneously utilizing a local interaction path composed of low-rank transformations to mine semantic relationships between fine-grained pathological attributes. Context-aware gating units are used to dynamically weight and fuse the dual-path features. The attribute prediction head then generates the probability distribution of clinical attributes in the image and extracts high-probability attribute mappings to form refined guiding attribute features. ; The hybrid cross-modal interaction module constructs a cross-attention mechanism with attribute features as the core navigation, refining and guiding attribute features. As a query vector, global visual features The mapping is done as keys and values, and a learnable spatial bias matrix B is introduced into the attention weight calculation to perform inductive bias correction. The hybrid cross-modal interaction module achieves accurate spatial recalibration and depth alignment of visual regions and medical attributes in multiple subspaces through multi-head parallel computation, generating multimodal context representations with explicit attribute enhancements. ; The input of the Transformer decoder is connected to the output of the hybrid cross-modal interaction module. It employs an autoregressive architecture combined with the generated text sequence, utilizing cascaded masked multi-head self-attention and cross-attention layers. The clinical attribute feature region in the model is strongly constrained and guided to recursively predict the probability distribution of the next term until a radiology diagnostic report is generated.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by the processor to perform the method according to any one of claims 1-6.

Images