A two-stage multi-module collaborative medical report generation framework based on knowledge embedding enhancement and bias matrix guidance

Abstract

This invention discloses a two-stage, multi-module collaborative medical report generation framework based on knowledge embedding enhancement and bias matrix guidance, comprising three main modules: knowledge triple enhancement features, phrase alignment visual features, and bias matrix-guided decoding. First, knowledge triples are injected as intermediate modalities into the visual and text feature spaces. Second, the phrase alignment module simulates the pre-service training process for radiologists, learning high-frequency phrases and standardized expressions in the field. Finally, the bias matrix-guided decoding module filters candidate phrases highly relevant to images based on clinical writing logic, generating a word-level bias matrix to constrain the decoding process. This invention effectively addresses the cross-modal fusion gap between medical images and text reports, as well as the generation uncertainty and inconsistency caused by the diversity of report descriptions. This framework outperforms existing advanced models in terms of report generation fluency and clinical efficacy indicators, significantly improving diagnostic efficiency and reducing the burden on healthcare workers.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and computer vision, and mainly to a medical image report generation framework based on knowledge embedding enhancement and bias matrix guidance in a two-stage, multi-module collaborative manner. Background Technology

[0002] Image description tasks aim to generate brief text descriptions for given images. In recent years, image description tasks have attracted widespread attention from researchers. The emergence and development of numerous technologies have significantly promoted research progress in image description tasks and also provided insights for radiology report generation.

[0003] Automated radiology report generation technology has garnered widespread attention due to its ability to effectively simplify the time-consuming and error-prone process of interpreting medical images and writing reports. This technology primarily uses deep learning algorithms to extract key information from medical images and automatically generate descriptive reports, thus improving the efficiency of radiologists. Traditional manual report writing is not only time-consuming and laborious for experienced physicians, but also more prone to errors by inexperienced physicians, potentially even jeopardizing patient safety. Automated generation technology can significantly reduce the burden on physicians, decrease diagnostic errors, and thereby improve the overall quality of diagnosis and treatment. Therefore, further development of this technology is of paramount importance.

[0004] Automatic radiology report generation is an extension of image description generation, which primarily involves generating concise text from images. However, there are fundamental differences between radiology report generation and general image description: the former requires longer reports with more standardized structures and necessitates the integration of specialized medical knowledge for anomaly identification and diagnostic description, resulting in significant visual semantic differences and a knowledge gap. Furthermore, general image description tasks do not require consistency in expression, while radiology reports must adhere to standardized clinical writing guidelines, making it difficult to directly transfer general image description methods to meet actual clinical needs. Radiology report generation, as an extension and expansion of image description tasks, not only requires producing longer text outputs but also demands higher accuracy in describing imaging findings in specific regions, while ensuring consistency and adherence to clinical standards in report expression.

[0005] The primary challenge in automated radiology report generation is the difficulty of cross-modal fusion of medical images and reports. The second challenge lies in the inconsistency in descriptions during automated radiology report generation: for cases with similar imaging findings, models often generate significantly different descriptive texts; even when generating reports multiple times for the same image, the output results are difficult to maintain consistency. This inconsistency forces the model to learn multiple ways of describing the same abnormality during training, affecting not only the stability of model learning but also causing the model to tend to learn simple, universal sentence templates, thus ignoring complex and crucial differences in pathological descriptions. Most existing methods directly generate reports from image feature mapping, failing to fully adhere to the writing standards and logical processes of real clinical reports. In actual clinical practice, radiologists who have undergone standardized training follow a unified report structure and descriptive standards, and even when faced with similar images, reports issued by different physicians can maintain a high degree of consistency. Therefore, integrating the standardized training process and standardized report writing paradigm of radiologists into model design is crucial to ensuring the medical rationality and diagnostic consistency of reports. Summary of the Invention

[0006] While existing image report generation methods alleviate cross-modal semantic differences to some extent, they typically fail to integrate standardized report writing logic into the model design or adapt the model to physician-standardized writing processes. To address these challenges, we propose a two-stage, multi-module collaborative framework based on knowledge embedding enhancement and bias matrix guidance. The model architecture is as follows: Figure 1 As shown, this method is fully integrated into clinical practice and can simultaneously address the diverse challenges in visual-text fusion and report generation.

[0007] First, we leverage knowledge triples to enhance the feature module, mining the relationships between medical entities to achieve fine-grained alignment of visual and textual features at the knowledge level. Second, inspired by the standardized training process for radiologists, we designed a phrase-aligned visual feature module to simulate the pre-job training process, allowing the model to learn high-frequency phrases in the domain and strengthening the binding relationship between visual features and medical phrases. Finally, to transform the aligned knowledge into report generation logic that conforms to clinical standards, we further propose a bias matrix-guided decoder module. This module explicitly models the clinical relationships between phrases by constructing a dynamic relationship matrix and uses this relationship as a structured bias-guided decoding process, thereby improving the clinical rationality and logical consistency of the generated report.

[0008] (1) Knowledge Triple Enhancement Feature Module

[0009] For a given knowledge graph Where z represents the number of triples. For entities, For relationships, the TransE algorithm is used to learn vector representations of entities and relationships. Next, a two-layer relation-aware graph attention network (R-GAT) is used to aggregate neighborhood information and update entity representations, thus incorporating structural semantics from the graph. For the first layer of the network, relation-based message vectors are first created. Next, the importance weights of neighbors to the target entity are calculated using an attention mechanism. And perform weighted aggregation on all incoming neighbors.

[0010]

[0011]

[0012] in Represents all entities that point to Neighbor set, attention weight This is obtained by normalizing all neighbors of the same target node. Further use... We enhance image features and text features separately. We use image features as the query and knowledge entity vectors as the key / value pair. Then, we inject the weighted and aggregated knowledge features into the image features as residuals.

[0013] (2) Phrase alignment visual feature module

[0014] First, the phrase library is encoded using a text encoder to obtain the phrase embeddings. To better understand and align features and phrases, a Transformer with L layers is used for modeling.

[0015] (3) Bias matrix guided decoder module

[0016] The semantic association is quantified by calculating the cosine similarity between all feature pairs, thereby dynamically generating a relation strength matrix. The similarity matrix is ​​scaled by a learnable scaling factor. Perform calibration to generate the final bias matrix. The report decoder uses phrase-aware visual features. Original visual features and bias matrix As three core guiding signals, the system works in concert through a multi-level attention mechanism to achieve accurate generation of radiology reports. Attached Figure Description

[0017] Figure 1 This is a framework diagram for generating medical image reports based on a two-stage, multi-module collaborative approach guided by knowledge embedding enhancement and bias matrix, provided in an embodiment of the present invention.

[0018] Figure 2A flowchart illustrating the phrase library construction process in the phrase visual feature module provided in this embodiment of the invention.

[0019] Figure 3 This is a schematic diagram of the bias matrix guiding the decoder to generate a report, provided in an embodiment of the present invention.

[0020] Figure 4 This is a comparative diagram of the generated final report provided in an embodiment of the present invention. The second column in the diagram represents the phrases predicted by the model in this paper. Detailed Implementation

[0021] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0022] The overall framework diagram of this model is as follows: Figure 1 As shown, the main components include three modules: input images and reports, knowledge triplet enhanced features, phrase-aligned visual features, and bias matrix-guided decoder, as well as the final output of the generated report. The following sections provide a detailed explanation of each step.

[0023] Input section: The model architecture adopts an encoder-decoder architecture, where the image encoder is responsible for processing the image... Extracting visual features Text encoder from report Extracting text features and performing a series of operations, the decoder... Receive raw visual features Processed visual features and bias matrix Autoregressive generation of diagnostic reports This generation process can be formally defined as ,in For the generated word sequence, each word The generation depends on all previous outputs. and all input conditions.

[0024] The following is a detailed description of the process in the knowledge triplet enhanced feature module:

[0025] ① First, a medical knowledge graph is constructed based on the RadGraph dataset. By identifying entities in the report and mapping them to standard medical relations, a set of (head entity, relation, tail entity) triples is formed. Five relations are defined: has_finding (discovery at anatomical location), modified_by (tail entity modifies head entity), absence_of (pathological discovery that is not found), related_to (general relevance), and normalized_to (variant mapping to standard concept).

[0026] ② Further, the TransE algorithm is used to embed the knowledge graph Kg to obtain low-dimensional dense vectors of all entities and relations. Then, a two-layer R-GAT network is used to update the entities, with relation types as edges during the update process.

[0027] ③ After enhancing image and text features using knowledge triples, to ensure that the enhanced features do not deviate from their original semantics, we designed a local contrastive learning loss. This loss calculates a weighted symmetric InfoNCE loss between the enhanced and unenhanced features for both image and text features, thereby constraining the enhancement process and improving the discriminative power of the features. The formula is as follows:

[0028]

[0029] in This represents the cosine similarity between the two sets of local features before and after enhancement. This indicates the importance of the j-th local position.

[0030] Introduction to the phrase alignment visual feature module process:

[0031] ① Due to differences in physicians' writing styles, clinical imaging reports often exhibit heterogeneity. To standardize report expression, clinical practice typically involves providing physicians with standardized pre-employment training to master standardized report writing techniques. The n-gram method was used to extract continuous word sequences from the reports and count their frequencies, obtaining candidate phrases such as "pleural effusion" and "nofocal consolidation." Based on this, UMLS was used to remove meaningless phrase combinations, such as "a" and "is normal." Next, the phrases were sorted in descending order of frequency, and 120 high-frequency phrases were selected. The specific process is as follows: Figure 2 As shown. The phrases include anatomical structures, pathological findings, and common medical terms.

[0032] ② Visual features and phrase features are aligned through a multi-layer Transformer. For the training alignment process, the binary cross-entropy loss is used as a supervision signal.

[0033] Introduction to the bias matrix-guided decoder module process:

[0034] ① First, the obtained visual feature sequence of phrase perception is... The initial set of nodes is considered as the graph's initial node set, and a fully connected graph is constructed on this set using a two-layer multi-head self-attention mechanism. During this process, the attention weight matrix... Defined as the dynamic weight of each connecting edge in the graph, it is used to quantify the semantic association strength between any two phrase nodes in the current image context.

[0035] For each attention head, the attention weight matrix is ​​used. To calculate relation weights, and through a matrix This represents neighbor information. The outputs of each attention head will be integrated and preliminarily enhanced features will be obtained through residual connections and layer normalization. Finally, the enhanced features are obtained by nonlinear transformation through a feedforward network (FFN). The specific formula is as follows:

[0036]

[0037]

[0038] ② Figure 3 The BAAL layer in this invention is a bias-enhanced self-attention layer. In this layer, the hidden state of the current generated word sequence is used as the query, and the phrase embedding is used as the expression. The data, along with the key, is input into the cross-attention module. Through similarity calculation and Softmax normalization, an association matrix is ​​dynamically generated. This is used to quantify the semantic association strength between the currently generated lexical unit and each phrase in the medical phrase set. Then, based on the association matrix... Phrase-level bias matrix representing clinical prior relationships between phrases Through matrix transformation Obtain the word sequence-level bias matrix The matrix The key relational prior injection mask multi-head attention mechanism is used to guide the output of subsequent reports, and the specific formula is as follows:

[0039]

[0040] ③ In the decoder's cross-attention layer, the decoder uses the hidden state at the current generation position as the query vector, and simultaneously incorporates phrase-aware visual features. As a key, the original visual features As a value, the semantic content most relevant to the current generation context is dynamically retrieved from the image. This mechanism ensures that the decoder can accurately focus on the corresponding visual evidence when describing a specific anatomical structure or lesion.

[0041] Results: A qualitative comparison was made between the model framework proposed in this invention, the open-source model R2Gen, and real-world reports. The results are as follows: Figure 4 As shown, the phrases predicted by the model can guide and participate in the generation of reports. Furthermore, compared with the R2Gen model, this invention provides a more detailed description of the disease and anatomical location.

Claims

1. A two-stage, multi-module collaborative medical report generation framework based on knowledge embedding enhancement and bias matrix guidance, comprising the following steps: S1. Design a knowledge triplet enhancement feature module, using knowledge triplets as an intermediate modality to enhance image and text features respectively, achieving fine-grained alignment of visual and text features at the knowledge level; S2. Design a phrase alignment visual feature module. By simulating the pre-job training process, the model learns high-frequency phrases in the field, enhancing the binding relationship between visual features and medical phrases. S3. Design a bias matrix-guided decoder module. By constructing a dynamic relation matrix to explicitly model the clinical associations between phrases, and using this association as a structured bias-guided decoding process, the clinical rationality and logical consistency of the generated report are improved.

2. The two-stage, multi-module collaborative medical report generation framework based on knowledge embedding enhancement and bias matrix guidance as described in claim 1, characterized in that, In step S1, the steps of the knowledge triplet enhanced feature module are as follows: 1) Based on the RadGraph dataset, medical knowledge triples are constructed. By identifying entities in the report and mapping them to standard medical relations, a set of (head entity, relation, tail entity) triples is formed. Five relations are defined: has_finding (discovery of anatomical location), modified_by (tail entity modifies head entity), absence_of (absence of pathological discovery), related_to (general relevance), and normalized_to (variant mapping to standard concept). 2) For a given knowledge triple, the TransE algorithm is used to embed the knowledge triple into a low-dimensional dense vector of all entities and relations. Then, a two-layer R-GAT network is used to update the entities. During the update process, the relation type is used as an edge, and the neighboring node and relation information are aggregated. The updated entity vector retains its own semantic features and also incorporates the semantics of neighboring nodes and edges. 3) Enhance image and text features using the obtained entity vectors respectively. Use image features as the query and knowledge entity vectors as the key / value pair. Then, inject the weighted aggregated knowledge features into the image features as residuals. During the residual injection process, design independent trainable gating parameters to dynamically control the amount of knowledge information injected into the image and text features.

3. The two-stage, multi-module collaborative medical report generation framework based on knowledge embedding enhancement and bias matrix guidance as described in claim 1, characterized in that, In step S2, the model learns high-frequency phrases within its domain, enhancing the binding relationship between visual features and medical phrases. A detailed explanation follows: Visual features and phrase features are aligned through a multi-layer Transformer, and the binary cross-entropy loss is used as a supervision signal for training the alignment process.

4. The two-stage, multi-module collaborative medical report generation framework based on knowledge embedding enhancement and bias matrix guidance as described in claim 1, characterized in that, In step S3, the steps of the bias matrix guiding the decoder module are as follows: 1) A fully connected graph is constructed from the visual features of phrase perception obtained in step S2 to quantify the semantic association strength between any two phrase nodes in the current image context. 2) Design a bias-enhanced self-attention layer, using the hidden state of the current generated word sequence as a query, and the phrase embedding representation as a key and value input into the cross-attention module. Through similarity calculation and Softmax normalization, a correlation matrix is ​​dynamically generated to quantify the semantic correlation strength between the current generated word and each phrase in the medical phrase set. 3) The output matrix of the bias-enhanced self-attention layer serves as a key relational prior injection mask multi-head attention mechanism to guide the output of subsequent reports.

Images