Method, system and equipment for automatically generating X-ray chest radiography report based on factual description enhancement and medium

Abstract

The invention discloses an X-ray chest radiography report automatic generation method, system and device based on factual description enhancement and a medium. The method comprises the following steps: firstly, constructing a medical entity extraction method based on a RadGraph model, and carrying out identification and structured extraction on clinical keywords to obtain factual description consisting of key medical entities; secondly, establishing a comparative learning method guided by factual description, enhancing semantic consistency between the image and the text from global and local levels, and extracting visual features with diagnostic value; establishing a historical similar case retrieval strategy independent of disease tags, and calculating visual semantic similarity to realize automatic retrieval of historical cases; and finally, proposing an evidence-driven chest radiography report generation method, constructing a cross-modal fusion network, and generating a chest radiography report with clinical accuracy and consistency. The system, the equipment and the medium automatically generate an X-ray chest radiography report based on factual description enhancement based on the method; according to the method, efficient and stable automatic retrieval is realized, the clinical accuracy of the generated chest radiograph report and the reliability of evaluation are improved, and the universality and robustness of the model are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of automatic text generation technology, and in particular to a method, system, device and medium for automatically generating chest X-ray reports based on factual description enhancement. Background Technology

[0002] Currently, the mainstream technical approaches for the automatic generation of chest X-ray reports mainly fall into two categories: report generation methods based on cross-modal alignment and knowledge-driven report generation methods. Both of these methods have been extensively studied in recent years and have, to some extent, promoted the development of automated report generation systems. However, significant technical bottlenecks still exist in key areas such as factual modeling, accuracy of image-text semantic alignment, and generalization ability.

[0003] The core of report generation methods based on cross-modal alignment is learning the deep semantic mapping between medical images and chest X-ray reports. By constructing a shared semantic space, medical images and their corresponding linguistic descriptions are aligned in latent representation, thereby improving image understanding and report generation quality. MGCA [WANG F, ZHOU Y, WANG S, et al. Multi-granularity cross-modal alignment for generalized medical visual representation learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 33536-33549.] proposed a multi-granularity alignment mechanism, including instance-level alignment between images and complete reports, lesion region-level alignment, and disease label-level alignment. Although this method enhances the model's focus on lesion regions, it still uses the complete report as the alignment target, failing to distinguish between "factual descriptions" (such as diseases, anatomical structures, observations, etc.) and "non-diagnostic language" (such as conjunctions, modal particles, vague statements, etc.). This introduces a large amount of semantic noise during the learning process, affecting the model's focus on key diagnostic information. PTUnifier [CHEN Z, DIAO S, WANG B, et al. Towards unifying medical vision-and-language pre-training via soft prompts[C]. Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023:23403-23413] introduces a prompt learning mechanism to achieve fine-grained alignment at both the instance and lexical levels, improving the ability to capture detailed semantics. This design enhances the model's ability to model the relationship between local words and image features in the report. However, this method also fails to filter semantic importance at the lexical level, resulting in a mixture of redundant language and diagnostic terminology, which affects the quality of image-text alignment and thus reduces the clinical accuracy of the generated report.MedCLIP [WANGZ, WU Z, AGARWAL D, et al. Medclip: Contrastive learning from unpaired medical images and text [C]. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022: 3876-3887] borrows from the general CLIP framework [RADFORD A, KIM JW, HALLACY C, et al. Learning transferable visual models from natural language supervision [C]. International Conference on Machine Learning, 2021: 8748-8763.], using large-scale unlabeled image and text data for pre-training the model to build a basic semantic bridge between medical images and chest X-ray reports, demonstrating strong transfer capabilities across multiple downstream tasks. However, similar to the aforementioned methods, MedCLIP still uses the complete report as a semantic anchor for alignment training, without explicitly optimizing the model's ability to learn "factual descriptions".

[0004] Knowledge-driven report generation methods enhance the prior knowledge support for language generation by retrieving historical reports or text fragments that are semantically similar to the target image from a large-scale database as references or templates for generating new reports. CXR-RePaiR [ENDO M,KRISHNAN R,KRISHNA V, et al. Retrieval-based chest x-ray report generation using a pre-trained contrastive language-image model[C]. Proceedings of Machine Learning for Health, Proceedings of Machine Learning Research,2021:209--219.] calculates the similarity between images and reports based on a pre-trained image-text matching model CLIP, retrieving the report closest to the current image from the training set and directly using it to generate the result. This strategy can significantly reduce the pressure of language modeling while improving the natural fluency of the report structure. However, since historical reports often cannot achieve a complete semantic match with the current case, biases can easily arise in terms of lesion location, severity, or co-occurring lesions, easily leading to "factual errors," such as incorrectly describing non-existent lesions or omitting newly appearing symptoms. X-REM [JEONG J, TIAN K, LI A, et al. Multimodal image-text matching improves retrieval-based chest x-rayreport generation[C]. Medical Imaging with Deep Learning, Proceedings of Machine Learning Research, 2024: 978--990.] introduces a two-stage retrieval mechanism. First, candidate reports are screened based on the image-text matching model ALBEF [LI J, SELVARAJU R, GOTMARE A, et al. Align before fuse: Vision and language representation learning with momentum distillation[C]. Advances in Neural Information Processing Systems, 2021: 9694-9705.], and then the image-text natural language inference module is used to reorder the reports and optimize the semantic relevance of the final generated content.While this strategy improves matching accuracy, it still uses the complete report as a basic template and lacks the ability to adaptively model newly emerging lesions in the image, easily leading to "template-based" generated content or loss of key new information. To alleviate these problems, some scholars have introduced structured medical knowledge (such as disease labels and knowledge graphs) to assist model training and enhance semantic modeling capabilities. For example, GSKET [YANG S, WUX, GE S, et al. Knowledge matters: Chest radiology report generation with general and specific knowledge[J]. Medical Image Analysis, 2022, 80: 102510.] incorporates historical reports with the same disease labels into the model, improving its generality and task specificity. M2KT [YANG S, WU X, GE S, et al. Radiology report generation with a learned knowledgebase and multi-modal alignment[J]. Medical Image Analysis, 2023, 86: 102798.] utilizes the similarity between image-text pairs to construct a knowledge base and distills knowledge to improve alignment quality. DCL [LI M, LIN B, CHEN Z, et al. Dynamic graph enhanced contrastive learning for chest x-ray reportgeneration[C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023:3334-3343.] further introduces dynamic graph representation to enhance image semantic expression and knowledge linkage. Although knowledge-driven methods have certain advantages in terms of report content structure, universality, and prior coverage, their over-reliance on disease labels when building the knowledge base leads to a sharp decline in model performance when labels are missing.

[0005] While current automated generation technology for chest X-ray reports has alleviated the burden on radiologists to some extent, it still faces several technical bottlenecks in practical application, mainly including the following:

[0006] (1) Lack of effective modeling of factual descriptions. Existing cross-modal alignment methods are usually trained using the entire chest X-ray report as a supervision signal, failing to effectively distinguish factual descriptions with key diagnostic value (such as lesion location, pathological features, and disease name) from linguistic components without clinical decision significance (such as conjunctions and template sentences). This coarse-grained modeling approach introduces semantic noise, leading to inaccurate semantic alignment between images and text, reducing the model's ability to extract key medical information, and ultimately affecting the diagnostic accuracy and clinical credibility of the generated report.

[0007] (2) The knowledge introduction mechanism relies on structured labels, which have poor universality and insufficient robustness. Existing technologies generally use structured disease labels to retrieve historical similar cases to introduce external knowledge and enhance the model's professionalism and judgment ability. However, this mechanism is highly dependent on manually constructed labeling systems. Once labels are missing, inaccurate, or lack coverage in new disease contexts, the model performance will significantly decline, making it difficult to maintain stability and universality in real clinical environments where labels are unavailable or unstable. Summary of the Invention

[0008] To address the problems existing in the prior art, the present invention discloses an automatic generation method, system, device, and medium for chest X-ray reports based on factual description enhancement. By explicitly distinguishing and extracting factual descriptions (strings consisting only of clinical keywords) from the chest X-ray report, the image encoder is guided to focus on semantic information with clinical diagnostic value, thereby reducing interference from non-diagnostic language and enhancing the semantic alignment accuracy between X-ray images and chest X-ray reports. Historical similar cases are retrieved based on the semantic similarity between X-ray images, without relying on manual disease labels. In practical application scenarios where labels are missing or incomplete, efficient and stable automatic retrieval can be achieved, improving the clinical accuracy and reliability of generated chest X-ray reports, and significantly enhancing the model's versatility and robustness.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] An automatic generation method for chest X-ray reports based on factual description enhancement includes the following steps:

[0011] Step 1: Construct a medical entity extraction method based on the RadGraph model to identify and extract clinical keywords, including disease, anatomical structure, and observation results, from chest X-ray reports. During the extraction process, redundant information or entities without diagnostic value are removed to obtain factual descriptions composed of key medical entities. The obtained factual descriptions serve as high-quality semantic supervision signals in cross-modal alignment.

[0012] Step 2: Establish a contrastive learning method guided by factual descriptions to enhance the semantic consistency between images and text from two levels: (1) global level, that is, the alignment between the image and the corresponding overall factual description; (2) local level, that is, the alignment relationship between local areas in the image and the corresponding medical entities; train the image encoder to effectively learn the correspondence between medical images and factual descriptions, reduce the interference of non-diagnostic language on visual semantic modeling, and thus extract visual features with diagnostic value.

[0013] Step 3: Establish a historical similar case retrieval strategy that does not rely on disease labels. Without relying on disease labels, construct a visual representation space using the visual features extracted by the image encoder trained in Step 2; and realize automatic retrieval of historical cases by calculating visual semantic similarity.

[0014] Step 4 proposes an evidence-driven method for generating chest X-ray reports. A cross-modal fusion network is constructed to effectively integrate the extracted visual features with the semantic information of chest X-ray reports corresponding to similar historical cases, generating a joint semantic representation, which is then used as the input to the text decoder. By providing cross-modal prior knowledge, the text decoder is guided to generate chest X-ray reports with clinical accuracy and consistency.

[0015] The specific method for step 1 is as follows:

[0016] 1.1 Entity Extraction and Screening: First, the chest X-ray report is input into the RadGraph-based entity recognition module to obtain an initial entity sequence containing clinical keywords; then, the extracted entity sequence is cleaned, specifically including: removing noisy entities that are not related to medicine, and when multiple medical entities exist at the same location, retaining the key entities with higher information density.

[0017] 1.2 Factual Subsequence Generation: After entity extraction and screening in step 1.1, the retained clinical entities are sorted according to their textual order in the original chest X-ray report. Then, the entity sequence is divided into several factual subsequences using the sentence-end punctuation marks as demarcation points. For subsequences containing entities labeled as "O-DA" or "OU" by the RadGraph model, the negative word "no" or the uncertain word "maybe" is added before the corresponding entity to enhance the completeness of semantic expression and the coherence of the context.

[0018] 1.3 Factual Description Construction: The [SEP] subsequences are separated to construct the factual description of the i-th sample:

[0019]

[0020] in, This represents the number of factual subsequences in the i-th sample.

[0021] The specific method for step 2 is as follows:

[0022] 2.1 Image and Text Feature Extraction Process

[0023] A pre-trained ResNet-101 model was selected as the image encoder, and visual features of the images were extracted based on the input set of chest X-rays X.

[0024]

[0025] Where X0 represents the global image features. For image encoders, B represents the batch size; a pre-trained six-layer SciBERT model is used as the text encoder to process factual descriptions S = {S...} 1 ,S 2 ,…,S B Encode the text to obtain the corresponding text features.

[0026]

[0027] Where, N R This represents the number of tokens in the factual description, where T0 is the global text feature. For a text encoder, two projection heads are introduced to project image features and text features onto the same dimension d, respectively. and To align image features and text features in the same spatial space;

[0028] 2.2 Alignment loss functions are constructed at both the global and local levels to optimize the visual encoder, thereby further enhancing the semantic consistency between chest X-ray images and their corresponding factual descriptions:

[0029] 2.2.1) Global-level image-text alignment loss function:

[0030] First, we define an alignment loss function from the image to the text direction to ensure the semantic consistency between medical images and their factual descriptions in the embedding space, as follows:

[0031]

[0032] Where, sim(X0,T0)=(X0·T0) T) / (||X0||·||T0||) represents the cosine similarity between the global visual representation X0 and the global textual representation T0, where τ1 is the temperature coefficient; this alignment loss function, Equation (4), encourages the global visual representation Matching global text representation Maximize the similarity between them, while suppressing the global text representation that does not match. The similarity between them;

[0033] Similarly, the alignment loss function from text to image direction can be expressed as:

[0034]

[0035] The alignment loss function, Equation (5), encourages global text representation. Matching global visual representation Maximize the similarity between them, while suppressing the mismatch with global visual representations. The similarity between them; the alignment loss functions in the two directions mentioned above together constitute the global-level image-text alignment loss function guided by factual description, which is defined as follows:

[0036]

[0037] 2.2.2) Local-level image-text alignment loss function:

[0038] First, local cross-modal text features are extracted using the cross-attention mechanism ATTN(Q,K,V). Among them, local text features As the query vector (Q), and the local visual feature X loc =[X1,…,X 49 As key vectors (K) and value vectors (V), the final output is local cross-modal text features. The calculation process is expressed as follows:

[0039]

[0040] Subsequently, the local textual features T in the chest X-ray report were analyzed. loc and local cross-modal text features Tokens in the same position are considered positive sample pairs, while combinations of other mismatched positions are considered negative sample pairs. Thus, the local-level image-text alignment loss function guided by factual description is defined as follows:

[0041]

[0042] Where, N R This indicates the number of tokens in the factual description, and τ2 is the temperature coefficient.

[0043] Finally, by combining the global-level image-text alignment loss function (Equation (6)) and the local-level image-text alignment loss function (Equation (8), a total contrastive learning objective function guided by factual description is constructed, as follows:

[0044]

[0045] The specific method for step 3 is as follows:

[0046] First, a visual representation space for the candidate set is constructed: the candidate set of historical similar cases for each sample to be retrieved is derived from image samples in the training set to prevent data leakage; assuming the training set contains n chest X-ray image samples, the ResNet-101 model pre-trained in step 2 is used as the image encoder. Visual features are extracted from these n chest X-ray images, and these visual features are placed in a set to construct a candidate set visual representation space H = {X0, X1, ..., X...}. n};

[0047] Then, for the target image x to be retrieved q Also using an image encoder Extract its visual features

[0048] Next, cosine similarity is used to measure the target image X. q The visual semantic similarity between X and the candidate set samples is calculated after obtaining all similarities. q The k candidate samples with the highest similarity were selected as historical similar cases. The calculation process is expressed as follows:

[0049]

[0050] The specific method for step 4 is as follows:

[0051] 4.1 Extracting Evidence Information Features: For the i-th sample, using the historical similar case retrieval strategy described in step 3, obtain historical similar cases consisting of k samples. Based on the RadGraph medical entity extraction method designed in step 1, historical similar cases are... Each historical chest X-ray report is converted into k factual descriptions; then, a text encoder with frozen parameters and a projection head are input to obtain evidence information features. Its processing can be represented as follows

[0052]

[0053] Where SE(·) represents the medical entity extraction method based on the RadGraph model; The total number of tokens in the evidence information features. Let be the number of tokens in the u-th historical chest X-ray report of the i-th sample;

[0054] 4.2 Constructing a cross-modal fusion network: used to integrate the visual features X extracted in step 2. i Features of evidence information extracted from similar historical cases The Cross-modal Fusion Network (CMF) is integrated into a single network. It consists of three sub-modules: a self-attention sub-layer, a cross-attention sub-layer, and a feedforward neural network sub-layer. Its overall structure is as follows:

[0055]

[0056] Among them, W qs W ks W vs W qc W kc and W vc All are learnable parameters; LN(·) and FFN(·) represent layer normalization and feedforward neural network sublayers, respectively;

[0057] 4.3 Generating Chest X-ray Reports: Integrating Cross-Modal Features The input text decoder uses an autoregressive approach to progressively generate chest X-ray reports. At time t, the decoder output depends on the previously generated token sequence. The training objective is to minimize the negative log-likelihood loss function, and its calculation process is as follows:

[0058]

[0059] Where M represents the maximum length of the chest X-ray report generated by the text decoder. This represents the sequence of words generated before time t.

[0060] An automated X-ray report generation system based on factual description enhancement includes:

[0061] The medical entity extraction module is used in step 1 to identify and extract clinical keywords, including diseases, anatomical structures, and observation results, from chest X-ray reports based on the RadGraph model, thereby obtaining a factual description composed of clinical keywords.

[0062] The factual description-guided contrastive learning module is used in step 2 to enhance the semantic consistency between images and text at two levels: global and local. It trains the image encoder to effectively learn the correspondence between medical images and factual descriptions, reduces the interference of non-diagnostic language on visual semantic modeling, and thus extracts visual features with diagnostic value.

[0063] The historical similar case retrieval module, which does not rely on disease labels, is used in step 3 to construct a visual representation space using the visual features extracted by the image encoder trained in step 2 without relying on disease labels; and to realize the automatic retrieval of historical cases by calculating visual semantic similarity.

[0064] Step 4, the evidence-driven chest X-ray report generation module, is used to effectively integrate the extracted visual features with the semantic information of chest X-ray reports corresponding to similar historical cases by constructing a cross-modal fusion network in step 4, generate a joint semantic representation, and use it as the input of the text decoder; by providing cross-modal prior knowledge, it guides the text decoder to generate chest X-ray reports with clinical accuracy and consistency.

[0065] An automated X-ray chest radiograph report generation device based on factual description enhancement includes:

[0066] Memory, used to store computer programs;

[0067] A processor is configured to implement the method for automatically generating chest X-ray reports based on factual descriptions when executing the computer program.

[0068] A computer-readable storage medium storing a computer program that, when executed by a processor, is capable of automatically generating chest radiograph reports based on a factual description-enhanced method for automatically generating chest radiograph reports.

[0069] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0070] (1) Improve cross-modal semantic alignment accuracy. By constructing a medical entity extraction method based on the RadGraph model, key clinical entities in chest X-ray reports are extracted to generate factual descriptions. Combined with a global and local level contrastive learning mechanism, the image encoder is guided to focus on visual semantic content related to diagnosis, thereby significantly improving the semantic alignment effect between images and text.

[0071] (2) Achieve retrieval of historical similar cases without disease labels. Without relying on disease labels, a visual representation space is constructed using the trained image encoder, and the automatic retrieval of historical similar cases is achieved by calculating visual semantic similarity, which alleviates the semantic difference problem between image and text modalities and improves the retrieval ability of the model in clinical application scenarios where labels are missing.

[0072] (3) Improve the clinical accuracy and consistency of chest X-ray report generation. By constructing an evidence-driven chest X-ray report generation method, the visual features of the current image are fused with the semantic information from similar historical cases to construct a joint semantic representation. This effectively guides the text generator to generate radiological reports that are more consistent with clinical logic and have more accurate content, thereby improving the quality and consistency of the reports.

[0073] In summary, this invention significantly improves semantic alignment of images and text by introducing factual description extraction, image-text comparison learning, unlabeled similar case retrieval, and evidence-driven report generation mechanisms. It enables automatic retrieval of similar cases without disease labels and guides the generation of accurate chest X-ray reports that conform to clinical reasoning paths by integrating traceable clinical evidence information. This significantly enhances the model's generalization ability and practicality in real clinical environments, demonstrating good practical value and promising prospects for wider application. Attached Figure Description

[0074] Figure 1 This is a flowchart of the present invention.

[0075] Figure 2 These are two factual descriptions of medical entity extraction based on the RadGraph model used in this invention.

[0076] Figure 3 This invention provides an algorithmic framework for a contrastive learning method guided by factual descriptions.

[0077] Figure 4 This invention provides an algorithm framework for a historical similar case retrieval strategy based on aligned visual features.

[0078] Figure 5 This is the algorithm framework for the evidence-driven chest X-ray report generation method of this invention.

[0079] Figure 6 This invention provides a cross-modal fusion network algorithm framework.

[0080] Figure 7 This refers to the historical similar cases and their factual descriptions corresponding to two samples in the MIMIC-CXR dataset of this invention. Detailed Implementation

[0081] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0082] like Figure 1 As shown, an automatic generation method for chest X-ray reports based on factual description enhancement specifically includes the following steps:

[0083] Step 1: Chest X-ray reports typically contain two types of descriptions: one is "factual description," which describes objective clinical observations; the other is "non-diagnostic language," such as conjunctions, modal particles, or semantically ambiguous statements. The former has direct value for clinical decision-making, while the latter, although helpful in enhancing the fluency and structure of the report, may introduce interference in the text-image alignment task. Therefore, this invention constructs a medical entity extraction method based on the RadGraph model to identify and structurally extract clinical keywords from chest X-ray reports, including disease, anatomical structure, and observation results. During the extraction process, redundant information or entities without diagnostic value are removed to obtain factual descriptions composed of key medical entities. The obtained descriptions serve as high-quality semantic supervision signals in cross-modal alignment.

[0084] The specific method is as follows:

[0085] 1.1 Entity Extraction and Screening: First, the chest X-ray report is input into the RadGraph-based entity recognition module to obtain an initial entity sequence containing clinical keywords. This sequence may contain redundant information and noise interference. Subsequently, the extracted entity sequence is cleaned, specifically including: removing noisy entities that are not related to medicine (such as "These" and other non-medical related content); when two entities exist at the same time, the key entity with higher information density is retained through redundancy filtering (for example, when two entities "1.9×1.0cm" and "1.0cm" exist at the same time, the former is retained and the latter is discarded).

[0086] 1.2 Factual Subsequence Generation: After entity extraction and filtering in step 1.1, the retained clinical entities are sorted according to their textual order in the original chest X-ray report. Then, using sentence-ending punctuation marks (e.g., period, question mark, etc.) as demarcation points, the entity sequence is divided into several factual subsequences. For subsequences containing entities labeled as "O-DA" or "OU" by the RadGraph model, the negative word "no" or the uncertain word "maybe" is added before the corresponding entity to enhance the semantic completeness and contextual coherence. For example... Figure 2 As shown: The upper part of the figure displays the entities and their relationships extracted using the RadGraph model, while the lower part displays the factual serialization extracted based on the above results. "O-DP", "A-DP", and "OU" represent different types of radiological entities, respectively, while "modify", "located_at", and "suggestive_of" represent the relationships between entities.

[0087] 1.3 Factual Description Construction: The [SEP] subsequences are separated to construct the factual description of the i-th sample:

[0088]

[0089] in, This represents the number of factual subsequences in the i-th sample.

[0090] This invention performs systematic noise filtering on RadGraph output; this invention preserves the order in which entities appear in the original text, thereby ensuring the continuity of semantic context.

[0091] Step 2, establish a contrastive learning method guided by factual descriptions, such as... Figure 3 As shown, this method utilizes the semantic correspondence between images and factual descriptions at the instance and token levels to optimize visual representation. It strengthens the semantic consistency between images and text at two levels: (1) the global level, i.e., the alignment between the image and the corresponding overall factual description; (2) the local level, i.e., the alignment between local regions in the image and the corresponding key medical entities; training the image encoder effectively learns the correspondence between medical images and factual descriptions, reduces the interference of non-diagnostic language on visual semantic modeling, and thus extracts visual features with diagnostic value.

[0092] The specific method is as follows:

[0093] 2.1 Image and Text Feature Extraction Process

[0094] A pre-trained ResNet-101 model was selected as the image encoder, and visual features of the images were extracted based on the input set of chest X-rays X.

[0095]

[0096] Where X0 represents the global image features. For image encoders, B represents the batch size; a pre-trained six-layer SciBERT model is used as the text encoder to process factual descriptions S = {S...} 1 ,S 2,…,S B Encode the text to obtain the corresponding text features.

[0097]

[0098] Where, N R This represents the number of tokens in the factual description, where T0 is the global text feature. For a text encoder, two projection heads are introduced to project image features and text features onto the same dimension d, respectively. and To align image features and text features in the same spatial space;

[0099] 2.2 Alignment loss functions are constructed at both the global and local levels to optimize the visual encoder, thereby further enhancing the semantic consistency between chest X-ray images and their corresponding factual descriptions:

[0100] 2.2.1) Global-level image-text alignment loss function:

[0101] First, we define an alignment loss function from the image to the text direction to ensure the semantic consistency between medical images and their factual descriptions in the embedding space, as follows:

[0102]

[0103] Where, sim(X0,T0)=(X0·T0) T ) / (||X0||·||T0||) represents the cosine similarity between the global visual representation X0 and the global textual representation T0, where τ1 is the temperature coefficient; this alignment loss function, Equation (4), encourages the global visual representation Matching global text representation Maximize the similarity between them, while suppressing the global text representation that does not match. The similarity between them;

[0104] Similarly, the alignment loss function from text to image direction can be expressed as:

[0105]

[0106] The alignment loss function, Equation (5), encourages global text representation. Matching global visual representation Maximize the similarity between them, while suppressing the mismatch with global visual representations. The similarity between them; the alignment loss functions in the two directions mentioned above together constitute the global-level image-text alignment loss function guided by factual description, which is defined as follows:

[0107]

[0108] 2.2.2) Local-level image-text alignment loss function:

[0109] First, local cross-modal text features are extracted using the cross-attention mechanism ATTN(Q,K,V). Among them, local text features As the query vector (Q), and the local visual feature X loc =[X1,…,X 49 As key vectors (K) and value vectors (V), the final output is local cross-modal text features. The calculation process is expressed as follows:

[0110]

[0111] Subsequently, the local textual features T in the chest X-ray report were analyzed. loc and local cross-modal text features Tokens in the same position are considered positive sample pairs, while combinations of other mismatched positions are considered negative sample pairs. Thus, the local-level image-text alignment loss function guided by factual description is defined as follows:

[0112]

[0113] Where, N R This indicates the number of tokens in the factual description, and τ2 is the temperature coefficient.

[0114] Finally, by combining the global-level image-text alignment loss function (Equation (6)) and the local-level image-text alignment loss function (Equation (8), a total contrastive learning objective function guided by factual description is constructed, as follows:

[0115]

[0116] Step 3: Establish a historical similar case retrieval strategy that does not rely on disease labels, such as... Figure 4 As shown, without relying on disease labels, a visual representation space is constructed using the visual features extracted by the image encoder trained in step 2; by calculating visual semantic similarity, automatic retrieval of historical cases is achieved; this retrieval strategy alleviates the semantic differences between image and text modalities and effectively improves the model's generalization ability and adaptability in clinical application scenarios such as missing labels and scarce samples.

[0117] The specific method is as follows:

[0118] First, a visual representation space for the candidate set is constructed: the candidate set of historical similar cases for each sample to be retrieved is derived from image samples in the training set to prevent data leakage; assuming the training set contains n chest X-ray image samples, the ResNet-101 model pre-trained in step 2 is used as the image encoder. Visual features are extracted from these n chest X-ray images, and these visual features are placed in a set to construct a candidate set visual representation space H = {X0, X1, ..., X...}. n};

[0119] Then, for the target image x to be retrieved q Also using an image encoder Extract its visual features

[0120] Next, cosine similarity is used to measure the target image X. q The visual semantic similarity between X and the candidate set samples is calculated after obtaining all similarities. q The k candidate samples with the highest similarity were selected as historical similar cases. The calculation process is expressed as follows:

[0121]

[0122] Since the extracted visual features are derived from the factual description-guided contrastive learning method in step 2, they possess clear medical semantic discrimination capabilities. Therefore, this strategy can achieve semantically accurate case matching without relying on disease labels.

[0123] Step 4 proposes an evidence-driven chest X-ray report generation method. A cross-modal fusion network is constructed to effectively integrate extracted visual features with semantic information from chest X-ray reports of similar historical cases, generating a joint semantic representation, which is then used as input to the text decoder. By providing cross-modal prior knowledge, the text decoder is guided to generate chest X-ray reports with clinical accuracy and consistency. Figure 5 As shown, this method improves the accuracy and reliability of generated reports by integrating factual language descriptions (evidence) from similar historical cases.

[0124] The specific method is as follows:

[0125] 4.1 Extracting Evidence Information Features: For the i-th sample, using the historical similar case retrieval strategy described in step 3, obtain historical similar cases consisting of k samples. Based on the RadGraph medical entity extraction method designed in step 1, historical similar cases are... Each historical chest X-ray report is converted into k factual descriptions; then, a text encoder with frozen parameters and a projection head are input to obtain evidence information features. Its processing can be represented as follows

[0126]

[0127] Where SE(·) represents the medical entity extraction method based on the RadGraph model; The total number of tokens in the evidence information features. Let be the number of tokens in the u-th historical chest X-ray report of the i-th sample;

[0128] 4.2 Constructing a cross-modal fusion network: used to integrate the visual features X extracted in step 2. i Features of evidence information extracted from similar historical cases Integration is performed; the Cross-modal Fusion Network (CMF) consists of three sub-modules: a self-attention sub-layer, a cross-attention sub-layer, and a feedforward neural network sub-layer, such as... Figure 6 As shown, its overall structure is as follows:

[0129]

[0130] Among them, W qs W ks W vs W qc W kc and W vc All are learnable parameters; LN(·) and FFN(·) represent layer normalization and feedforward neural network sublayers, respectively;

[0131] 4.3 Generating Chest X-ray Reports: Integrating Cross-Modal Features The input text decoder uses an autoregressive approach to progressively generate chest X-ray reports. At time t, the decoder output depends on the previously generated token sequence. The training objective is to minimize the negative log-likelihood loss function, and its calculation process is as follows:

[0132]

[0133] Where M represents the maximum length of the chest X-ray report generated by the text decoder. This represents the sequence of words generated before time t.

[0134] The present invention provides an evaluation mechanism for report generation quality in different scenarios.

[0135] In actual clinical practice, different diagnostic and treatment scenarios have varying requirements for the level of detail in chest X-ray reports: emergency diagnostic scenarios prefer concise and focused reports, while routine examination scenarios emphasize comprehensive and detailed information. Existing assessment methods often overlook this difference and fail to accurately reflect the adaptability of report generation systems across multiple scenarios. Therefore, this invention constructs a report quality assessment mechanism adapted to multiple clinical scenarios, simulating the required length M of the reference report under different usage scenarios during the testing phase. gt The requirements were assessed. Specifically, during testing, the generated report content remained unchanged, but the reference report was truncated to adapt to different assessment settings: in general scenarios, the complete reference report was used as the comparison basis and marked "Cpl." (Completion); in specific scenarios (such as emergency diagnosis), the reference report was truncated to a preset length to simulate the need for report conciseness in clinical practice. This mechanism makes the evaluation of the report generation system more consistent with the needs of real clinical use environments, helping to comprehensively measure its practicality and effectiveness in different task scenarios.

[0136] Experimental results

[0137] a. Experiment Setup Instructions

[0138] 1) Dataset Introduction:

[0139] This invention validates the effectiveness of the proposed FSE method on two standard datasets widely used in the field of automated chest X-ray report generation—MIMIC-CXR and IU X-ray. Consistent with existing research, this invention uses the Findings section of the chest X-ray report as a reference report for evaluation. The statistics after data cleaning are shown in Table 1.

[0140] Table 1. Statistical breakdown of MIMIC-CXR and IU X-ray datasets

[0141]

[0142] 2) Explanation of evaluation indicators:

[0143] To comprehensively evaluate report generation performance, this invention employs both Natural Language Generation (NLG) and Clinical Efficacy (CE) metrics to measure the linguistic similarity and clinical consistency between the generated report and the reference report. NLG metrics include BERTScore (BS), BLEU-n (BL-n, where n represents n-gram matching of orders 1 to 4), METEOR (MTR), and ROUGE-L (RL). CE metrics include F1-CheXbert14 (CX14, the micro-mean F1 score based on CheXpert annotations for 14 categories of chest diseases and supporting equipment), F1-CheXbert5 (CX5, the micro-mean F1 score based on CheXpert annotations for 5 common chest abnormalities), and F1-RadGraph (RG, the degree of overlap between clinical entities and their relationships extracted from the generated report based on RadGraph parsing and the actual report). Higher scores on all these metrics indicate better report quality. The CE metric was calculated using the f1chexbert and radgraph toolkits.

[0144] 3) Experimental setup instructions:

[0145] In terms of model structure, this invention uses ResNet101 as the image encoder, SciBERT as the text encoder, and the R2Gen Transformer with a memory mechanism as the text decoder. The training strategy is as follows: On the MIMIC-CXR dataset, the first stage (a sex description-guided contrastive learning method) uses the AdamW optimizer with an initial learning rate of 5e-5, trained for 100 epochs; the second stage (an evidence-driven report generation method) uses the RAdam optimizer with a learning rate of 5e-5, trained for 50 epochs. On the IU X-ray dataset, given its smaller data size, the first stage is skipped, and the process proceeds directly to the second stage. The model is initialized using the parameter weights pre-trained on the MIMIC-CXR dataset, using the RAdam optimizer with a learning rate of 1e-4, trained for 30 epochs. On the validation set, the model comprehensively considers the RG, CX14, and BLEU-4 scores, selecting the optimal version for final test set evaluation.

[0146] b. Explanation of main experimental results

[0147] Table 2 compares the performance of existing mainstream methods on the MIMIC-CXR dataset.

[0148]

[0149]

[0150] In actual clinical practice, the required level of detail in chest X-ray reports varies across different diagnostic and treatment scenarios. Therefore, this invention constructs a report quality assessment mechanism adaptable to multiple clinical scenarios, simulating the required length M of the reference report under different usage scenarios during the testing phase. gt The requirements were assessed. Specifically, during testing, the generated report content remained unchanged, but the reference report was truncated to adapt to different assessment settings: in general scenarios, the complete reference report was used as the comparison basis and marked "Cpl." (Completion); in specific scenarios (such as emergency diagnosis), the reference report was truncated to a preset length to simulate the clinical need for report conciseness. The proposed FSE method was compared with 14 generative methods, including: R2Gen, R2GenCMN, GSKET, CMCA, CvT2DistillGPT2, M2KT, DCL, METransformer, OpenLLaMA-7B, SA, MMTN, CAMANet, RAMT, and FMVP, as well as a retrieval method: X-REM. CXR-RePaiR was not included in the comparison because it only generated the diagnostic conclusion portion, and RGRG used a different data partitioning strategy.

[0151] Table 3 compares the performance of existing mainstream methods on the IU X-ray dataset.

[0152]

[0153] Experimental results on the MIMIC-CXR and IU X-ray datasets are shown in Tables 2 and 3, respectively, where FSE-n indicates that the proposed method incorporates knowledge from n historical similar cases when generating reports. For partially missing M... gt The performance of the comparison method is for reference only and does not participate in the final ranking. As can be seen from the experimental results in Table 2, FSE-1 demonstrates strong comprehensive performance compared to various mainstream generative and retrieval methods. Overall, generative methods generally outperform retrieval methods in terms of natural language generation quality and clinical consistency, while the FSE-1 method of this invention achieved best or second-best results in both evaluation dimensions, demonstrating good competitiveness and stability. Specifically, in a specific scenario (i.e., M... gt ∈{60,80,90,100}) and general scenarios (i.e., M) gt Under the condition of (=Cpl.), FSE-1 exhibits excellent performance in both NLG and CE metrics. Especially in specific scenarios M gtAt a resolution of 80, FSE-1 achieved scores of 0.397, 0.238, and 0.470 on the BLEU-1 (BL-1), F1-RadGraph (RG), and F1-CheXbert14 (CX14) metrics, respectively, significantly outperforming most comparative methods. Although in specific scenarios M... gt At a resolution of 60, FSE-1 slightly underperforms CvTDistillGPT2 on some NLG metrics, but the method of this invention still achieves a relative improvement of 4.1% in overall performance. Furthermore, as shown in Table 3, FSE-20 also performs excellently on the IU X-ray dataset in terms of NLG and CE metrics. In particular, it achieves scores of 0.504 and 0.607 on the two key metrics CX5 and CX14, respectively, significantly higher than most generative methods, further validating the stability and generalization ability of the method of this invention in small-scale data scenarios.

[0154] b. Ablation Experiment Instructions

[0155] 1) Contribution analysis of each module:

[0156] To further verify the effectiveness of each functional module proposed in this invention, ablation experiments were conducted on the MIMIC-CXR dataset, and the experimental results are summarized in Table 4 to quantitatively analyze the impact of each module on report generation performance. The "BASE" model is the basic contrast scheme, using the same image encoder (ResNet101) and text decoder (memory-driven Transformer) as this invention, but without introducing factual description guidance or historical similar case modules. In the second stage experiment introducing historical similar cases, five similar cases (i.e., FSE-5) were uniformly retrieved. If the "factual description / complete report" settings of the first stage (factual description-guided contrastive learning method) and the second stage (evidence-driven report generation method) are consistent, the text encoder is frozen during training. Experimental results show that each module contributes positively to the final performance, as detailed below:

[0157] The “BASE” model outperforms schemes (a) and (b), indicating that training the image-text alignment directly with the complete report is challenging, possibly due to “non-diagnostic language” in chest X-ray reports interfering with the cross-modal alignment process between factual descriptions and medical images.

[0158] Solution (e) significantly outperforms the “BASE” model in the Clinical Consistency (CE) index, verifying that the factual description-based comparative learning method proposed in this invention can effectively improve the model’s correspondence between medical images and clinical keywords, thereby enhancing the clinical effectiveness of the generated report.

[0159] FSE-5 outperforms schemes (c) and (d), indicating that using only L...global Or just introduce L local Neither loss function can achieve optimal results; however, combining the two can achieve more effective cross-modal semantic alignment.

[0160] Scheme (f) is slightly inferior to scheme (g) and FSE-5, indicating that simply splicing and integrating historical similar case information is insufficient to realize its full value. In contrast, the cross-modal fusion module (CMF) designed in this invention can more fully explore the key information in historical cases and improve the quality of report generation.

[0161] Compared with scheme (e), FSE-5 improved by 2.1%, 4.7% and 5.1% on the three key indicators of BLEU-2 (BL-2), F1-CheXbert5 (CX5) and F1-CheXbert14 (CX14), respectively, further demonstrating that the introduction of historical similar cases can provide empirical clinical clues for report generation and significantly improve the accuracy and credibility of the reports.

[0162] Combining both NLG and CE metrics, FSE-5 significantly outperforms schemes (a), (b), and (g), highlighting the core role of the factual descriptions extracted in this invention in comparative learning and evidence-driven report generation.

[0163] Table 4 Ablation experiment results on the MIMIC-CXR dataset.

[0164]

[0165] Terminology Explanation

[0166] In the first stage (a contrastive learning method guided by factual descriptions):

[0167] "F / R" indicates the text type used for contrastive learning, where "F" stands for Factual Serialization, which retains only the key medical facts from the chest X-ray report; and "R" stands for Report, which is the original text content without extraction.

[0168] L global This represents the instance-level alignment loss function guided by factual descriptions, as detailed in Equation (6).

[0169] L local The token-level alignment loss function is described in terms of factual descriptions. See formula (8) for details.

[0170] In the second phase (evidence-driven report generation method):

[0171] "F / R" indicates the presentation format of historically similar cases, where "F" means presented in a factual sequence format and "R" means presented in a complete report format.

[0172] "Fusion" refers to a method of fusing multi-source information. This method includes: "CMF" stands for Cross-Modal Fusion, which refers to deep semantic interaction between images and text through an attention mechanism; "Cat" stands for concatenation fusion, which refers to directly concatenating the embedding representations of different modalities along the feature dimension.

[0173] The best performing values ​​are marked in bold, and the second best values ​​are marked with an underline.

[0174] 2) Impact analysis of the number of historically similar cases:

[0175] To investigate the impact of the number of historically similar cases on report generation performance, a comparative experiment was conducted on the proposed FSE model using different numbers of historically similar cases (n∈{0,1,3,5,10,20}) on the MIMIC-CXR dataset. Model performance was evaluated from two dimensions: a specific clinical scenario (concise diagnosis) and a general clinical scenario (routine diagnosis). Detailed results are shown in Table 5. Experimental results indicate that introducing historically similar cases can provide relevant empirical clinical knowledge for the text generation process, effectively improving the medical accuracy and language quality of the report. Specifically: in a specific scenario (i.e., M... gt At a ratio of 100, FSE-1 improved the BLEU-1 (BL-1) metric by 3.5% compared to FSE-0 (which did not include historical cases) and the F1-CheXbert14 (CX14) metric by 4.3%, demonstrating the positive role of similar case information in improving clinical consistency. In general scenarios, FSE-1 also brought significant performance improvements, validating the versatility and adaptability of historical similar cases across multiple scenarios. Although FSE-1 performed well in traditional natural language generation metrics (such as BLEU, ROUGE-L) and F1-RadGraph (RG), FSE-5 achieved a more balanced performance between natural language quality and clinical consistency metrics, making it the best overall performance setting in the current experiments. It is noteworthy that when the number of historical similar cases further increased (such as FSE-10), the model performance declined. This may be due to: the model introducing redundant interference when processing a large amount of historical similar case information; and the increased number and complexity of model parameters leading to an increased risk of overfitting. In summary, the appropriate inclusion of historically similar cases plays an important role in improving the accuracy and fluency of report generation, but their number needs to be moderately controlled to avoid a negative performance decline.

[0176] Table 5. Impact of different historical similarity numbers on the MIMIC-CXR dataset on FSE performance.

[0177]

[0178] c. Qualitative Analysis Explanation

[0179] 1) Visual verification of similar historical cases:

[0180] To further verify the supporting role of historically similar cases in report generation, Figure 7 This study presents a visual comparison between two target samples in the MIMIC-CXR test set and their corresponding three historically similar cases. Regarding textual factual descriptions: the length of the color bars indicates the level of detail in the factual descriptions for each case, and the richness of entries is positively correlated with the breadth of clinical findings. Experimental observations revealed that key factual descriptions (such as lesion location, disease type, and pathological features) in historically similar cases highly match those in the target samples, providing authentic and effective diagnostic clues for the text generation process. This factual description-based alignment design maintains good clinical information matching capabilities even when there are significant differences in perspective between the target image and similar cases, further validating the significant advantages of the "fact-guided comparative learning method" in improving cross-modal retrieval accuracy.

Claims

1. A method for automatically generating chest X-ray reports based on factual description enhancement, characterized in that, Specifically, the following steps are included: Step 1: Construct a medical entity extraction method based on the RadGraph model to identify and extract structured clinical keywords, including disease, anatomical structure, and observation results, from chest X-ray reports. During the extraction process, redundant information or entities without diagnostic value are removed to obtain factual descriptions composed of key medical entities. The obtained factual descriptions serve as high-quality semantic supervision signals in cross-modal alignment. Step 2: Establish a factual description-guided comparative learning method to enhance the semantic consistency between the image and the chest X-ray report from two levels: (1) global level, that is, the alignment between the image and the corresponding overall factual description; (2) local level, that is, the alignment relationship between the local area in the image and the corresponding medical entity; train the image encoder to effectively learn the correspondence between the medical image and the factual description, reduce the interference of non-diagnostic language on visual semantic modeling, and thus extract visual features with diagnostic value. Step 3: Establish a historical similar case retrieval strategy that does not rely on disease labels. Without relying on disease labels, construct a visual representation space using the visual features extracted by the image encoder trained in Step 2. Automatic retrieval of historical medical records can be achieved by calculating visual semantic similarity; Step 4 proposes an evidence-driven method for generating chest X-ray reports. A cross-modal fusion network is constructed to effectively integrate the extracted visual features with the semantic information of chest X-ray reports corresponding to similar historical cases, generating a joint semantic representation, which is then used as the input to the text decoder. By providing cross-modal prior knowledge, the text decoder is guided to generate chest X-ray reports with clinical accuracy and consistency.

2. The method for automatically generating chest X-ray reports based on enhanced factual description as described in claim 1, characterized in that, The specific method for step 1 is as follows: 1.1 Entity Extraction and Screening: First, the chest X-ray report is input into the entity recognition module based on the RadGraph model to obtain an initial entity sequence containing clinical keywords; Subsequently, the extracted entity sequence is cleaned, specifically including: removing noisy entities that are not related to medicine; and when multiple medical entities exist at the same location, prioritizing the retention of key entities with higher information density. 1.2 Factual Subsequence Generation: After entity extraction and filtering in step 1.1, the retained clinical entities are sorted according to their textual order in the original chest X-ray report. Then, the entity sequence is divided into several factual subsequences using the sentence-end punctuation marks as delimiters. For subsequences containing entities labeled as "O-DA" or "OU" by the RadGraph model, the negative word "no" or the uncertain word "maybe" is added before the corresponding entity to enhance the completeness of semantic expression and the coherence of the context. 1.3 Factual Description Construction: The [SEP] subsequences are separated to construct the factual description of the i-th sample: in, This represents the number of factual subsequences in the i-th sample.

3. The method for automatically generating chest X-ray reports based on enhanced factual description as described in claim 1, characterized in that, The specific method for step 2 is as follows: 2.1 Image and Text Feature Extraction Process A pre-trained ResNet-101 model was selected as the image encoder, and visual features of the images were extracted based on the input set of chest X-rays X. Where X0 represents the global image features. For image encoders, B represents the batch size; a pre-trained six-layer SciBERT model is used as the text encoder to process factual descriptions S = {S...} 1 ,S 2 ,…,S B Encode the text to obtain the corresponding text features. Where, N R This represents the number of tokens in the factual description, where T0 is the global text feature. For a text encoder, two projection heads are introduced to project image features and text features onto the same dimension d, respectively. and To align image features and text features in the same spatial space; 2.2 Alignment loss functions are constructed at both the global and local levels to optimize the visual encoder, thereby further enhancing the semantic consistency between chest X-ray images and their corresponding factual descriptions: 2.2.1) Global-level image-text alignment loss function: First, we define an alignment loss function from the image to the text direction to ensure the semantic consistency between medical images and their factual descriptions in the embedding space, as follows: Where, sim(X0,T0)=(X0·T0) T ) / (||X0||·||T0||) represents the cosine similarity between the global visual representation X0 and the global textual representation T0, where τ1 is the temperature coefficient; this alignment loss function, Equation (4), encourages the global visual representation Matching global text representation Maximize the similarity between them, while suppressing the global text representation that does not match. The similarity between them; Similarly, the alignment loss function from text to image direction can be expressed as: The alignment loss function, Equation (5), encourages global text representation. Matching global visual representation Maximize the similarity between them, while suppressing the mismatch with global visual representations. The similarity between them; the alignment loss functions in the two directions mentioned above together constitute the global-level image-text alignment loss function guided by factual description, which is defined as follows: 2.2.2) Local-level image-text alignment loss function: First, local cross-modal text features are extracted using the cross-attention mechanism ATTN(Q,K,V). Among them, local text features As the query vector (Q), and the local visual feature X loc =[X1,…,X 49 As key vectors (K) and value vectors (V), the final output is local cross-modal text features. The calculation process is expressed as follows: Subsequently, the local textual features T in the chest X-ray report were analyzed. loc and local cross-modal text features Tokens in the same position are considered positive sample pairs, while combinations of other mismatched positions are considered negative sample pairs. Thus, the local-level image-text alignment loss function guided by factual description is defined as follows: Where, N R This indicates the number of tokens in the factual description, and τ2 is the temperature coefficient. Finally, by combining the global-level image-text alignment loss function (Equation (6)) and the local-level image-text alignment loss function (Equation (8), a total contrastive learning objective function guided by factual description is constructed, as follows:

4. The method for automatically generating chest X-ray reports based on enhanced factual description as described in claim 1, characterized in that, The specific method for step 3 is as follows: First, a visual representation space for the candidate set is constructed: the candidate set of historical similar cases for each sample to be retrieved is derived from image samples in the training set to prevent data leakage; assuming the training set contains n chest X-ray image samples, the ResNet-101 model pre-trained in step 2 is used as the image encoder. Visual features are extracted from these n chest X-ray images, and these visual features are placed in a set to construct a candidate set visual representation space H = {X0, X1, ..., X...}. n }; Then, for the target image x to be retrieved q Also using an image encoder Extract its visual features Next, cosine similarity is used to measure the target image X. q The visual semantic similarity between X and the candidate set samples is calculated after obtaining all similarities. q The k candidate samples with the highest similarity were selected as historical similar cases. The calculation process is expressed as follows:

5. The method for automatically generating chest X-ray reports based on enhanced factual description as described in claim 1, characterized in that, The specific method for step 4 is as follows: 4.1 Extracting Evidence Information Features: For the i-th sample, using the historical similar case retrieval strategy described in step 3, obtain historical similar cases consisting of k samples. Based on the RadGraph medical entity extraction method designed in step 1, historical similar cases are... Each historical chest X-ray report is converted into k factual descriptions; then, a text encoder with frozen parameters and a projection head are input to obtain evidence information features. Its processing can be represented as follows Where SE(·) represents the medical entity extraction method based on the RadGraph model; The total number of tokens in the evidence information features. Let be the number of tokens in the u-th historical chest X-ray report of the i-th sample; 4.2 Constructing a cross-modal fusion network: used to integrate the visual features X extracted in step 2. i Features of evidence information extracted from similar historical cases The Cross-modal Fusion Network (CMF) is integrated into a single network. It consists of three sub-modules: a self-attention sub-layer, a cross-attention sub-layer, and a feedforward neural network sub-layer. Its overall structure is as follows: Among them, W qs W ks W vs W qc W kc and W vc All are learnable parameters; LN(·) and FFN(·) represent layer normalization and feedforward neural network sublayers, respectively; 4.3 Generating Chest X-ray Reports: Integrating Cross-Modal Features The input text decoder uses an autoregressive approach to progressively generate chest X-ray reports. At time t, the decoder output depends on the previously generated token sequence. The training objective is to minimize the negative log-likelihood loss function, and its calculation process is as follows: Where M represents the maximum length of the chest X-ray report generated by the text decoder. This represents the sequence of words generated before time t.

6. The system for automatically generating chest X-ray reports based on factual description enhancement according to any one of claims 1 to 5, characterized in that, include: The medical entity extraction module is used in step 1 to identify and extract clinical keywords, including diseases, anatomical structures, and observation results, from chest X-ray reports based on the RadGraph model, thereby obtaining a factual description composed of clinical keywords. The factual description-guided contrastive learning module is used in step 2 to enhance the semantic consistency between images and text at two levels: global and local. It trains the image encoder to effectively learn the correspondence between medical images and factual descriptions, reduces the interference of non-diagnostic language on visual semantic modeling, and thus extracts visual features with diagnostic value. The historical similar case retrieval module, which does not rely on disease labels, is used in step 3 to construct a visual representation space using the visual features extracted by the image encoder trained in step 2 without relying on disease labels; and to realize the automatic retrieval of historical cases by calculating visual semantic similarity. Step 4, the evidence-driven chest X-ray report generation module, is used to effectively integrate the extracted visual features with the semantic information of chest X-ray reports corresponding to similar historical cases by constructing a cross-modal fusion network in step 4, generate a joint semantic representation, and use it as the input of the text decoder; By providing cross-modal prior knowledge, the text decoder is guided to generate chest X-ray reports with clinical accuracy and consistency.

7. The system for automatically generating chest X-ray reports based on factual description enhancement according to any one of claims 1 to 5, characterized in that, include: Memory, used to store computer programs; A processor is configured to implement the method for automatically generating chest X-ray reports based on factual descriptions when executing the computer program.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, is capable of automatically generating a chest radiograph report according to the method for automatically generating a chest radiograph report based on factual description enhancement as described in any one of claims 1 to 5.

Images