A method for automatically generating a diagnostic report with pixel-level localization evidence of an x-ray chest radiograph

Abstract

This invention proposes an automated diagnostic report generation method based on pixel-level localization evidence from chest X-rays, belonging to the fields of medical image processing and artificial intelligence. By constructing a neural network comprising multimodal feature extraction, a large language model, image segmentation, and collaborative output modules, this invention achieves pixel-level localization evidence for lesion text, enabling reliable annotation of lesion descriptions in the diagnostic report within the image. During the training phase, visual and text encoders extract image and text features respectively, the large language model generates a structured report, the image segmentation module generates pixel-level masks using language embedding as cues, and the collaborative output module establishes a correspondence between the report and the mask through special markers, achieving localizable joint prediction. This invention requires no manual prompts, supports fully automated analysis, and possesses good interpretability and clinical deployment adaptability, making it suitable for intelligent assisted diagnostic scenarios involving medical images such as chest X-rays.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and artificial intelligence technology, specifically relating to an automatic diagnostic report generation method based on pixel-level localization evidence from chest X-ray images, which is particularly suitable for automatic report generation of chest X-ray images and synchronous pixel-level localization of lesion areas. Background Technology

[0002] Medical image analysis is an important research direction in the field of computer-aided diagnosis. One of its core tasks is to automatically extract information about lesions, organs, or tissues from medical images, providing objective evidence for clinical diagnosis, treatment planning, and efficacy evaluation. Among various medical imaging modalities, chest X-rays have become the most commonly used imaging method for screening and diagnosing chest diseases due to their advantages such as low cost, low radiation, and ease of operation. However, chest X-rays are characterized by tissue overlap, low contrast, and diverse lesion morphologies, requiring a high level of experience from physicians in interpreting the images, leading to frequent misdiagnosis and missed diagnosis. Therefore, developing intelligent chest X-ray analysis methods has significant clinical application value.

[0003] In recent years, deep learning technology has made significant progress in the field of medical image analysis. In image segmentation, convolutional neural network structures, represented by U-Net and its variants, have achieved accurate extraction of organ boundaries and lesion regions through encoder-decoder architecture and skip connection mechanisms. Building on this, with the introduction of the Transformer architecture, models such as Swin-UNet and TransUNet have further improved global context modeling capabilities, achieving excellent performance in various medical image segmentation tasks. In particular, the emergence of the Segment Anything Model (SAM) and its subsequent version SAM2 has provided powerful foundational model support for general image segmentation. Through a cue learning mechanism, it can generate accurate segmentation masks based on user-provided points, bounding boxes, or text descriptions.

[0004] In the field of medical text generation, the development of multimodal large language models has opened up new avenues for medical image understanding. Models such as LLaVA-Med and BiomedGPT, through fine-tuning visual instructions on large-scale medical image-text pairs, have achieved tasks such as description generation, visual question answering, and report generation for medical images. These models can understand anatomical structures and abnormalities in images and output diagnostic descriptions in natural language, demonstrating powerful semantic understanding and generation capabilities.

[0005] While the aforementioned technologies have made achievements in both segmentation and text generation, existing methods generally suffer from a key drawback: segmentation and text generation tasks are independent, lacking a pixel-level text generation mechanism. Segmentation models, such as SAM and U-Net, can accurately locate lesion regions, but their output is limited to pixel-level masks and cannot generate natural language explanations. In clinical practice, doctors not only need to know "where" the lesion is, but also "what it is" and "what its characteristics are," which pure segmentation output cannot meet. Multimodal large language models, such as LLaVA-Med, can generate descriptive reports, but their output is limited to text and cannot annotate the image with pixel-level evidence. When a report mentions "a small nodule visible in the left upper lung field," doctors still need to manually locate the corresponding region in the image; this "text-image" separation reduces diagnostic efficiency and reliability. In actual clinical workflows, doctors typically need to obtain both lesion location and descriptive information simultaneously. To achieve both functions simultaneously using existing technologies, it is often necessary to call multiple independent models sequentially or rely on manual correlation of results, which not only increases system complexity but also reduces analysis efficiency and the consistency of results.

[0006] In summary, existing technologies still struggle to achieve high-precision report generation and segmentation localization simultaneously in the field of chest X-ray analysis, and lack specific optimizations for this modality. Therefore, there is an urgent need to propose an automatic diagnostic report generation method based on pixel-level localization evidence from chest X-rays. This method should integrate multimodal feature extraction, language model inference, and pixel-level segmentation localization to achieve simultaneous output of the diagnostic report and segmentation mask within a unified framework. Furthermore, specific optimizations should be performed for the characteristics of X-ray images to enhance the model's application value and deployment feasibility in clinical scenarios. Summary of the Invention

[0007] The purpose of this invention is to provide an automatic diagnostic report generation method based on pixel-level localization evidence from chest X-rays, aiming to solve the problems of segmentation and reporting separation in existing technologies for chest X-ray analysis and diagnosis, lack of image basis for lesion description in diagnostic reports, and high complexity of model deployment.

[0008] To achieve the above objectives, the present invention provides a method for automatically generating diagnostic reports based on pixel-level localization evidence from chest X-rays, comprising the following steps:

[0009] Step 1: Obtain chest X-ray data.

[0010] At least one chest X-ray image is acquired as input, the image being acquired by medical imaging equipment or from a publicly available medical image dataset. The chest X-ray image includes thoracic anatomical structures, including but not limited to the lungs, heart, ribs, diaphragm, and possible lesion areas, such as nodules, masses, consolidation, pleural effusion, and other abnormalities.

[0011] Step 2: Medical image preprocessing.

[0012] The input chest X-ray images are preprocessed, including size adjustment, intensity normalization, and optional data augmentation during the training phase, to eliminate differences in numerical distribution caused by different acquisition conditions and provide a unified data input format for subsequent model training and inference.

[0013] Step 3: Construct a diagnostic report generation network with pixel-level localization evidence.

[0014] A neural network for automatically generating diagnostic reports with end-to-end pixel-level localization evidence is constructed, wherein the neural network includes at least the following modules:

[0015] Multimodal feature extraction module: A pre-trained visual encoder is used to extract visual features from the input X-ray chest film to obtain high-quality feature representations containing spatial and semantic information; a pre-trained language encoder is used to encode user instructions or prompt text, converting the input text into a sequence of word vectors for recognition by a large language model.

[0016] The large language model module fuses visual features from the projection layer with textual instructions and then inputs this fusion into a pre-trained large language model. The large language model generates diagnostic report text using an autoregressive approach. The diagnostic report includes information such as lesion type, location, morphology, size, boundary features, and clinical recommendations.

[0017] Image segmentation module: Using the language output features of the large language model module as the prompt input for the segmentation model, the module generates pixel-level segmentation masks for the corresponding lesion areas of the original chest X-ray.

[0018] Collaborative Output Module: This module synchronously outputs the diagnostic report generated by the large language model module and the segmentation mask generated by the image segmentation module, forming a localizable joint prediction result of "text + mask". The collaborative output module establishes a correspondence between the report text and the segmentation mask using a special marker [SEG]: when the [SEG] marker appears in the diagnostic report, it indicates that the text description before or after the marker corresponds to the currently output segmentation mask. In this way, pixel-level localization and traceability of the report content are achieved.

[0019] Step 4: Train the neural network.

[0020] The neural network is trained using triplet data containing images, diagnostic reports, and segmentation annotations. The original chest X-ray is input, and the data passes through a multimodal feature extraction module, a large language model module, an image segmentation module, and a collaborative output module to obtain the corresponding medical report and segmentation mask. The training process jointly optimizes the text generation loss and the segmentation loss, achieving collaborative learning for both report generation and segmentation tasks.

[0021] Step 5: Repeat step 4 until the training error of the neural network converges to below a preset threshold or the number of training rounds reaches a preset upper limit; save the trained network parameters as a pre-trained model file.

[0022] Step Six: Load the pre-trained model and perform the same preprocessing operations as in Step Two on the new chest X-ray; input the preprocessed medical image into the neural network, and output the corresponding diagnostic report and segmentation mask end-to-end. The diagnostic report describes the lesion features in the image in natural language, and the segmentation mask locates the corresponding region on the image using pixel-level annotations.

[0023] Beneficial effects

[0024] Compared with existing technologies, this invention has the following advantages: This invention eliminates the need for manual segmentation prompts (such as points or boxes) or pre-diagnostic information. After inputting the original chest X-ray image, the model automatically completes the entire process of lesion detection, report generation, and region segmentation. Compared to existing technologies that require manual pre-diagnosis or manual annotation, this invention achieves truly fully automated analysis, reducing the need for manual intervention and improving the efficiency of clinical workflows. The diagnostic report and segmentation mask output by this invention correspond to and complement each other, providing clinicians with rich decision support information. The segmentation mask intuitively displays the spatial distribution of lesions, while the diagnostic report details the morphological characteristics and clinical significance of the lesions. The lesion description text in the diagnostic report can display corresponding prompts in the image. The combination of these two aspects makes the model's judgment basis more transparent and traceable. This feature is of great significance for applications with high reliability requirements in medical scenarios, helping to increase doctors' trust and acceptance of AI-assisted diagnostic systems. Attached Figure Description

[0025] Figure 1 Flowchart of an automatic diagnostic report generation method based on pixel-level localization evidence from chest X-ray

[0026] Figure 2 A schematic diagram of an automatic diagnostic report generation method based on pixel-level localization evidence from chest X-rays. Detailed Implementation

[0027] To more clearly illustrate the method for automatically generating diagnostic reports based on pixel-level localization evidence from chest X-rays according to the present invention, the implementation of the present invention will be described in detail below with reference to specific embodiments. These embodiments provide specific implementation methods and operating procedures without departing from the technical solution of the present invention, but the scope of protection of the present invention is not limited to the following embodiments.

[0028] This example proposes an automatic diagnostic report generation method based on pixel-level localization evidence from chest X-rays. This method includes, but is not limited to, the following functional modules: chest X-ray acquisition module, image preprocessing module, multimodal feature extraction module, large language model module, image segmentation module, and collaborative output module.

[0029] Step 1: Acquire at least one 2D chest X-ray as the input image. The chest X-ray can be derived from data acquired by medical imaging equipment or from a pre-constructed medical imaging dataset. The chest X-ray includes thoracic anatomical structures, including but not limited to the lungs, heart, ribs, diaphragm, and potential lesion areas. These lesion areas include abnormalities such as nodules, masses, consolidation, and pleural effusion.

[0030] Step 2: Perform preprocessing on the input chest X-ray to reduce the impact of different acquisition conditions on the generated report and segmentation results.

[0031] Specifically, the preprocessing process includes the following steps: First, the X-ray chest film is resized or resampled to ensure that the image size meets the input requirements of the neural network; second, the pixel intensity of the image is normalized to map the image grayscale value to a preset numerical range to unify the numerical distribution range of different images; during the training phase, data augmentation operations can be further introduced, such as random cropping, random rotation, and introducing Gaussian noise, to enhance the model's adaptability to different imaging conditions.

[0032] Step 3: Input the medical image processed in Step 2 into the deep learning model. The neural network mainly includes a multimodal feature extraction module, a large language model module, an image segmentation module, and a collaborative output module, as detailed below:

[0033] Multimodal feature extraction module: The multimodal feature extraction module is used to extract high-quality image features from input medical images and map user text commands into corresponding word vector sequences.

[0034] The multimodal feature extraction module is divided into a visual encoder and a text encoder. The visual encoder uses a pre-trained Vision Transformer (ViT) structure as the visual backbone network, specifically employing the rad-dino model. This encoder, trained on top of DINOv2, is specifically designed for processing chest X-rays, combining lightweight design with powerful image feature extraction capabilities. The input image is first divided into non-overlapping image patches, and the corresponding visual features are output through the pre-defined rad-dino visual encoder. The text encoder encodes user commands or text prompts into word vector sequences that can be recognized by large language models, facilitating text comprehension and report generation.

[0035] Large Language Model Module: After receiving visual features and text instructions, the large language model generates a structured diagnostic report and embeds the output language into the image segmentation module as a prompt.

[0036] Specifically, visual features are mapped to the input space of the language model through a projection layer (MLP) to generate a sequence of visual features. These visual features are then concatenated with text vectors along the sequence dimension and input into the pre-trained large language model backbone. A diagnostic report text is generated through an autoregressive approach. The diagnostic report includes information such as lesion type, location, morphology, size, boundary features, and clinical recommendations. The large language model used is Vicuna-7B. This model has a moderate number of parameters, high inference efficiency, and is easily integrated with visual models, making it suitable for deployment in common clinical computing environments.

[0037] Image segmentation module: The image segmentation module is based on the Segment Anything Model 2 (SAM2) structure. It uses the language embedding output of the large language model module as a prompt to decode medical X-ray chest films and generate high-quality pixel-level masks of the corresponding lesions.

[0038] Specifically, the language embeddings output from the last layer of the large language model are used as the cue inputs for the segmentation model. SAM2 includes the following sub-modules: an image encoder, which extracts multi-scale features from the input X-ray chest image based on the Hiera hierarchical visual Transformer structure to generate corresponding image embedding features; a cue encoder, which maps the language embedding cue output from the large language model to the semantic space of the cue encoder after being projected by a projection layer, and maps the processed language embedding cue into the corresponding cue embedding vector; and a mask decoder, which outputs the image embedding and cue embedding to the mask decoder and generates a pixel-level segmentation mask through multi-head self-attention, cross-attention mechanisms and convolution operations.

[0039] Collaborative Output Module: This module synchronously outputs the diagnostic report generated by the large language model module and the segmentation mask generated by the segmentation module, forming a localizable joint prediction result of "text + mask". The collaborative output module establishes a correspondence between the report text and the segmentation mask using a special marker [SEG]: when the [SEG] marker appears in the diagnostic report, it indicates that the text description before or after the marker corresponds to the currently output segmentation mask. In this way, pixel-level localization and traceability of the report content are achieved.

[0040] Step 4: Train the neural network using triplet data containing images, diagnostic reports, and segmentation annotations. The training process jointly optimizes the text generation loss and segmentation loss, achieving collaborative learning for both report generation and segmentation tasks. The specific process is as follows:

[0041] First, the preprocessed chest X-ray from step two is input into a neural network to obtain a diagnostic report through a multimodal feature extraction module and a large language model module. Then, the language embedding from the last layer of the large language model is input into the image segmentation module as a prompt, outputting the corresponding segmentation prediction result. Next, the errors between the generated diagnostic report and the actual report, and between the segmentation prediction result and the actual segmentation annotation are calculated using the text generation loss function and the segmentation loss function, respectively. The total loss function is defined as the weighted sum of the text generation loss and the segmentation loss.

[0042]

[0043] in The segmentation loss weighting coefficient is set to 1.0 in this invention.

[0044] (1) Text generation loss: The autoregressive cross-entropy loss function is used, defined as:

[0045]

[0046] Where L is the text length of the generated report. Let I be the true label of the t-th word, P be the model's predicted probability, I be the input image, and T be the input text.

[0047] (2) Segmentation loss: A combination of Dice loss and binary cross-entropy loss is used, defined as:

[0048]

[0049] in For Dice's loss, For binary cross-entropy loss, and These are weighting coefficients, which are set to 2.0 and 0.5 respectively in this invention.

[0050] Dice loss is defined as:

[0051]

[0052] The binary cross-entropy loss is defined as:

[0053]

[0054] in Total number of pixels Let be the predicted probability of the i-th pixel. For real labels, This is a smoothing term.

[0055] Finally, the visual encoders of the multimodal feature extraction module and the image segmentation module are frozen. LoRa fine-tuning is performed on the large language model, and the cue encoder and mask decoder of the image segmentation module are also fine-tuned. Based on the error loss value, the AdamW optimizer is used to optimize the parameters. By using gradient information, the network parameters are gradually adjusted to minimize the loss function value, thereby progressively improving the model's segmentation accuracy and stability.

[0056] Step 5: Repeat step 4 until the training error of the neural network converges to below the preset threshold or the number of training iterations reaches the preset upper limit; save the trained neural network parameters as a pre-trained model file.

[0057] Step 6: During the inference phase, the saved pre-trained model parameters are loaded, and the new chest X-ray undergoes the same preprocessing operations as in Step 2. The preprocessed medical image is then input into the neural network, and the corresponding medical diagnostic report and image segmentation results are output end-to-end. The diagnostic report describes the lesion features in the image in natural language, and the segmentation mask locates the corresponding region on the image using pixel-level annotations. This implementation achieves fully automated processing from inputting the medical image to report generation and segmentation mask output.

[0058] Through the above specific implementation methods, the present invention realizes an automatic generation method for diagnostic reports based on pixel-level localization of X-ray chest images. It achieves the generation of diagnostic reports based on pixel-level localization of X-ray chest images, and has made significant technical progress in terms of segmentation accuracy, report quality, model interpretability, automation level and deployment adaptability, and has broad clinical application prospects.

[0059] The above description is only a preferred embodiment of the present invention. Any equivalent transformations or substitutions made within the scope of the technical concept and claims of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically generating diagnostic reports based on pixel-level localization evidence from chest X-rays, characterized in that, Includes the following steps: Step 1: Obtain at least one chest X-ray image as the input image; The chest X-ray images are obtained from medical imaging equipment or publicly available medical image datasets, and include chest anatomy and possible lesion areas. The chest anatomy includes the lungs, heart, ribs, and diaphragm, and the lesion areas include nodules, masses, consolidation, and pleural effusion. Step 2: Perform preprocessing operations on the input chest X-ray image; The preprocessing includes size adjustment, intensity normalization, and optional data augmentation during the training phase to eliminate differences in numerical distribution caused by different acquisition conditions. Step 3: Construct an end-to-end diagnostic report generation network with pixel-level localization evidence. The neural network includes a multimodal feature extraction module, a large language model module, an image segmentation module, and a collaborative output module. Step 4: Train the neural network using triplet data containing images, diagnostic reports, and segmentation annotations, and jointly optimize the text generation loss and segmentation loss; Step 5: Repeat step 4 until the training error of the neural network converges to below the preset threshold or the training rounds reach the preset upper limit. Save the trained network parameters as a pre-trained model file. Step 6: Load the pre-trained model, perform the same preprocessing operation as in Step 2 on the new chest X-ray, input the preprocessed medical image into the neural network, and output the corresponding diagnostic report and segmentation mask end-to-end.

2. The method according to claim 1, characterized in that, The multimodal feature extraction module includes: a visual encoder for extracting visual features from the input chest X-ray; and a text encoder for encoding user instructions or prompts into a sequence of word vectors.

3. The method according to claim 1, characterized in that, The large language model module receives the concatenated visual features and text word vectors, generates a structured diagnostic report through autoregression, and transmits the output language embedding to the image segmentation module as a prompt.

4. The method according to claim 1, characterized in that, The image segmentation module uses the language embedding output by the large language model module as a prompt to decode the X-ray chest film and generate a pixel-level segmentation mask.

5. The method according to claim 1, characterized in that, The collaborative output module outputs the diagnostic report and the segmentation mask synchronously to form a localizable joint prediction result of "text + mask"; the collaborative output module establishes the correspondence between the report text and the segmentation mask through special markers.

6. The method according to claim 1, characterized in that, In step four, the total loss function is defined as the weighted sum of the text generation loss and the segmentation loss: in The segmentation loss weight coefficient is set to 1.0 in this invention. The text generation loss adopts the autoregressive cross-entropy loss function: Where L is the text length of the generated report. Let I be the true label of the t-th word, P be the model's predicted probability, I be the input image, and T be the input text. The segmentation loss uses a combination of Dice loss and binary cross-entropy loss. in For Dice's loss, For binary cross-entropy loss, and These are weighting coefficients, which are set to 2.0 and 0.5 respectively in this invention. Dice loss is defined as: The binary cross-entropy loss is defined as: in Total number of pixels Let be the predicted probability of the i-th pixel. For real labels, This is a smoothing term.

Images