Chest X-ray image cross-modal analysis method based on visual thinking chain

Abstract

The invention discloses a chest X-ray image cross-modal analysis method based on a visual thinking chain. The chest X-ray image cross-modal analysis method comprises the following steps: S1, acquiring and preprocessing original data; the original data comprises a CXR image and a diagnosis report corresponding to the CXR image; s2, disassembling the diagnosis report into independent diagnosis statements; s3, annotation information is labeled on the independent diagnosis statements; s4, constructing a visual thinking chain generation model according to the decomposition step of generating the reasoning chain; s5, training a segmentation module and a text generation module of the visual thinking chain generation model; s6, establishing a unified modal representation structure to fuse the visual clues and the text clues; s7, finely adjusting the visual thinking chain generation model; and S8, generating a matrix table containing questions, reasoning steps and answers according to the trained visual thinking chain generation model. According to the method, the analysis process of the chest X-ray image is decomposed into a plurality of interpretable reasoning steps, and the analysis thought of a doctor is simulated by combining vision and text interaction, so that the interpretability and the accuracy of an analysis result are improved.

Claims

1. A cross-modal analysis method for chest X-ray images based on visual thinking chain, characterized in that: The following steps are involved: S1. Acquire and preprocess raw data; the raw data includes CXR images and diagnostic reports corresponding to the CXR images; S2. Decomposing the diagnostic report into independent diagnostic statements; S3. Annotate the independent diagnostic statement with annotation information; S4. Construct a visual thinking chain generation model based on the decomposition steps of generating a reasoning chain; S5, training the segmentation module and text generation module of the visual thinking chain generation model; S6. Establish a unified modality representation structure to integrate visual and textual cues; S7, fine-tuning the visual thinking chain generation model; S8. Generate a matrix table including the question, the reasoning steps and the answer based on the trained visual thinking chain generation model.

2. The chest X-ray image cross-modal analysis method based on visual thinking chain according to claim 1 is characterized in that: In step S2, the diagnosis report is disassembled using NLP tools according to the ABCDE rule.

3. The chest X-ray image cross-modal analysis method based on visual thinking chain according to claim 1 is characterized in that: In step S3, a computer vision annotation tool is used to annotate each independent diagnostic statement.

4. The chest X-ray image cross-modal analysis method based on visual thinking chain according to claim 1 is characterized in that: The decomposition steps of generating the reasoning chain include: S41, acquiring an image segmentation task based on the original data, and acquiring the visual clues through the image segmentation task; S42. Acquire text clues based on the visual clues; S43, obtaining multiple relevant visual clues based on the text clues; S44, matching the relevant visual clues and text clues to obtain a conclusive description; S45. Generate an explainable report.

5. The chest X-ray image cross-modal analysis method based on visual thinking chain according to claim 3 is characterized in that: The segmentation module includes a SAM model; the text generation module includes a Llama 3 model; and step S5 includes: S51, modifying the prompt encoder of the SAM model; removing the bounding box prompt and introducing the text category prompt; and iterating using the point sampling strategy; S52. The text generation module converts the current analysis step into a text description according to the label category; then inputs the text description, the original visual clues and text clues of the data, and the conclusive description into the large language model; and trains the large language model to understand the causal relationship between the annotation information in the current analysis step and the conclusive description.

6. The chest X-ray image cross-modal analysis method based on visual thinking chain according to claim 4 is characterized in that: The step S6 comprises: S61, sampling the visual cues using a VQ-GAN model, and obtaining a visual representation codebook through discrete quantization; S62, encoding and decoding the visual clues using an encoder and a decoder of a variational autoencoder; S63, reconstructing the CXR image by minimizing the loss function: S64: Merge the visual representation codebook with the text dictionary of the large language model to construct a modality-unified word segmenter; the word segmenter encodes the input visual clues or the input text clues into corresponding discrete tags.

Images