Pre-training method and system for chest x-ray image based on multi-modal multi-granularity features

Abstract

The present application relates to the field of medical imaging technology, and specifically relates to a chest X-ray image pre-training method and system based on multi-modal multi-granularity features. The method comprises: constructing a chest X-ray image dataset based on multi-modal multi-granularity features, the dataset including chest X-ray images and related radiology reports; using a pre-trained Faster R-CNN model to label the anatomical parts of the images, and structuring the radiology reports to associate the text information with the anatomical part labels; constructing a multi-modal multi-granularity feature extraction and fusion model, extracting coarse-grained and fine-grained features of images and texts respectively through Beit and Bert models based on Transformer, and fusing these features through a network based on a bidirectional attention mechanism; finally, training the model, using multiple loss functions to optimize the model performance, and performing system testing. The method aims to improve the automatic and intelligent judgment ability of chest X-ray images, and enhance the accuracy and efficiency of medical diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical imaging technology, in particular to a chest X-ray image pre-training method and system based on multi-modal multi-granularity features, which is mainly used to improve the automatic and intelligent judgment ability of chest X-ray images. BACKGROUND

[0002] Medical imaging technology, especially chest X-ray films, plays an important role in medical diagnosis, but the massive data makes further analysis and mining complex. With the use of multi-modal clinical information becoming the norm, combining patient demographic characteristics, main symptoms and laboratory examination reports for diagnosis has become a routine practice. However, this mode is limited by the limitations of professional knowledge and medical resources, and has caused a huge workload for doctors and radiologists.

[0003] Machine learning, especially deep learning, provides the possibility of automated and intelligent medical diagnosis. Although AI-based medical image diagnosis has made significant progress, how to integrate medical images with their clinical background knowledge is still a major challenge.

[0004] Current multi-modal clinical diagnosis systems mainly rely on non-unified ways to fuse information, and these methods are usually divided into early or late coarse-grained fusion. These methods have some problems, such as model training specific to modalities, which leads to the inability to encode inter-modal connections and associations, diagnosis-oriented fusion requires a large amount of labeled data and human cost, and coarse-grained feature alignment cannot meet the fine-grained needs of the medical imaging field.

[0005] Therefore, in order to reduce the dependence on labeled data and more effectively fuse multi-modal information, it is necessary to develop a more intelligent and more generalizable model. Current research is focusing on utilizing radiology reports of chest X-ray images, as these reports often contain rich domain knowledge. By utilizing detailed textual explanations provided by radiologists, it is possible to help achieve representation learning of medical images without relying on any manually labeled data. This approach not only improves the diagnostic accuracy of the model, but also significantly reduces the need for large amounts of labeled data, promoting more accurate and efficient medical diagnosis. These self-supervised learning methods through radiology reports have become a new training paradigm for chest X-ray films, which can automatically obtain supervision from free-text radiology reports.

[0006] Zhang et al. in the paper “Contrastive learning of medical visual representations from paired images and text” published in Machine Learning for Healthcare Conference. PMLR, 2022: 2-25 proposed ConVIRT to contrast the differences between the features of radiographs and the potential embeddings of sentences in radiology reports. By taking advantage of the naturally occurring pairing between image and text data, visual representations are learned. Specifically, ConVIRT improves visual representations by maximizing the consistency between real image-text pairs and random pairs through bidirectional contrast between image and text modalities. ConVIRT is applied to the pre-training of medical image encoders and demonstrates its ability to generate high-quality image feature representations that capture subtle visual features required for medical image understanding tasks. However, this method uses different model architectures for feature extraction of medical images and text, failing to fully embody the consistency of medical image features and text features. In addition, this method fails to effectively fuse the information of the two modalities of medical images and text, resulting in image features and text features failing to form complementary relationships, thus causing the features extracted by the model to fail to fully embody the advantages of multi-modal information. At the same time, this method uses coarse-grained feature alignment, comparing the overall image features with the text features for contrast learning. This approach can only extract the overall features of different modalities of data, and cannot align the details between different modalities, which does not meet the demand for fine-grained feature extraction in the field of medical images.

[0007] Zhou et al. (2022) proposed a Transformer-based self-supervised framework, REFERS, which performs autoregressive report modeling and research report matching. This framework uses a set of transformer-based backbone networks to extract features from chest X-ray images and report texts, respectively. Compared with previous work, this framework uses the same architecture to process multi-modal data, better reflecting the consistency of multi-modal features. Meanwhile, the mask-based language model is introduced in the research method, and the autoregressive report modeling method is used to extract text features in the report. On this basis, the MRM model proposed by Zhou et al. (2023) integrates the autoregressive modeling of X-ray images and the autoregressive report modeling of reports, complementing each other, so that the model better understands multi-modal features. Progress has been made on different tasks and datasets. Chen et al. (2023) proposed the idea of integrating mask self-supervised learning (MRM model) and contrastive self-supervised learning (ConVIRT model), with the two self-supervised learning methods complementing each other, and proposed the CMITM model, which achieved better results. This series of mask-based autoregressive modeling self-supervised learning frameworks uses coarse-grained feature alignment to align the overall image features and text features. This method can only extract overall features of different modal data, and cannot align the details between different modalities. These methods cannot meet the needs of medical image field for fine-grained feature alignment between different modalities.

[0008] In the related research of fine-grained feature extraction and alignment, Liao et al. (2021) proposed a method of maximizing the mutual information between image and text local features to realize representation learning. Mutual information quantifies the statistical correlation between two random variables. This method learns at the sentence and local image region level, which is more efficient than learning global descriptors, enabling the image encoder to extract useful feature representations for subsequent image analysis. However, similar to the problem faced by the ConVIRT model, this method fails to fully reflect the consistency of medical image features and text features, and does not effectively integrate the information of medical images and texts of two modalities. In addition, the fine-grained feature alignment method is not explicit alignment, and the fine-grained feature alignment effect is not analyzed in depth, resulting in poor model interpretability and inability to fully reflect the relationship between sentences and local image region fine-grained features.

[0009] In the study of Huang et al. (2021), the GLoRIA framework is proposed to jointly learn global and local representations of medical images by leveraging radiology reports. Specifically, the GLoRIA framework uses an attention mechanism to perform contrastive learning between sub-regions in paired chest X-ray images and words in reports, obtaining global-local representations of images. The framework learns attention weights of specific words in sub-regions of chest X-ray images to construct context-aware local image representations, and demonstrates good generalization ability on multiple datasets and different downstream tasks. However, the text fine-grained features used by this method are at the word level, and a single word cannot fully reflect the local region features in the image. In addition, this framework also has the problems of not embodying the consistency of multi-modal features and not performing multi-modal feature fusion. SUMMARY

[0010] The purpose of the present application is to overcome the problem of insufficient fusion information caused by the existing multi-modal clinical diagnosis system mainly relying on non-uniform multi-modal fusion mode, such as the connection and correlation between modalities cannot be encoded due to model training specific to modalities, the need for a large amount of labeled data and human cost for diagnosis-oriented fusion, and the inability of coarse-grained feature alignment to meet the fine-grained needs of the medical image field. The present application provides a chest X-ray image pre-training method and system based on multi-modal multi-granularity features, which develops a more intelligent and more generalizable model, reduces the dependence on labeled data, and more effectively fuses multi-modal information, thereby improving the accuracy and efficiency of medical diagnosis.

[0011] In view of the above, the present application provides a chest X-ray image pre-training method and system based on multi-modal multi-granularity features, which specifically comprises the following steps:

[0012] S1, constructing a chest X-ray image dataset based on multi-modal multi-granularity features, further comprising the following steps:

[0013] S11, obtaining a MIMIC-CXR dataset; the MIMIC-CXR dataset comprises chest X-ray images and radiology reports;

[0014] S121, analyzing the chest X-ray images using a pre-trained Faster R-CNN model, and labeling anatomical parts in the chest X-ray images to obtain anatomical part labels;

[0015] S122, structuring the radiology reports to obtain structured radiology reports; the structured radiology reports comprise the association between the text information of the radiology reports and the anatomical part labels;

[0016] S13, construct a chest X-ray image dataset based on multi-modal multi-granularity features; the chest X-ray image dataset based on multi-modal multi-granularity features specifically includes the chest X-ray image, the structured radiology report, and the anatomical site annotation; the chest X-ray image and the radiology report are coarse-grained features; and the anatomical site annotation and the correlation are fine-grained features;

[0017] S2, construct a multi-modal multi-granularity feature extraction and fusion model, which includes a multi-modal multi-granularity feature extraction sub-model and a multi-modal multi-granularity feature fusion sub-model; and the following steps are included:

[0018] S21, perform image preprocessing on the chest X-ray image to obtain an image input sequence;

[0019] S22, extract multi-modal multi-granularity features, specifically including: using a coarse-grained feature extractor based on a Beit model architecture of Transformer to extract coarse-grained features of the image input sequence; using a region feature extractor to extract fine-grained features of the image input sequence; and using a Bert model based on Transformer to extract coarse-grained features and fine-grained features of the text data;

[0020] S23, input the multi-modal multi-granularity features into a multi-modal multi-granularity feature fusion network based on a bidirectional attention mechanism, generate query, key, and value matrices for each modality, and calculate self-attention scores and cross-attention scores between modalities; and iteratively update the multi-modal multi-granularity features using the self-attention scores and the cross-attention scores;

[0021] S3, pre-train the multi-modal multi-granularity feature extraction and fusion model using multiple loss functions, and optimize the multi-modal multi-granularity feature extraction and fusion model parameters; the loss functions include a matching loss function, a contrastive loss function, a mask reconstruction loss, and a bounding box prediction loss;

[0022] The matching loss function is used to measure the correlation between different modal features;

[0023] The contrastive loss function is used to distinguish different categories or features;

[0024] The mask reconstruction loss is used to reconstruct the original data in the case that part of the information in the input data is masked;

[0025] The bounding box prediction loss is used for the target detection task;

[0026] S4, test and verify the chest X-ray image pre-training method of the multi-modal multi-granularity features.

[0027] Compared with the prior art, the present application has the beneficial effects of:

[0028] 1、The present application constructs a chest X-ray image dataset based on multi-modal multi-granularity features through an automated method, uses a pre-trained Faster R-CNN model to label the anatomical parts of the image, and simultaneously performs structured processing on the radiology report. This method can effectively integrate image and text data, label and utilize fine-grained features in image and text data through an automated method, not only reducing labor costs and improving processing speed, but also improving the accuracy and efficiency of medical diagnosis.

[0029] 2、The present application uses an advanced multi-modal multi-granularity feature extraction and fusion model. By using Beit and Bert models based on Transformer to extract coarse-grained and fine-grained features of images and texts respectively, and through a network based on a bidirectional attention mechanism to fuse features. Through this multi-granularity feature extraction and fusion, the present application can effectively process multi-modal medical image and text data, enhancing the generalization ability and interpretability of the model.

[0030] 3、The present application uses multiple loss functions to train the multi-modal multi-granularity feature extraction and fusion model. This enables the model to fully understand multi-modal multi-granularity data features, enhancing the generalization ability of the model and making it suitable for various downstream tasks such as disease classification, disease detection, and radiology report generation.

[0031] 4、The method and system of the present application aim to improve the automated and intelligent judgment ability of chest X-ray images in clinical diagnosis through deep learning and fusion of multi-modal multi-granularity features, which helps to improve the automation level of medical image analysis and reduce the workload of doctors. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Pre-training method flowchart for multi-modal multi-granularity feature chest X-ray image

[0033] Figure 2 Pre-training system architecture diagram for multi-modal multi-granularity feature chest X-ray image

[0034] Figure 3 Fine-grained labeling result diagram for chest X-ray image

[0035] Figure 4 Multi-modal multi-granularity feature extraction and fusion model schematic diagram

[0036] Figure 5 Multi-modal multi-granularity feature fusion network based on bidirectional attention mechanism schematic diagram

[0037] Figure 6 Multi-task test result comparison diagram

[0038] Figure 7 Generating a result image for text DETAILED DESCRIPTION

[0039] The application will be described in further detail below with reference to test examples and specific embodiments. However, this should not be understood as limiting the scope of the above-mentioned subject matter of the application to the following examples only, but any technology implemented on the basis of the content of the present application falls within the scope of the present application.

[0040] Example 1

[0041] Figure 1 The complete process of the pre-training method for chest X-ray images in this embodiment is shown from data set construction, feature extraction and fusion, to model training and testing, and the specific steps are as follows:

[0042] Step 101: Constructing a multi-modal and multi-granularity chest X-ray image data set

[0043] In this embodiment, the MIMIC-CXR data set is first obtained, which includes a large number of chest X-ray images and their corresponding radiology reports. The pre-trained Faster R-CNN model is used to analyze the chest X-ray images, automatically identify and label 29 key anatomical parts. The structured processing is performed on the radiology report to extract the text information describing each anatomical part in detail, and the text is associated with the corresponding anatomical part label. In addition, through sentence-level text analysis, it is ensured that each labeled anatomical part is accurately matched with the description in the report. In the constructed multi-modal and multi-granularity chest X-ray image data set, each data includes a chest X-ray image, a structured radiology report, and an anatomical part label.

[0044] Step 102: Constructing a multi-modal and multi-granularity feature extraction and fusion model

[0045] First, the chest X-ray images are preprocessed, including adjusting the images to 224x224 pixels and further dividing them into 14x14 image blocks of 16x16 pixels. These image blocks are mapped to a vector space through an embedding layer to form an image input sequence. The Beit model architecture based on Transformer is used to extract coarse-grained features from these image blocks, and a special region feature extractor is used to further extract fine-grained features from these coarse-grained features. For text data, the Bert model based on Transformer is used to extract coarse-grained and fine-grained features corresponding to the image modality. All these features are input into a fusion network based on a bidirectional attention mechanism to realize the deep integration of image and text modality features.

[0046] Step 103: Model training

[0047] The constructed multi-modal multi-granularity feature extraction and fusion model is used for model training. A matching loss function is used to optimize the correlation between different modal features, a contrast loss function is used to enhance the model's ability to distinguish different categories or features, a mask reconstruction loss is used to reconstruct the input data with some information masked, and a bounding box prediction loss is used to improve the performance of the target detection task. These loss functions work together to optimize the overall performance of the model, especially in terms of accuracy and robustness when dealing with multi-modal multi-granularity data.

[0048] Step 104: Test results

[0049] After training, the pre-trained model is tested using multiple public datasets to evaluate its performance in real-world applications. The test includes classification tasks, target detection tasks, and text generation tasks to comprehensively verify the model's multi-task processing ability. The test results show that the method not only improves the performance of automatic analysis of chest X-ray images, but also significantly improves the accuracy and efficiency of diagnosis.

[0050] Example 2

[0051] Example 2 demonstrates a specific application of the multi-modal multi-granularity feature-based chest X-ray image pre-training method.

[0052] Figure 2 The flowchart of the multi-modal multi-granularity feature-based chest X-ray image pre-training method shows the complete process from data acquisition to model training. Specifically, it includes acquiring chest X-ray images and related radiology reports from the MIMIC-CXR dataset, using a pre-trained Faster R-CNN model for automatic annotation of anatomical parts, and structuring the radiology report to extract text descriptions related to the image. In addition, the figure also describes the steps of extracting coarse-grained and fine-grained features of images and texts using Beit and Bert models based on Transformer, and performing feature fusion through a network based on bidirectional attention mechanism. Finally, the model training is optimized through multiple loss functions.

[0053] Figure 3 The fine-grained annotation result map of the chest X-ray image shows the results of fine-grained annotation of anatomical parts through the process described in Figure 2 This includes labeling key anatomical structures such as the left lung, right lung, and mediastinum, with each structure clearly identified by an accurate bounding box to facilitate the model's better understanding and recognition of specific features in the image.

[0054] Figure 2 and Figure 3 Together show the comprehensive effect of the multi-modal multi-granularity feature-based chest X-ray image pre-training method of this embodiment. Figure 2The complete training and data processing flow of the system is provided, and Figure 3 This flow is verified in practical application to accurately label and process chest X-ray images, thereby embodying the practical application value of the method in improving the accuracy and efficiency of medical image analysis. In this way, not only the automatic processing capability of chest X-ray images is significantly improved, but also more accurate and detailed support is provided for clinical diagnosis.

[0055] The specific steps of the chest X-ray image pre-training method based on multi-modal multi-granularity features adopted in this embodiment are as follows:

[0056] Step 101: Constructing a multi-modal multi-granularity chest X-ray image dataset

[0057] This embodiment constructs a multi-modal multi-granularity chest X-ray image dataset based on the MIMIC-CXR (Medical Information Mart for Intensive Care Chest X-ray) dataset. The MIMIC-CXR dataset provides a total of 377,110 chest X-ray images and their corresponding radiology reports in 217,013 independent radiology studies. In order to obtain the fine-grained features of these multi-modal data and the association between fine-grained features, this embodiment will label the 29 anatomical parts commonly found in chest X-ray images and analyze the sentences in the radiology reports related to the 29 anatomical parts.

[0058] In this embodiment, for each set of chest X-ray images and their corresponding radiology reports, the processing steps are divided into two parts. One part is to process the chest X-ray images, using a pre-trained Faster R-CNN (Region-based Convolutional Neural Network) model to analyze and label the anatomical parts in the chest X-ray images. The other part is to process the radiology reports to generate structured radiology reports. By analyzing the sentence-level text information, the described anatomical parts are determined, and these information is associated with the corresponding anatomical parts. The sentence-level text information is the text content in units of sentences.

[0059] Finally, a chest X-ray image dataset D with multi-modal multi-granularity feature labeling is formed. After removing abnormal cases where anatomical structures are not detected, a total of 190454 cases are included, each chest X-ray image and radiology report labeling D i can be expressed as:

[0060] D i ={I i ,T i ,Ri}

[0061] where I i represents the chest X-ray image of the i-th case. T i represents the structured radiology report of the i-th case. R i represents the set of anatomical site annotations of the i-th case, if the chest X-ray image of case i contains n anatomical sites, then R i is:

[0062] R i = {r i1 , r i1 , …, r in}

[0063] where r ij represents the j-th anatomical site annotation of the i-th case, the structure is:

[0064]

[0065] where N ij represents the name of the j-th annotated anatomical site. B represents the descriptive text for this anatomical site extracted from the structured radiology report. B ij represents a rectangular bounding box, represented by (x ij , y ij , w ij , h ij ), where x ij and y ij are the coordinates of the top-left corner of the bounding box, w ij is the width of the rectangular bounding box, and h ij is the height of the rectangular bounding box.

[0066] The specific steps for processing the chest X-ray images in the MIMIC-CXR dataset are as follows:

[0067] 1、Using Chest ImaGenome dataset, which contains 1571 manually annotated chest X-ray images. The annotation includes left lung, right lung, cardiac silhouette, mediastinum, left lower lung zone, right lower lung zone, right hilar structures, left hilar structures, upper mediastinum, left costophrenic angle, right costophrenic angle, left mid lung zone, right mid lung zone, aortic arch, right upper lung zone, left upper lung zone, right hemidiaphragm, right clavicle, left clavicle, left hemidiaphragm, right apical zone, trachea, left apical zone, carina, superior vena cava (SVC), right atrium, cavoatrial junction, abdomen, spine. A total of 29 rectangular bounding boxes of common anatomical regions in chest X-ray images are annotated in the form of (x, y, w, h), where x and y are the coordinates of the top-left corner of the bounding box, and w and h represent the width and height of the rectangular bounding box, respectively. A Faster-RCNN model is pre-trained using these annotations to detect bounding boxes in chest X-ray images.

[0068] 2、Using the pre-trained Faster-RCNN model in step 1 to process the frontal chest X-ray images in the MIMIC-CXR dataset to obtain preliminary rectangular bounding box annotations of anatomical regions.

[0069] 3. Based on the preliminary rectangular bounding box label obtained in step 2 and the frontal chest X-ray image, use the MedSAM (Segment Anything in Medical Images) model to further correct and label the common anatomical parts in the chest X-ray image. The specific steps are as follows: take the preliminary rectangular bounding box label as the region prompt of the MedSAM model, input the corresponding frontal chest X-ray image into the MedSAM model, and thus obtain more refined and accurate anatomical part segmentation region. Regenerate the rectangular bounding box label according to the anatomical part segmentation region output by the MedSAM model.

[0070] The specific steps of processing the radiology reports in the MIMIC-CXR dataset are as follows:

[0071] 1. Process the scene graph data of 500 artificially labeled radiology reports in the Chest ImaGenome dataset, and obtain a structured radiology report training set. The specific processing method is to structure the free text form of the radiology report. The structured radiology report is divided into three parts: reason (REASON), finding (FINDING), and diagnostic opinion (IMPRESSION). The reason (REASON) describes the patient's self-reported symptoms, medical history, etc. The finding (FINDING) describes the specific imaging findings observed in the chest X-ray image. The sentences in the finding (FINDING) part correspond to a set of anatomical parts, representing the anatomical parts described by the sentence. The diagnostic opinion (IMPRESSION) describes the summary opinion and clinical impression made by the radiologist. Let S = {s1, s2, s3, …, sn} be a radiology report in free text form containing n sentences, then the text format of the structured radiology report can be represented as: n

[0072] Structured report

[0073] = {REASON: {S REASON}, FINDING: {S FINDING : R}, IMPRESSION: {S IMPRESSION}}

[0074] Among them, is the set of sentences corresponding to the reason (REASON) part, is the set of sentences corresponding to the finding (FINDING) part, is the set of sentences corresponding to the diagnostic opinion (IMPRESSION) part, and R is the set of anatomical parts corresponding to the sentence in S FINDING .

[0075] ​2. Constructing large language model prompt words according to the text format of the structured radiology report in step 1. The prompt words contain three parts of description, 1. Task goal: generate structured radiology report text. 2. Task requirement: specific format of structured radiology report 3. Task description: meaning of each part of the structured radiology report.

[0076] 3. According to the text format of the structured radiology report and the 500 example structured radiology report training set, using the large language model prompt words, training the large language model, fine-tuning its parameters, and obtaining the fine-tuned large language model. This embodiment uses Meta's open source large language model Llama 3 for experiments.

[0077] 4. Using the large language model prompt words obtained in step 2 and the fine-tuned large language model Llama 3 obtained in step 3, structure the radiology reports in the MIMIC-CXR dataset. Obtain structured radiology reports and sentence-level fine-grained features and the association between common anatomical parts in chest X-ray images.

[0078] Step 102: Constructing a multi-modal multi-granularity feature extraction and fusion model

[0079] A multi-modal multi-granularity feature extraction and fusion model is constructed, which includes a multi-modal multi-granularity feature extraction network and a multi-modal multi-granularity feature fusion network based on a bidirectional attention mechanism. The overall structure of the model can be seen in Figure 4 , and the multi-modal multi-granularity feature fusion network based on a bidirectional attention mechanism can be seen in Figure 5 .

[0080] The multi-modal multi-granularity feature extraction network uses a group of models based on the Transformer framework to extract features of image modalities and text modalities respectively. For image data, a Beit model architecture based on Transformer is used to extract coarse-grained features, and a region feature extractor is used to extract fine-grained features. For the text modality, a Bert model based on Transformer is used to extract coarse-grained and fine-grained features corresponding to the image modality according to the structured radiology report generated in step 101.

[0081] The coarse-grained feature extractor for the image modality is composed of an image embedding layer and 12 stacked Transformer encoding modules, each including a multi-head self-attention layer, a normalization processing layer and a fully connected layer.

[0082] The region feature extractor is composed of an average pooling layer. It obtains the output vector of the coarse-grained feature extractor and generates region fine-grained features according to the chest X-ray image anatomy part rectangular bounding box label generated in step 101.

[0083] The multi-modal multi-granularity feature fusion network based on the bidirectional attention mechanism is stacked using multiple bidirectional multi-modal attention modules. The bidirectional multi-modal attention module contains a linear projection layer and a layer normalization operation to generate the query, key and value matrix of each modality, and calculate the self-attention score and cross-attention score of each modality, which are used to iteratively update and fuse multi-modal multi-granularity features. The specific structure is as shown in Figure 5 .

[0084] For the set of multi-modal multi-granularity feature annotation data generated according to step 101, first fill the chest X-ray image to a square, and then adjust the size to 224x224 pixels. The adjusted image is divided into 14x14 image blocks P = {p1, p1, p1, …, p N} of size 16x16. The image blocks P are mapped into a vector space through an embedding layer to obtain image block embeddings for training. In order to capture the relative position relationship between image blocks, a position embedding encoding is additionally added to each image block embedding. In addition, an embedding vector e cls is added to generate a global representation of the image. The final image input sequence E = {e cls , e1, e1, e1, …, e N} is constructed, where N is the number of image blocks.

[0085] The image input sequence is processed through 12 stacked Transformer encoding modules. The processing of each Transformer encoding module includes the following steps: first, the attention score is calculated through the multi-head self-attention mechanism, and the input sequence E is weighted using the attention score. Then, the weighted result is connected in residual with the original input sequence, and normalized. Next, the full connection layer is calculated and normalized again, and the residual connection is repeated 12 times, and finally the output sequence I = {i cls , i1, i2, i3, …, i N} is obtained, which represents the coarse-grained features of the chest X-ray image.

[0086] The region feature extractor processes the output sequence I = {i cls , i1, i2, i3, …, i N} and the anatomical region R = {r1, r2, r3, …, r M}, where M represents the number of region attributes of the chest X-ray image. For any anatomical region r m (m∈M), the image blocks P Rm are obtained according to its rectangular annotation box, where P Rm is a subset of image blocks P. The PRm Corresponding output sequence I Rm Input region feature extractor, obtain fine-grained features i corresponding to the anatomical part Rm The formula is as follows:

[0087] i Rm = avgpool(I Rm ),

[0088] Fine-grained image features corresponding to the anatomical part can be obtained

[0089] For the text attribute in the multi-modal multi-granularity feature annotation data, input it into the Bert model based on the Transformer to generate coarse-grained text features T = {t cls ,t1,t2,t3,…,t N}, and text features corresponding to the fine-grained features of the anatomical part

[0090] The obtained image modal and text modal fine-grained features and coarse-grained features are spliced to obtain image multi-granularity features and text multi-granularity features

[0091] The obtained multi-modal multi-granularity features are input into the multi-modal multi-granularity feature fusion network based on the bidirectional attention mechanism for feature updating and fusion. The process of generating query, key and value matrices for each modality by the bidirectional multi-modal attention module is as follows:

[0092]

[0093]

[0094] Wherein, Q, K, V represent the query, key, and value matrices in the attention mechanism, wherein, l represents the index of the current layer, LP(·) and Norm(·) represent linear projection and normalization operation respectively. The calculation formula of bidirectional attention is:

[0095]

[0096]

[0097]

[0098] Wherein, and represent the bidirectional attention scores of the image and text modalities respectively. and represent the self-attention scores of the image and text modalities respectively, and denote the cross-attention scores of image and text modalities respectively, d k denote the dimension of the feature, and λ is the cross-attention weight parameter, which is set to 1.0 by default.

[0099] The image and text modalities features of each layer are updated and fused using the bidirectional attention scores. Computing the self-attention scores enables the model to better understand and fuse the potential connections between coarse-grained and fine-grained features. The cross-attention scores enable better fusion of information between the image and text multi-modal features.

[0100] The above-mentioned and bidirectional attention scores are used to update and fuse the image and text modality features, as shown in the following formula:

[0101]

[0102] where FeedForward(·) represents a normalization layer and a multilayer perceptron (MLP).

[0103] This embodiment uses bidirectional attention to capture the internal coarse-grained and fine-grained feature correlations between image and text modalities. It also mines the cross-modal correlations between image and text.

[0104] Step 103: Model training

[0105] This embodiment uses multiple loss functions for pre-training, which respectively enhance the ability of the pre-trained model to handle different tasks, so that the multi-modal multi-grained feature extraction and fusion model can be applied to multiple downstream tasks. The loss functions include a matching loss function a contrast loss function a mask reconstruction loss a bounding box prediction loss

[0106] The matching loss function is used to measure the correlation between different modal features. By minimizing this loss, the model can better associate and fuse information from different modalities, thereby improving performance in multi-modal tasks. The contrast loss function is used to distinguish different classes or features. By maximizing the similarity between similar samples and minimizing the similarity between different samples, the discriminative ability of the model is enhanced, making the model applicable to retrieval, classification, and contrast learning tasks.

[0107] The mask reconstruction loss is used to reconstruct the mask from the image and text modalities, which can be used to evaluate the quality of the reconstructed mask and improve the performance of the model in tasks such as image and text captioning.

[0108] The mask reconstruction loss is used to reconstruct the mask from the image and text modalities, which can be used to evaluate the quality of the reconstructed mask and improve the performance of the model in tasks such as image and text captioning. ​For the case where part of the information in the input data is masked, the original data is reconstructed. By minimizing the loss, the model can better learn the potential structure and features of the data, improve the performance under incomplete data, and increase the performance of the generation task, making the model more outstanding in tasks such as text generation.

[0109] Boundary box prediction loss For the target detection task, by minimizing the difference between the predicted boundary box and the true boundary box, the accuracy of the model in image detection and positioning tasks is improved.

[0110] The parameter settings of the model are as follows: save a checkpoint every 50000 steps, and configure a high-frequency checkpoint saving strategy. The optimizer uses AdamW, the initial learning rate is set to 1e-4, the weight decay is 0.01, and the learning rate multiplier is 2. The learning rate scheduling strategy is linear scheduling, the learning rate is kept at 1e-4, the batch size is 64, the total training epoch number is set to 101, and the warm-up strategy is used for the first 2500 steps.

[0111] The acceleration configuration includes different step batch normalization (SyncBN), FP16 optimization level O1, dynamic FP16 loss scaling, random number seed 42, gradient accumulation step number 1, and gradient clipping threshold 1.0.

[0112] The text data processing configuration is set to 60 for the maximum number of words and the maximum number of tokens, the mask probability is 0.5, the maximum number of masks is 12, and the whole word mask is used. The skip word probability is 0.2, and the skip word size is 3.

[0113] Step 104: test results

[0114] The present embodiment tests the results of multiple tasks on multiple public data sets. Among them, including classification tasks, target detection tasks, text generation tasks, showing the good generalization ability and strong multi-task adaptation ability of the pre-training method and system proposed in the present embodiment.

[0115] The classification task is tested on the public datasets NIH ChestX-ray, CheXpert and RSNA. The NIH ChestX-ray dataset contains approximately 112,120 frontal chest X-ray images with multi-label annotations of 14 chest pathologies. The average AUC (Area Under Curve) achieved on NIH ChestX-ray is 86.3. It is higher than the 80.0 of the traditional ImageNet pre-training method, 83.2 of the MedKLIP model, 84.7 of the REFERS model, and 85.9 of the MRM model. The CheXpert is also a multi-label classification dataset, containing multi-label annotations of 5 chest pathologies. The average AUC achieved on CheXpert is 91.4. It is higher than the 87.3 of ConVIRT, 88.1 of the GLoRIA model, 88.2 of the REFERS model, and 88.7 of the MRM model. The RSNA dataset defines a binary classification problem, where each chest X-ray is classified as pneumonia or normal. The average AUC achieved on RSNA is 93.7. It is higher than the 81.3 of ConVIRT, 88.6 of the GLoRIA model, 92.7 of the REFERS model, and 93.3 of the MRM model.

[0116] The object detection task is tested on the SIIM-ACR Pneumothorax Segmentation (SIIM) dataset. This dataset is used to identify the segmentation of pneumothorax disease in chest X-rays. The Dice coefficient is used to measure the similarity between the predicted segmentation result and the true segmentation result. The pre-training method proposed in this embodiment achieves a Dice coefficient score of 91.8 on the SIIM dataset. It is higher than the 59.9 of ConVIRT, 63.4 of the GLoRIA model, 89.7 of the REFERS model, and 91.4 of the MRM model.

[0117] For the radiology report generation task, the FINDINGS section in the radiology report in the MIMIC-CXR dataset is used as reference. The text generation result is Figure 7 . The results show that the embodiment can effectively complete the text generation task.

[0118] Embodiment 3

[0119] This embodiment describes a system for chest X-ray image pre-training, which combines multi-modal (image and text) and multi-granularity (coarse-grained and fine-grained) feature processing techniques to improve the automation and accuracy of medical image analysis.

[0120] The system includes the following components:

[0121] Image Acquisition Device: High-resolution medical imaging device for acquiring chest X-ray images.

[0122] Data Processing Unit: High-performance server equipped with multi-core processors and multiple GPUs, dedicated for processing and analyzing image and text data.

[0123] Storage System: High-speed and large-capacity storage system for securely storing chest X-ray images and related radiology reports.

[0124] Display Unit: High-resolution display for presenting analysis results.

[0125] The system includes the following software and algorithms:

[0126] Image Preprocessing Module: Adjusts image size and contrast to optimize image quality.

[0127] Automatic Annotation Module: Uses Faster R-CNN algorithm to automatically identify and annotate anatomical parts in images. Uses large language model to structure radiology report text.

[0128] Natural Language Processing Module: Uses BERT algorithm for text extraction and structured processing to extract key information from radiology reports corresponding to images.

[0129] Feature Fusion Network: Transformer-based network for fusing image and text features, supporting bidirectional attention mechanism to enhance feature understanding.

[0130] Specific Implementation Steps:

[0131] S1, Data Acquisition and Input

[0132] Use the image acquisition unit to acquire the patient's chest X-ray image and import the corresponding radiology report.

[0133] S2, Data Preprocessing

[0134] Image data is preprocessed by the data processing unit, such as adjusting resolution, adjusting picture format, etc., and uses Faster R-CNN and MedSAM algorithms to extract fine-grained image feature annotations; text data is annotated and key information is extracted by a large language model.

[0135] S3, Feature Extraction and Fusion

[0136] Image Feature Extraction: Image processing module combines BEIT model and regional feature extractor to obtain coarse-grained and fine-grained image features.

[0137] Text feature extraction: The text processing module uses the BERT model to extract relevant text features from radiology reports.

[0138] The image and text features are input into a fusion network based on a bidirectional attention mechanism to achieve deep feature fusion.

[0139] S4, model training and optimization

[0140] A variety of loss functions are used in the data processing unit to train the feature fusion model, and a variety of loss functions are applied to optimize the model performance.

[0141] S5, result output and display

[0142] The processed image and text analysis results are presented on the display unit, providing intuitive diagnostic assistance information. Doctors can view detailed image annotations and text analysis results through the interactive interface, and make further diagnosis and treatment planning.

[0143] This embodiment describes in detail the composition and workflow of the chest X-ray image pre-training system based on multi-modal and multi-granularity features, emphasizing its potential in improving the automation and accuracy of medical image analysis.

[0144] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal multi-granularity feature-based chest X-ray image pre-training method, characterized in that, The method comprises the following steps: S1, constructing a chest X-ray image dataset based on multi-modal multi-granularity features, further comprising the following steps: S11, obtaining a MIMIC-CXR dataset; the MIMIC-CXR dataset comprises chest X-ray images and radiology reports; S121, analyzing the chest X-ray images using a pre-trained Faster R-CNN model and labeling anatomical parts in the chest X-ray images to obtain anatomical part labels; S122, structuring the radiology reports to obtain structured radiology reports; the structured radiology reports comprise the association between the text information of the radiology reports and the anatomical part labels; S13, constructing a chest X-ray image dataset based on multi-modal multi-granularity features; the chest X-ray image dataset based on multi-modal multi-granularity features specifically comprises the chest X-ray images, the structured radiology reports, and the anatomical part labels; the chest X-ray images and the radiology reports are coarse-grained features; the anatomical part labels and the association are fine-grained features; S2, constructing a multi-modal multi-granularity feature extraction and fusion model, the multi-modal multi-granularity feature extraction and fusion model comprising a multi-modal multi-granularity feature extraction sub-model and a multi-modal multi-granularity feature fusion sub-model; comprising the following steps: S21, performing image preprocessing on the chest X-ray images to obtain an image input sequence; S22, extracting multi-modal multi-granularity features, specifically comprising: using a coarse-grained feature extractor based on a Beit model architecture based on Transformer to extract coarse-grained features of the image input sequence; using a region feature extractor to extract fine-grained features of the image input sequence; using a Bert model based on Transformer to extract coarse-grained features and fine-grained features of the text data; S23, inputting the multi-modal multi-granularity features into a multi-modal multi-granularity feature fusion network based on a bidirectional attention mechanism to generate query, key and value matrices for each modality and calculate self-attention scores and cross-attention scores between modalities; iteratively updating the multi-modal multi-granularity features using the self-attention scores and the cross-attention scores; S3, pre-training the multi-modal multi-granularity feature extraction and fusion model using multiple loss functions to optimize the multi-modal multi-granularity feature extraction and fusion model parameters; the loss functions comprise a matching loss function, a contrastive loss function, a mask reconstruction loss, and a bounding box prediction loss; The matching loss function is used to measure the association between different modal features; The contrastive loss function is used to distinguish different categories or features; The mask reconstruction loss is used to reconstruct the original data in the case that part of the input data is masked; The bounding box prediction loss is used for object detection tasks; S4, testing and verifying the multi-modal multi-granularity chest X-ray image pre-training method.

2. The multi-modal multi-granularity feature-based chest X-ray image pre-training method according to claim 1, characterized in that, The anatomical sites are composed of left lung, right lung, cardiac silhouette, mediastinum, left lower lung zone, right lower lung zone, right hilar structures, left hilar structures, superior mediastinum, left costophrenic angle, right costophrenic angle, left middle lung zone, right middle lung zone, aortic arch, right upper lung zone, left upper lung zone, right diaphragm, right clavicle, left clavicle, left diaphragm, right apical region, trachea, left apical region, carina, superior vena cava, right atrium, vena cava-atrial junction, abdomen, and spine.

3. The multi-modal and multi-granularity feature-based chest X-ray image pre-training method according to claim 1, characterized in that, The sentence-level text information describes an anatomical site, and the sentence-level text information is associated with the anatomical site.

4. The multi-modal and multi-granularity feature-based chest X-ray image pre-training method according to claim 1, characterized in that, The step S121 further comprises the following steps: S1211, using a pre-trained Faster-RCNN model to process the frontal chest X-ray images in the MIMIC-CXR dataset to obtain preliminary rectangular bounding box annotations of the anatomical regions; S1212, based on the preliminary rectangular bounding box annotations and the frontal chest X-ray images, using a MedSAM model to further correct and annotate the anatomical sites in the chest X-ray images.

5. The multi-modal and multi-granularity feature-based chest X-ray image pre-training method according to claim 1, characterized in that, The step S122 further comprises the following steps: S1221, selecting part of the cases for manual annotation and structured processing to obtain training set and test set samples of the structured radiology report text, and the annotated text format is as follows: wherein, is a set of sentences corresponding to the REASON part, is a set of sentences corresponding to the FINDING part, is a set of sentences corresponding to the IMPRESSION part, is is the set of anatomical sites corresponding to the sentences in the S1222, constructing a large language model prompt word according to the text format; S1223, training the large language model using the large language model prompt word and the training set and test set samples, fine-tuning the parameters of the large language model, and obtaining the fine-tuned large language model; S1224, using the large language model prompt word and the fine-tuned large language model to structure the radiology reports in the MIMIC-CXR dataset, obtaining the structured radiology reports, and the association between the sentence-level fine-grained features and the anatomical sites.

6. The multi-modal and multi-granularity feature based chest X-ray image pre-training method according to claim 1, characterized in that, The image preprocessing in the step S21 comprises the following steps: S211, fill the chest X-ray image to a square, adjust to pixels, further divided into image blocks of size ​ S212, mapping the image blocks to a vector space through an embedding layer to obtain image block embeddings for training; S213, adding position embedding encoding in each image block embedding so that each image block contains position information; S214, adding a classification embedding vector to generate a global representation of the image, forming an image input sequence.

7. The multi-modal and multi-granularity feature based chest X-ray image pre-training method according to claim 1, characterized in that, In the step S23, the process of generating query, key and value matrices for each modality feature is as follows: wherein, is the layer of image multi-granularity features, is the layer of text multi-granularity features, respectively represent the query, key, value matrices of the image multi-granularity features of the layer in the attention mechanism, respectively represent the query, key, value matrices of the text multi-granularity features of the layer in the attention mechanism, and respectively represent the linear projection and normalization operations; The calculation formula of bidirectional attention is as follows: wherein, and denote the bi-directional attention scores for image and text modalities, respectively; and denote the self-attention scores for image and text modalities, respectively, and denote the cross-attention scores for image and text modalities, respectively, denotes the dimension of the features, and λ is a cross-attention weight parameter, set to 1.

0. Using the bidirectional attention scores of the image and text modalities, update and fuse the image modality and text modality features, and the formula is as follows: wherein, represents a normalization layer and a multi-layer perceptron.

8. The multi-modal and multi-granularity feature based chest X-ray image pre-training method of claim 1, wherein, The step S4 further comprises testing the results of multiple tasks using a public dataset to pre-train the multi-modal multi-granularity feature chest X-ray image method; the tasks include classification tasks, object detection tasks, and text generation tasks.

9. The pre-training method of claim 4, wherein, The multi-modal multi-granularity feature extraction sub-model further comprises: For image data, coarse-grained features are extracted using a coarse-grained feature extractor based on the Beit model architecture of Transformer, and fine-grained features are extracted using a region feature extractor; the coarse-grained feature extractor is composed of an image embedding layer and 12 stacked Transformer encoding modules, each of which includes a multi-head self-attention layer, a normalization processing layer and a fully connected layer; the region feature extractor is composed of an average pooling layer, which obtains the output vector of the coarse-grained feature extractor, and generates region fine-grained features according to the preliminary rectangular bounding box label; For text data, a Bert model based on Transformer is used to extract coarse-grained features and fine-grained features corresponding to image modalities according to the structured radiology report.

10. The multi-modal multi-granularity feature based chest X-ray image pre-training method of claim 4, wherein, Using the MedSAM model to further correct and label the chest X-ray image specifically includes: taking the preliminary rectangular bounding box label as the region prompt of the MedSAM model, inputting the corresponding front chest X-ray image into the MedSAM model, and then regenerating the rectangular bounding box label according to the anatomical part segmentation region output by the MedSAM model.

11. A system for chest X-ray image pre-training, characterized in that, The system is used to perform a chest X-ray image pre-training method based on multi-modal multi-granularity features according to any one of claims 1 to 10, comprising: Image acquisition device: configured with a high-resolution medical imaging device for acquiring chest X-ray images; Data processing unit: including at least one multi-core CPU and at least one GPU for processing image and text data; Storage system: a storage device with sufficient capacity for storing chest X-ray images and radiology report data; Display unit: at least one high-resolution display for displaying image analysis results; Image processing module: containing an automatic labeling module, using FasterR-CNN algorithm to automatically identify and label anatomical parts in the image; the image processing module further includes a MedSAM algorithm for extracting fine-grained features of the image; Text processing module: containing natural language processing function, using BERT algorithm to extract and structure text information in the radiology report; Feature fusion module: containing a network based on Transformer for fusing image and text features and supporting bidirectional attention mechanism; the feature fusion module further includes a plurality of loss functions, including matching loss, contrast loss and mask reconstruction loss, for optimizing the training process of the model.

Images