Anatomical region guided medical vision-language pre-training system

Abstract

The invention belongs to the technical field of medical image processing, and particularly relates to a medical vision-language pre-training system guided by an anatomical area. The system comprises an anatomical region and report clause alignment module, an internal representation learning reinforcement module and an external representation learning reinforcement module. The alignment module is composed of a target detector, a text parser and an automatic alignment normal form between the target detector and the text parser, and the model is endowed with fine-grained alignment capability through comparative learning; the internal characterization learning reinforcement module uses an image encoder and an image label identification decoder to associate image features with corresponding labels, and provides a function of open disease category detection; the external representation learning reinforcement module utilizes comparative learning of a soft label and a hard label to improve relevance between different image-report pairs. According to the method, the anatomical region is taken as the center, and meanwhile, the fine-grained semantic association of the image-report to the interior and the exterior is enhanced, so that the downstream task accuracy based on the current pre-training system can be remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of medical image processing, and in particular relates to an anatomical region guided medical vision-language pre-training system. Background Art

[0002] In recent years, vision-language pre-trained models have achieved remarkable success. These models are trained using millions of web image-text pairs and learn representations by matching images with their corresponding captions without the need for time-consuming and laborious manual labeling. In the medical field, this model is also gaining more and more attention. Through the joint learning method of medical images and text, the representations of medical images and reports are directly learned from paired radiology reports, and the learned representations are transferred to downstream tasks.

[0003] Regarding the medical visual language pre-training model, on the one hand, considering that the pathology usually occupies a small part of the entire image, some researchers have tried to introduce different contrastive learning strategies to learn local representations, capture fine-grained semantics in medical images, and facilitate the localization of subsequent downstream tasks, such as medical target detection, medical semantic segmentation, and fine-grained medical question answering. When performing local alignment, most researchers use the image block-word alignment method. However, the image blocks directly extracted from medical images can neither represent the lesion area nor the anatomical organ; in addition, a single independent word will break up the lesion and the location description related to the lesion, and such an alignment method lacks clinical interpretability. On the other hand, considering that medical reports are more professional and abstract than the description sentences of natural images, some researchers have explored how to parse medical reports so that the model can understand and model complex medical professional terms and capture key information related to the lesion. Some researchers use natural language analysis tools to decouple medical reports. However, after obtaining the structured medical reports, they still simply apply text encoders to directly extract features, failing to fully utilize their rich information to guide the model's understanding of the current image.

[0004] There are two main deficiencies in existing work: (1) Lack of interpretability and clinical relevance. Previous methods mainly performed local alignment of images and text at the level of "image blocks-words", which has low clinical credibility. For example, the image blocks extracted from chest X-rays can neither represent the lesion area nor the anatomical organs. In addition, decomposing the entire sentence into individual words will lead to a large loss of semantic context. For example, if the original report is "pleural effusion in the left lower lung", separating "left lower lung" from "pleural effusion" will lead to a large loss of semantics in the sentence: it is impossible to know the specific location of "pleural effusion" or the specific condition of "left lower lung". Therefore, the practice of aligning the two further reduces interpretability and even interferes with model optimization. (2) Insufficient representation learning of image-report pairs. Compared with natural image descriptions, medical reports are usually longer and contain richer medical knowledge, which brings additional challenges to fully understanding the reports. However, most previous methods still rely on simple encoding of the original report to extract text features, which cannot fully understand the entire report and capture the key parts describing the lesion. In addition, the semantic associations between different image-text pairs are ignored, and hard labels are used in contrastive learning, resulting in numerous false negative samples. Summary of the invention

[0005] The purpose of the present invention is to provide an anatomical region-guided medical vision-language pre-training system that can significantly improve the model representation capability and the accuracy of downstream tasks, so as to address the shortcomings of the prior art.

[0006] The anatomical region-guided medical vision-language pre-training system provided by the present invention learns medical multimodal representations from paired radiology reports through a medical image-text joint learning method. It includes: an anatomical region and report sentence alignment module, an internal representation learning reinforcement module, and an external representation learning reinforcement module; the anatomical region and report sentence alignment module is composed of a target detector, a text parser, and an automatic alignment paradigm between the two; the target detector is used to locate the anatomical region in the image, and the text parser splits the entire report into clauses related to the anatomical region description, and structures them into the form of a triple <anatomical region, imaging observation, whether it exists>, and on the basis of the two, it follows the alignment rules formulated by professional radiologists to form an automatic alignment paradigm for anatomical regions and report sentences, and uses contrastive learning to give the model fine-grained alignment capabilities; the internal representation learning reinforcement module acts on the image-text pair, uses the image encoder and the image label recognition decoder to associate the image features with their corresponding labels, and provides the function of open disease category detection; the external representation learning reinforcement module acts between multiple image-report pairs, and uses contrastive learning based on both soft labels and hard labels to improve the correlation between different image-report pairs. The present invention focuses on anatomical regions and strengthens the fine-grained semantic associations inside and outside the image-text pairs, which can significantly improve the accuracy of downstream tasks based on the current pre-training system.

[0007] (I) The anatomical region and report sentence alignment module includes a target detector, a text parser, and an automatic alignment paradigm between the two. The specific process is as follows:

[0008] Use the pre-trained object detector Faster R-CNN [1] to extract the i M A The bounding box (bbox) of the anatomical region A i , expressed as:

[0009]

[0010] In the formula, represents the bounding box of each anatomical region, The coordinates of each bbox; The corresponding anatomical category (such as right hilum); C pre For all predefined categories.

[0011] The complete report R is converted into i Decoupling to M T triples, represented as:

[0012]

[0013] Among them, each triple It is defined as <anatomical region, imaging observation, presence or absence> (e.g. <lung, pneumothorax, presence>), where the anatomical region in the triple belongs to a predefined set C ana .

[0014] Due to the predefined categories of anatomical regions in images and reports C pre and C ana The number is not uniform, and there is semantic overlap (such as C ana The definition of "lungs" also includes C pre In the example above, we propose an alignment paradigm between anatomical regions and report sentences under the guidance of professional radiologists: Given a predefined category a∈C of a bounding box pre , and a predefined category of anatomical regions in a triplet pos∈C ana , consider the following three scenarios:

[0015] Case 1: pos and a are exactly the same, or there is only a slight difference in the way they are expressed. For example, pos is "left hilum" and a is "left hilum structure", and the two can be paired directly.

[0016] Case 2: If case 1 is not satisfied, pair pos with the smallest a that can cover it. For example, if pos is "right atrium", the smallest a that can cover it is "heart contour". Both cases 1 and 2 can be formalized as:

[0017]

[0018] Case 3: If there is a one-to-many relationship between pos and a, two solutions are provided: merge the anatomical region bounding boxes or further split the clauses:

[0019]

[0020] Merge anatomical region bounding boxes: Merge two / multiple bounding boxes into a whole region and pair it with the current clause For example, the bounding box of the left diaphragm surface and the bounding box of the right diaphragm surface are merged into the complete diaphragm surface area; assuming that the coordinates of the two are and The combined coordinates are expressed as:

[0021]

[0022] Split clause: Split the current clause into smaller units, pairing them with two or more bounding boxes. and For example, the “diaphragm surface” is split into the “left diaphragm surface” and the “right diaphragm surface” and paired with their corresponding anatomical bounding boxes respectively.

[0023] After obtaining the local anatomical region-sentence pair for each sample, the fine-grained contrastive learning loss L is calculated arsa , to maximize the cosine similarity between pairs of anatomical regions and report clauses:

[0024]

[0025] Where H(·) represents the cross entropy; Indicates the current clause; The cosine similarity vector representing the current anatomical region and each sentence in the entire report; Represents the cosine similarity vector between the current sentence and each anatomical region in the image; One-hot encoded label vector representing anatomical region and clause level, y ii =1 (the similarity between the current anatomical region and its corresponding report clause is 1), and the other elements are 0.

[0026] In the present invention, the internal representation learning reinforcement module and the external representation learning reinforcement module make full use of the "imaging observation" and "whether there is" information in the triplet, and simultaneously strengthen the representation learning within a single image-text pair and between multiple image-text pairs.

[0027] (ii) The internal representation learning reinforcement module constructs a label for each image using “radiological observation” and “existence” information Among them, M Q is a predefined disease category. If the current image has disease category j, then otherwise, Set the current image I i As input, after the image encoder f enc Then get the visual features in is the feature of each image block in the feature sequence, M Z is the sequence length. i As a self-attention based image label recognition decoder f dec The key vector (Key) and value vector (Value) of As the query vector (Query), where q j For a single disease category in the set, the predicted category of the current image is obtained through decoding by the self-attention mechanism, and the binary cross entropy loss is used as the loss function:

[0028]

[0029] Where N represents the total number of image report pairs in the current dataset.

[0030] Image label recognition decoder based on self-attention dec Compared with the linear classification layer, it can provide the function of open disease category detection, such as encountering a new disease category q new , add it to the predefined disease category set Q.

[0031] (III) The external representation learning reinforcement module constructs each image label using “imaging observation” and “existence” information "Soft labels" are constructed by calculating the cosine similarity between different image labels:

[0032]

[0033] Among them, R represents all reports in the current dataset, I represents all images in the current dataset, Represents the cosine similarity between the current report and the current image.

[0034] After obtaining the soft label, use the hard label y′ i and soft tags The weighted average of is taken as the final similarity label:

[0035]

[0036] Among them, α is the weight parameter, and α=0.2 in actual training. One-hot encoded label vectors representing image and report levels, y′ ii = 1 (the cosine similarity between image-report pairs of the same patient is 1), and the other elements are 0.

[0037] Use KL divergence as the loss function to minimize the distance between label and prediction similarity:

[0038]

[0039] Among them, p(I i ,R) and p(R i ,I) is the predicted similarity.

[0040] In actual training, in order to ensure the stability of training and better generalization, when optimizing L soft At the same time, we add the hard label-based contrastive learning loss L hard :

[0041]

[0042]

[0043] The final external representation enhancement loss L inter :

[0044] L inter =L hard +L soft (15)

[0045] Tags Here (or ) and the predicted similarity p(I i ,R)(or p(R i ,I)) are both probability distributions, so using KL divergence as the loss function to measure the difference between the two distributions is statistically significant. i It is in the form of a one-hot encoded vector, so the commonly used cross entropy loss can be used.

[0046] The present invention uses the anatomical region and report sentence alignment module, the internal representation learning reinforcement module and the external representation learning reinforcement module to perform joint learning of medical images and texts. During the training process, the loss functions of the three are taken into account at the same time, so that the model can learn fine-grained medical multimodal representations from paired radiology reports. The medical vision-language pre-training system trained on this basis can be used for downstream tasks such as classification, segmentation, and report generation, and significantly improves the accuracy of downstream tasks.

[0047] The advantages of the present invention mainly include:

[0048] (1) Anatomical knowledge is introduced into the medical vision-language pre-training system. The report is structured into a triplet of <anatomical region, imaging observation, existence> centered on the anatomical region. Fine-grained alignment between the anatomical region and the report clauses is completed under the guidance of professional radiologists.

[0049] (2) Continue to make full use of the “imaging observation” and “existence” information in the triplet, while strengthening the representation learning within a single image-text pair and between multiple image-report pairs.

[0050] (3) The present invention achieves leading image classification and segmentation results on the public datasets NIH X-ray, CheXpert, COVIDx, RSNA and SIIM. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a block diagram of the anatomical region guided medical vision-language pre-training system of the present invention.

[0052] Figure 2 It is a structural diagram of the anatomical region-guided medical vision-language pre-training system of the present invention.

[0053] Figure 3 It is an invention of an anatomical region-guided medical vision-language pre-training system performance visualization heat map. DETAILED DESCRIPTION

[0054] The specific details of the present invention are described below in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, the present invention includes an anatomical region and report sentence alignment module, an internal representation learning reinforcement module, and an external representation learning reinforcement module. The workflow of the present invention is:

[0056] (i) The anatomical region and report sentence alignment module consists of an object detector, a text parser, and an automatic alignment paradigm between the two. Specifically, the pre-trained object detector Faster R-CNN is used to obtain the anatomical region and report sentence alignment from each image. i M A The anatomical region bounding box (bbox) is used, and the complete report R is converted into a text parser Radgraph. i Decoupling to M T Triples:

[0057]

[0058] in, represents the bounding box of each anatomical region, are the coordinates of each bounding box; For its corresponding anatomical category, C pre is a predefined category. Each triple It is defined as <anatomical region, imaging observation, existence>, where the anatomical region in the triple belongs to the predefined set C ana .

[0059] Since the predefined categories C of anatomical regions in images and texts pre and C ana The number is not uniform and there is semantic overlap. The present invention proposes an automated alignment paradigm to strictly align the anatomical regions and report sentences of any image-report pair. Specifically, for a predefined category a∈C pre The bounding box and the predefined category pos∈C ana The anatomical regions in the triplet of , consider the following three cases:

[0060] Case 1: pos and a are exactly the same, or just slightly different in the way they are expressed.

[0061] Case 2: If case 1 is not satisfied, pair pos with the smallest a that can cover it.

[0062] Case 3: If there is a one-to-many relationship between pos and a, two solutions are provided, namely merging the anatomical region bbox or further splitting the clauses.

[0063] Merge anatomical region bounding boxes: Merge two / multiple bounding boxes into a whole region and pair with the current clause Or split clause: split the current clause into smaller units, pairing them with two / more bounding boxes and

[0064] After obtaining the local anatomical region-report clause pair for each sample, the fine-grained contrastive learning loss L is calculated arsa , to maximize the cosine similarity between pairs of images and text

[0065]

[0066] Where H(·) represents the cross entropy; Indicates the current clause; The cosine similarity vector representing the current anatomical region and each sentence in the entire report; Represents the cosine similarity vector between the current sentence and each anatomical region in the image; represents the one-hot encoded label vector, y ii =1 (the similarity between the current anatomical region and its corresponding report clause is 1), and the other elements are 0.

[0067] (II) On the basis of the above, we continue to use the “radiological observation” and “existence” information in the triplet to strengthen the representation learning within and between image-text pairs. The internal representation learning reinforcement module integrates the “radiological observation” and “existence” information to construct a label for each image. Among them, M Q is a predefined disease category. If the current image has disease category j, then otherwise,

[0068] Set the current image I i As input, the visual feature Z is obtained after passing through the image encoder i , which is used as the key vector (Key) and value vector (Value) of the self-attention-based image label recognition decoder, and the predefined disease categories As the query vector (Query), decode to get the predicted category of the current image, and use BCE Loss as the loss function:

[0069]

[0070] (III) The external representation learning reinforcement module also uses image labels constructed by "imaging observation" and "existence" Construct "soft labels" by calculating the cosine similarity between different image labels After obtaining the soft label, use the weighted average of the hard label and the soft label as the final similarity label:

[0071]

[0072] KL divergence is used as the loss function to minimize the distance between label and prediction similarity

[0073] Among them, p(I i ,R) and p(R i ,I) is the predicted similarity.

[0074] In actual training, when optimizing L soft At the same time, we add the hard label-based contrastive learning loss L hard To ensure training stability and better generalization:

[0075]

[0076] The final external representation enhancement loss L inter :

[0077] L inter =L hard +L soft (9)

[0078] Finally, the loss function that takes all three into account is optimized, enabling the model to learn fine-grained medical visual representations from paired radiology reports:

[0079] L=L arsa +L bce +L inter (10)

[0080] During the pre-training process, the MIMIC-CXR v2.0.0 dataset was used. Considering that the side view is not involved in the downstream tasks, the side view in the dataset was removed, the imaging observations and imaging conclusions in the report were retained, and reports with less than three words were removed, resulting in 210,000 image-report pairs. For each image, it was first resized to 256×256, and on this basis, a random cropping strategy was used to obtain a 224×224 image as the input of the image encoder. ResNet50 and ViT-B / 16 were used as image encoders, respectively, and BioClinicalBERT was used as a text encoder. The system was trained for 50 rounds on 4 RTX 3090 GPUs, with a batch size of 72 on each GPU. The gradient descent used the AdamW optimizer with a learning rate of 4e -5 , learning rate warm-up and learning rate annealing decay strategies are used in training. Weight decay is used to suppress model overfitting, and the decay rate is 5e -2 .

[0081] The system was validated in two major tasks: image classification and segmentation. (1) Medical image classification tasks: experiments were conducted on four representative datasets, namely NIH X-ray, CheXpert, RSNA, and COVIDx. Linear Probe Classification (fixing other model parameters and training only the last linear layer) was used to verify the migration capability of the pre-trained image encoder in this system. Similar to the processing of images in the pre-training dataset, each image in the downstream task was first resized to 256×256, and then a random cropping strategy was used on this basis to obtain an image of 224×224 as the input of the image encoder. (2) Medical image segmentation tasks: experiments were conducted on two medical semantic segmentation datasets, namely SIIM and RSNA. The image processing method was the same as the classification task. When the backbone network used ResNet50, the U-Net framework was used, and when ViT-B / 16 was used, the SETR framework was used. During fine-tuning, the backbone network (model encoder) was frozen and the model decoder was trained.

[0082] Compared with previous methods, this system has achieved better results in both classification and segmentation tasks. As shown in Table 1, this system has achieved competitive results in classification tasks compared with previous methods, especially on the COVIDx dataset, where the disease category "COVID-19" is not seen by the system during pre-training. This system has achieved significant improvements on this dataset, reflecting its strong generalization. In addition, this system outperforms previous methods at different data ratios on the NIH X-ray dataset (14 classification tasks). As shown in Table 2, this system has achieved significant improvements in segmentation tasks compared with previous methods, mainly due to the fine-grained alignment training method based on anatomical regions and report clauses.

[0083] Figure 3 A heat map showing the correspondence between images and disease terms. This system can accurately highlight the relevant areas corresponding to specific diseases and assist the model in accurate classification.

[0084] Table 1

[0085]

[0086]

[0087] Table 2

[0088]

[0089] References

[0090] [1]Ren S,He K,Girshick R,et al.Faster r-cnn:Towards real-time object detection with region proposal networks[J].Advances in neural informationprocessing systems,2015,28.

[0091] [2]Jain S, Agrawal A, Saporta A, et al. Radgraph: Extracting clinical entities and relations from radiology reports[J]. arXiv preprint arXiv:2106.14463,2021.

Claims

1. An anatomical region guided medical vision-language pre-training system, characterized in that: Through joint learning of medical images and texts, medical multimodal representations are learned from paired radiology reports; including: anatomical region and report sentence alignment module, internal representation learning reinforcement module, external representation learning reinforcement module; among them: The module for aligning anatomical regions and report clauses consists of a target detector, a text parser, and an automated alignment paradigm between the two. The target detector is used to locate the anatomical regions in the image, and the text parser splits the entire report into clauses related to the description of the anatomical regions and structures them into a triplet of <anatomical region, imaging observation, existence>. Based on the two, the module follows the alignment rules formulated by professional radiologists to form an automated alignment paradigm between anatomical regions and report clauses, and uses contrastive learning to give the model fine-grained alignment capabilities. The internal representation learning reinforcement module acts on the image-report pair, uses the image encoder and the image label recognition decoder to associate the image features with their corresponding labels, and provides the function of open disease category detection; The external representation learning reinforcement module acts on multiple image-report pairs, and utilizes comparative learning based on both soft labels and hard labels to enhance the correlation between different image-report pairs.

2. The medical vision-language pre-training system according to claim 1, characterized in that: The anatomical region and report sentence alignment module includes a target detector, a text parser, and an automatic alignment paradigm between the two. The specific process is as follows: Use the pre-trained object detector Faster R-CNN from each image I i M A The bounding box (bbox) of the anatomical region A i , expressed as: In the formula, represents the bounding box of each anatomical region, are the coordinates of each bounding box; For its corresponding anatomical category, C pre for all predefined categories; Use the text parser Radgraph to convert the complete report to R i Decoupling to M T triples, represented as: In the formula, each triple It is defined as <anatomical region, imaging observation, existence>, where the anatomical region in the triple belongs to the predefined set C ana ; Due to the predefined categories of anatomical regions in images and reports C pre and C ana The number of anatomical regions is not uniform, and there is semantic overlap. Under the guidance of professional radiologists, an alignment paradigm between anatomical regions and report sentences is proposed: given a predefined category a∈C of a bounding box pre , and a predefined category of anatomical regions in a triplet pos∈C ana , consider the following three scenarios: Case 1: pos and a are exactly the same, or just slightly different in expression, and the two can be paired directly; Case 2: If case 1 is not satisfied, pair pos with the smallest a that can cover it; both cases 1 and 2 can be formalized as: Case 3: If there is a one-to-many relationship between pos and a, two solutions are provided: Merge anatomical region bounding boxes or further split clauses: Merge anatomical region bounding boxes: Merge two / multiple bounding boxes into a whole region and pair it with the current clause Assume that the coordinates of the two are and The combined coordinates are expressed as: Split clause: Split the current clause into smaller units, pairing them with two or more bounding boxes. and After obtaining the local anatomical region-sentence pair for each sample, the fine-grained contrastive learning loss L is calculated arsa , to maximize the cosine similarity between pairs of anatomical regions and report clauses: Where H(·) represents the cross entropy; Indicates the current clause; represents the cosine similarity vector between the current anatomical region and all clauses in the entire report; represents the cosine similarity vector between the current clause and all anatomical regions in the image; One-hot encoded label vector representing anatomical region and clause level, y ii =1, and the other elements are 0.

3. The medical vision-language pre-training system according to claim 2, characterized in that: The internal representation learning reinforcement module constructs a label for each image using "radiological observation" and "existence" information Among them, M Q is a predefined disease category. If the current image has disease category j, then otherwise, Set the current image I i As input, after the image encoder f enc Then get the visual features in is the feature of each image block in the feature sequence, M Z is the sequence length; Z i As a self-attention based image label recognition decoder f dec The key vector (Key) and value vector (Value) of As the query vector (Query), where q j For a single disease category in the set, the predicted category of the current image is obtained through decoding by the self-attention mechanism, and the binary cross entropy loss (BCE Loss) is used as the loss function: Where N represents the total number of image report pairs in the current dataset.

4. The medical vision-language pre-training system according to claim 3, characterized in that: The external representation learning reinforcement module constructs each image label using "imaging observation" and "existence" information "Soft labels" are constructed by calculating the cosine similarity between different image labels: Among them, R represents all reports in the current dataset, I represents all images in the current dataset, cosine similarity between the current report and the current image; After obtaining the soft label, use the hard label y′ i and soft labels The weighted average of is taken as the final similarity label: Among them, α is the weight parameter, One-hot encoded label vectors representing image and report levels, y′ ii =1, other elements are 0; Use KL divergence as the loss function to minimize the distance between label and prediction similarity: Among them, p(I i ,R) and p(R i ,I) is the predicted similarity; In actual training, in order to ensure the stability of training and better generalization, when optimizing L soft At the same time, we add the hard label-based contrastive learning loss L hard : The final external representation enhancement loss L inter : L inter =L hard +L soft (15)

Images