Cross-modal clustering guide image encoder training method

Abstract

The invention relates to a training method of a cross-modal clustering guide image encoder, which combines comparative learning with a clustering algorithm, and seamlessly utilizes global and local features of a medical image and a radiology text. This approach facilitates more universal and robust medical visual representation learning. Image sub-regions and text sentences are regarded as basic units for global and local comparative learning. The global image and the global text features are respectively obtained by applying attention pooling to the local image and the local text features extracted by the encoder, and then are used for calculating global contrast loss. A cross-modal clustering algorithm utilizes local image-textual features obtained through cross-modal attention to generate additional negative samples, which are then used to calculate local contrast losses.

Description

Technical Field

[0001] The present invention relates to a data preprocessing method, and in particular to a training method for an image encoder based on cross-modal clustering guidance and contrastive learning. Background Art

[0002] In the field of natural images, the use of large-scale labeled datasets has significantly promoted the development of deep learning and achieved remarkable achievements in visual recognition. However, in medical imaging, it is a considerable challenge to obtain high-quality manually labeled datasets, as this requires annotation by experienced doctors, which is time-consuming and expensive. Therefore, the progress of deep learning in the field of medical imaging has been limited to a certain extent. In response to the problem of insufficient labeled datasets in the field of natural images, a pre-training method is usually used, that is, the model is first pre-trained on a large-scale natural image dataset, and then the learned general image representation is transferred to downstream tasks, thereby improving the convergence speed and generalization performance of the downstream model. However, due to the domain difference between natural images and medical images, directly transferring models pre-trained on natural image datasets such as ImageNet to downstream medical imaging tasks often produces suboptimal results. To solve this problem, a mainstream approach is to use self-supervised methods to pre-train large-scale unlabeled medical image datasets. However, this approach still faces two major challenges: first, the amount of medical image data available for pre-training is much less than that of natural image data; second, medical images emphasize local features and fine-grained information within the image, rather than just global information.

[0003] To cope with the limited medical image data problem, a natural idea is to leverage paired radiology reports to learn transferable medical visual representations through vision-language pre-training (VLP). Radiology reports are written by medical experts such as radiologists in clinical practice, providing valuable semantic information to provide supervision for images. This helps offset the scarcity and high annotation cost of image data and helps the model learn richer and deeper feature representations of medical images. In recent years, VLP has made significant progress in the medical field, especially through the design of reasonable pre-training tasks such as contrastive learning and generative reconstruction tasks. However, most contrastive learning methods are designed from a global perspective and do not take into account detailed information in images and texts. While the generated representations are suitable for image-level tasks such as image classification, they are not ideal for pixel-level tasks that require fine-grained details such as object detection and semantic segmentation.

[0004] Some works have taken local detail information into account. A contrastive learning strategy based on an attention mechanism is used to align sub-region image representations with word-level representations in reports, allowing the model to pay more attention to specific parts of the image and keywords in the text. Similarly, MGCA developed a token-by-token alignment model with a bidirectional attention mechanism to further refine the correspondence between image and text. AdaMatch introduced an adaptive patch extraction (AdaPatch) module that adaptively selects image regions based on the actual size and location of the lesion, thereby more accurately capturing lesion features. Although these word-level local alignment strategies enhance the model's ability to understand local details, they generally ignore the semantic structure information at the sentence level of medical reports. Medical reports often contain complex syntactic and semantic relationships, and the association between words is crucial to understanding the overall meaning of the report.

[0005] Although LOVT utilizes semantic information at the sentence level for contrastive learning, it only extracts negative samples from sentences within the same sample, resulting in an insufficient number of negative samples and limiting the model performance. Summary of the invention

[0006] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: how to provide an image encoder that can accurately extract medical image features.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solution: a training method of a cross-modal clustering guided image encoder, comprising the following steps:

[0008] S1: Receive a dataset The dataset consists of N pairs of medical images and reports. Respectively represent the i-th medical image and the i-th report, and the medical image and the report correspond one to one; preprocess the N pairs of data in G;

[0009] S2: The N pairs of preprocessed data are used to obtain a batch, which is then passed through the image encoder E I Map a batch of medical images in G to the latent space to obtain local image features and global image features Through the text encoder E R Map the reports in a batch in G to the latent space to obtain local text features and global text features Where p = 1, 2, .. P, m = 1, 2, … M, P represents the number of local image blocks in a medical image, and M represents the number of sentences in a report.

[0010] S3: Use four multi-layer perceptrons MLPs to and Mapped to the same dimension D.

[0011] In the current batch: Calculate the global alignment loss from the global medical image to the report When the medical images are matched one by one As the anchor point anchor, at the same time, the corresponding As positive samples, other as negative samples.

[0012] In the current batch: Calculate the global alignment loss from the global report to the medical image When the medical images are matched one by one As the anchor point anchor, at the same time, the corresponding As positive samples, other as negative samples.

[0013] S4: and The sum is taken as the total global loss L global ;

[0014]

[0015] S5: Compute attention representation from local image to report and reports to the local image attention representation

[0016] In the current batch: Calculate the local alignment loss of medical images to report When the medical images are matched one by one As an anchor, the medical image corresponding to the anchor and Two cross-modal representations computed via cross-attention and and Represent the cross-modal representation from local image to local text and the cross-modal representation from local text to local image, respectively. As positive samples, other as negative samples.

[0017] In the current batch: calculate the bidirectional boundary loss L bml When the medical images are matched one by one As an anchor, it will be Most similar cluster center Medical images in the associated cluster are considered false negative samples.

[0018] In the current batch: Calculate the local alignment loss reported to the medical image

[0019] S6: L bml and The sum is taken as the local alignment loss L local .

[0020]

[0021] S7: Randomly mask with a preset probability Get the local image features after masking Connect on channel dimension D and use To complete The fused image features are obtained, and the fused image features are decoded and reconstructed through the decoding module, and the image decoding and reconstruction loss L is calculated for the fused image features. re .

[0022] S8: L global , L local and L re The weighted sum is used to obtain the overall loss L, and E is updated inversely according to the value of L. I and E R The parameters are returned to S2 until the value of L stops decreasing and the trained E is obtained. I and E R .

[0023] L=λ global L global +λ local L local +λ re L re (13)

[0024] where λ global ,λ local and λ re are the corresponding hyper parameters respectively.

[0025] Furthermore, the local image features are obtained in S2 and global image features The steps are:

[0026] Use ResNet50 as E I ,Will The representation encoded into P sub-regions is used as the local image feature, denoted as Where p represents the position index of the p-th sub-region, and C I is the dimension of local image features, through Perform self-attention pooling to obtain global image features

[0027] Furthermore, the local text features are obtained in S2 and global text features The steps are:

[0028] Using BioClinicalBERT as the text encoder R , each report is encoded into M sentence-level representations, denoted as Where m represents the sentence index of the mth sentence in the i-th report, and C R is the dimension of local text features, and M is not fixed between different texts. Perform self-attention pooling instead of using [CLS] tags to obtain global text features

[0029] Furthermore, the calculation in S3 The process is:

[0030]

[0031] The global alignment loss from global medical images to reports is:

[0032]

[0033] Where B represents the batch size and τ1 represents the temperature hyperparameter.

[0034] Furthermore, the calculation in S3 The process is:

[0035]

[0036] Furthermore, the calculation in S5 and The process is:

[0037] Local features obtained from encoder using criss-cross attention and Get the corresponding cross-modal representation in and For each sentence, by letting Participate in all paired images The attention process is used to obtain cross-modal representation The attention representation based on local image to report is expressed as:

[0038]

[0039] where Q,K,V∈R D×D are different learnable matrices respectively, and σ represents the sigmoid function.

[0040] The formula for calculating the report to the local image attention representation is:

[0041]

[0042] Furthermore, the calculation in S5 The process is:

[0043] from Collect all the sentence representations in the lth mini-batch, denoted as from Collect all the sentence representations in the lth mini-batch, denoted as where j∈[1,M l ],M l Represents the number of sentences in the lth mini-batch. The corresponding l-th mini-batch contains all the sentences Cluster into K clusters The cluster center is denoted by for according to With each cluster center Arrange clusters C in descending order of cosine similarity l and cluster center c l ,get and at this time, With cluster center The most similar cluster The elements in Select the closest The cluster center As Difficult negative samples.

[0044] When computing the local alignment loss for medical images to be reported, The corresponding difficult negative sample set Used as additional negative samples, medical images to report local alignment loss Defined as:

[0045]

[0046] where τ3 is the temperature hyperparameter, It is a weighted matrix. The values ​​on the diagonal represent the weights of difficult negative samples. The intermediate variable H(.) has no actual meaning.

[0047] Furthermore, the calculation of L in S5 bml The process is: Most similar cluster center The medical sentence representations in the associated cluster are regarded as false negative samples, and the similarity between the anchor point and the false negative sample should be higher than the hard negative sample but lower than the positive sample:

[0048]

[0049] Where Δx l,j is the difference between two cosine similarities, and α and β are two threshold parameters.

[0050] Furthermore, the calculation in S5 The process is:

[0051]

[0052] Where h represents MLP, S represents the stop gradient operation, and sim calculates the cosine similarity.

[0053] Furthermore, in S7, the fused image features are subjected to image decoding and reconstruction loss L re for:

[0054]

[0055] Where F represents the fused image features, D I Represents the image decoder, and head represents the upsampling head.

[0056] Compared with the prior art, the present invention has at least the following advantages:

[0057] The present invention aims to extract more fine-grained image and text representations from both global and local perspectives. By combining contrastive learning with a clustering algorithm, a novel clustering guided negative sampling (CGNS) technique is proposed. This technique not only increases the number of negative samples in contrastive learning, but also improves the learning efficiency and representation ability of the model by giving greater weight to difficult negative samples. In order to retain low-level image features that are beneficial to downstream fine-grained tasks, the present invention designs a cross-modal masked image reconstruction module, which uses local text-to-image features obtained through cross-attention to complete the masked local image features. This enhances the cross-modal information interaction capability of the model. A large number of experimental results show that the method of the present invention demonstrates excellent data efficiency and image representation extraction capabilities on downstream tasks such as classification, detection, and segmentation. These results demonstrate the effectiveness and superiority of the CM-CGNS method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of negative sample sampling guided by cross-modal clustering proposed in the present invention.

[0059] Figure 2 The overall architecture diagram of CM-CGNS.

[0060] Figure 3 Details of the cross-modal attention module.

[0061] Figure 4 Visualization of the cross-modal attention map. The sentences corresponding to the four images are: (a) "Persistent bilateral small effusions, larger on the left, but the amount of effusion has decreased." (b) "No pneumothorax." (c) "The moderate effusion on the left has increased." (d) "Mild to severe bilateral pleural effusions, larger on the right than on the left."

[0062] Figure 5 t-SNE visualization of the representations of the last layer of the image encoder. Five true labels from the CheXpert 5x200 dataset are shown. DETAILED DESCRIPTION

[0063] The present invention is described in further detail below.

[0064] This paper focuses on joint representation learning from paired medical images and radiology reports. We propose a new method, Cross-Modal Clustering-Guided Negative Sampling (CM-CGNS), which combines contrastive learning with a clustering algorithm to seamlessly exploit global and local features of medical images and radiology texts. This method promotes more general and robust medical visual representation learning. We regard image subregions and text sentences as basic units for global and local contrastive learning. Global image and global text features are obtained by applying attention pooling to the local image and local text features extracted by the encoder, respectively, and are subsequently used to compute the global contrastive loss. A cross-modal clustering algorithm utilizes the local image-text features obtained through cross-modal attention to generate additional negative samples, which are subsequently used to compute the local contrastive loss.

[0065] The introduction of multimodal clustering improves the overall performance of the model, such as Figure 1 As shown. In addition, in order to learn more accurate local features and finer pixel-level details, mask image reconstruction is essential. To this end, the present invention designs a cross-modal mask image reconstruction module to reconstruct the masked image area using cross-modal text information.

[0066] A cross-modal clustering guided image encoder training method comprises the following steps:

[0067] S1: The overall structure of the method is as follows Figure 2 As shown. Receive a data set The dataset consists of N pairs of medical images and reports. They represent the i-th medical image and the i-th report respectively, and the medical images and reports correspond one to one.

[0068] Preprocessing of N pairs of data in G Since the downstream dataset only contains frontal chest images, we removed all side views from the dataset, randomly cropped the images to 224×224, and normalized them. In addition, we extracted the text of the "Impression" and "Findings" sections from the text reports to obtain a detailed description of the medical disease, and removed those reports that were empty or had less than 3 terms.

[0069] When aligning medical images and text, in addition to standard global contrastive learning, we also adopt local contrastive learning to capture the fine-grained image information required for downstream image tasks. By combining a clustering algorithm with contrastive learning, we increase the number of negative samples used for sentence-level alignment and give greater weights to difficult negative samples to enhance the learning efficiency and representation ability of the model. We design a cross-modal image reconstruction module to further utilize fine-grained image information and reconstruct masked local image features using local text-to-image features obtained through cross-attention, thereby improving the model's cross-modal information interaction ability while retaining low-level image features that are beneficial to downstream fine-grained tasks.

[0070] S2: The N pairs of preprocessed data are used to obtain a batch, which is then passed through the image encoder E I Map a batch of medical images in G to the latent space to obtain local image features and global image features Through the text encoder E R Map the reports in a batch in G to the latent space to obtain local text features and global text features Where p = 1, 2, .. P, m = 1, 2, … M, P represents the number of local image blocks in a medical image, and M represents the number of sentences in a report.

[0071] Specifically, the local image features are obtained in S2 and global image features The steps are:

[0072] Use ResNet50 as E I ,Will The representation encoded into P sub-regions is used as the local image feature, denoted as Where p represents the position index of the p-th sub-region, and C I is the dimension of local image features, through Perform self-attention pooling to obtain global image features

[0073] Specifically, the local text features are obtained in S2 and global text features The steps are:

[0074] Medical reports usually contain long sentences and require reasoning between words, so BioClinicalBERT is used as the text encoder. R ,It is pre-trained on the MIMIC III dataset and uses the self-attention mechanism to establish long-range semantic dependencies between words in medical reports.,Each sentence in a medical image-report usually describes one or more sub-regions in the image, indicating that there is a correspondence between the report sentence and the image sub-region, e.g. Figure 2 We extract sentence-level features from reports to align with image subregions, which is different from word-level feature extraction in MGCA and GloRIA. is encoded into M sentence-level representations, denoted as Where m represents the sentence index of the mth sentence in the i-th report, and C R is the dimension of local text features, and M is not fixed between different texts. Perform self-attention pooling instead of using [CLS] tags to obtain global text features

[0075] S3: Use four multi-layer perceptrons MLPs to and Mapped to the same dimension D.

[0076] In the current batch: Calculate the global alignment loss from the global medical image to the report When the medical images are matched one by one As the anchor point anchor, at the same time, the corresponding As positive samples, other as negative samples.

[0077] In the current batch: Calculate the global alignment loss from the global report to the medical image When the medical images are matched one by one As the anchor point anchor, at the same time, the corresponding As positive samples, other as negative samples.

[0078] Specifically, the global alignment loss from the global medical image in S3 to the report The process is: Use cosine similarity to calculate the similarity between the image and the report

[0079]

[0080] The global alignment loss from global medical images to reports is:

[0081]

[0082] Where B represents the batch size and τ1 represents the temperature hyperparameter.

[0083] Compute the global alignment loss from global report to medical image The process is:

[0084]

[0085] S4: In multimodal learning, we aim to achieve strong mutual dependencies between images and texts, i.e., images and their corresponding reports should provide complementary information. Therefore, maximizing the mutual information between images and reports helps enhance the model's ability to understand the semantic associations between different modalities. However, directly calculating the mutual information is often challenging because it requires accurate joint and marginal probability distributions. To address this issue, we use the InfoNCE loss to approximate the mutual information and estimate the lower bound of the mutual information through contrastive learning. Specifically, paired medical images and reports should have similar semantic information after mapping to the latent space, paired images and reports should be close to each other in the embedding space, while non-matching pairs should be far apart. Cosine similarity is used to calculate the similarity between images and reports, and the total global loss is calculated.

[0086] Will and The sum is taken as the total global loss L global ;

[0087]

[0088] S5: Compute attention representation from local image to report and reports to the local image attention representation

[0089] In the current batch: Calculate the local alignment loss of medical images to report When the medical images are matched one by one As an anchor, the medical image corresponding to the anchor and Two cross-modal representations computed via cross-attention and and Represent the cross-modal representation from local image to local text and the cross-modal representation from local text to local image, respectively. As positive samples, other (and the hard negatives obtained by our CM-CGNS method) are used as negative samples.

[0090] In the current batch: calculate the bidirectional boundary loss L bml When the medical images are matched one by one As an anchor, it will be Most similar cluster center Medical images in the associated cluster are considered false negative samples.

[0091] In the current batch: Calculate the local alignment loss reported to the medical image

[0092] In downstream detection and segmentation tasks, the model needs to learn more accurate edge information and more detailed pixel-level feature information of objects. However, both global and local alignment process high-level image features extracted by the encoder, which may ignore the edge and detail features of the image, thus affecting the performance on downstream tasks. To solve this problem, we designed a cross-modal masked image reconstruction (CM-MIR) module to enhance the model's cross-modal information fusion capability while obtaining low-level detail information of the image. Specifically, for the local image representation obtained from the image encoder, Randomly mask with a preset probability Get the local image features after masking Then, concatenate on the channel dimension D and use To complete The fused image features are obtained, and the fused image features are decoded and reconstructed through a decoding module [the decoding module consists of five convolutional layers and bilinear interpolation].

[0093] Specifically, S5 calculates and The process is:

[0094] Global alignment focuses on the overall consistency between images and their reports, but ignores the specific details in the image. In the medical field, fine-grained information in images (such as tumor boundaries and lesion areas) is often crucial, and relying solely on global alignment may ignore these key details. Therefore, we need to introduce a local alignment module.

[0095] First, we need to generate cross-modal representations. Specifically, we use the local features obtained from the encoder using cross-attention and Get the corresponding cross-modal representation in and like Figure 3 As shown. Pathological information usually occupies only a small part of the entire image or text. Considering that not all sub-regions / sentences contain useful semantic information, we choose to use the sigmoid function instead of the softmax function, because softmax assumes that each local unit (sub-region or sentence) contributes to the final cross-modal representation in a constant proportion (even if it is small). This assumption may not be appropriate when processing data with a lot of irrelevant or noisy information, because it artificially assigns a certain weight to each part, even if these parts do not carry important information. In contrast, the sigmoid function does not require the sum of the output probabilities to be 1, which is more in line with the characteristics of medical data.

[0096] For each sentence m, by letting Participate in all paired images The attention process is used to obtain cross-modal representation The attention representation based on local image to report is expressed as:

[0097]

[0098] where Q,K,V∈R D×D are different learnable matrices respectively, and σ represents the sigmoid function.

[0099] The formula for calculating the report to the local image attention representation is:

[0100]

[0101] In addition, in contrastive learning, the quality of the learned representation improves with the increase in the number of high-quality negative samples. However, in the process of local text alignment, since sentences in different reports may have semantic similarities, simply sampling negative examples from all sentences in a mini-batch may lead to feature collapse, making it difficult for the model to distinguish between positive and negative sample pairs. To the best of our knowledge, other methods such as LOVT and PRIOR only use sentence representations from the same report as negative samples. Although they solve the above problems, since the number of sentences in a single report is usually small (sometimes only two or three sentences), this may limit the performance and generalization ability of the model, and the model may not be able to fully distinguish between positive and negative samples. To solve this problem, we propose a cross-modal clustering-guided negative sample sampling (CM-CGNS) method, which combines clustering and contrastive learning, increases the number of negative samples required for contrastive learning, and gives more weight to difficult negative samples, further improving the learning efficiency and feature representation ability of the model.

[0102] Specifically, calculate from S5 The process is:

[0103] from Collect all the sentence representations in the lth mini-batch, denoted as from Collect all the sentence representations in the lth mini-batch, denoted as where j∈[1,M l ],M l Represents the number of sentences in the lth mini-batch. The corresponding l-th mini-batch contains all the sentences Cluster into K clusters The cluster center is denoted by for according to With each cluster center Arrange clusters C in descending order of cosine similarity l and cluster center c l ,get and at this time, With cluster center The most similar cluster The elements in Select the closest The cluster center As The difficult negative samples are Most likely to include Same semantic information.

[0104] When computing the local alignment loss for medical images to be reported, The corresponding difficult negative sample set Used as additional negative samples, thereby increasing the number of negative samples and giving each sample’s corresponding difficult negative sample a greater weight, medical images to the reported local alignment loss Defined as:

[0105]

[0106] where τ3 is the temperature hyperparameter, It is a weighted matrix. The values ​​on the diagonal represent the weights of difficult negative samples. The intermediate variable H(.) has no practical meaning. The optimal weight is determined by grid search.

[0107] Specifically, S5 calculates L bml The process is:

[0108] False negative samples are samples that should be considered as negative samples but are very similar to positive samples in some aspects. These negative samples are very close to positive samples in feature space and should not be pushed away, otherwise they may distort the learned representation. Most similar cluster center The medical sentence representations in the associated cluster are regarded as false negative samples, and the similarity between the anchor point and the false negative sample should be higher than the hard negative sample but lower than the positive sample:

[0109]

[0110] Where Δx l,j is the difference between the two cosine similarities, α and β are two threshold parameters, specifically, it is the anchor point (sentence representation ) and false negative samples The cosine similarity between the anchor point and the positive sample is subtracted from the cosine similarity between the anchor point and the positive sample.

[0111] L bml is the bidirectional margin loss, which is used to calculate the model's loss. This loss function aims to ensure that false negative samples are neither too close to positive samples nor pushed too far away. It achieves this through the ReLU function. When Δx l,j The model is penalized when it exceeds a predefined threshold. bml Limit Δx ij Make Keep it in the range [-β,-α], so as to keep an appropriate distance between false negative samples and positive samples in the feature space.

[0112] α and β are two threshold parameters used to determine Δx l,j The effective range of α is usually a small positive value, while β is a large positive value. The loss function Lbml Will punish Δx l,j Less than -α or greater than -β. This ensures that the distance between the false negative sample and the positive sample is within a reasonable range, neither too close nor too far.

[0113] Specifically, S5 calculates The process is:

[0114] When calculating the local alignment loss reported to medical images, adjacent image subregions may contain similar information due to the spatial structure between image subregions and the overlapping receptive fields caused by convolution. Directly using adjacent image subregions as negative samples may lead to feature collapse. Therefore, the SimSiam alignment method is adopted, which constructs a twin network to learn image representation, reducing the dependence on the number of negative samples. The local alignment loss formula reported to medical images is as follows:

[0115]

[0116] Where h represents MLP, S represents the stop gradient operation, and sim calculates the cosine similarity.

[0117] S6: L bml and The sum is taken as the local alignment loss L local .

[0118]

[0119] S7: Randomly mask with a preset probability Get the local image features after masking Connect on channel dimension D and use To complete The fused image features are obtained, and the fused image features are decoded and reconstructed through the decoding module, and the image decoding and reconstruction loss L is calculated for the fused image features. re .

[0120] S8: L global , L local and L re The weighted sum is used to obtain the overall loss L, and E is updated inversely according to the value of L. I and E R The parameters are returned to S2 until the value of L stops decreasing and the trained E is obtained. I and E R .

[0121] L=λ global L global +λlocal L local +λ re L re (13)

[0122] where λ global ,λ local and λ re are the corresponding hyper parameters respectively.

[0123] Experiments and Results

[0124] 1. Dataset

[0125] MIMIC-CXR 2.0.0: We pre-trained our model on the MIMIC-CXR 2.0.0 dataset, which is the second public version of MIMIC-CXR. The dataset contains 377,110 images in JPG format and structured labels extracted from 227,827 free-text radiology reports. We pre-processed the dataset. Since the dataset for downstream tasks does not include lateral chest images, we removed all lateral chest images and only retained the frontal chest images. In addition, we only retained the impression and finding sections in the free-text reports to obtain a detailed description of the medical condition, and discarded reports that were empty or contained less than three tags. Finally, we obtained 232,286 image-text pairs.

[0126] COVIDx: The COVIDx dataset contains more than 30,000 chest X-ray (CXR) images from more than 16,600 patients, including 16,490 COVID-19 positive images from more than 2,800 patients. The original validation set was used as test data, and 10% of the original training set was manually allocated for validation.

[0127] CheXpert: We use CheXpert as a classification dataset. CheXpert is a large chest X-ray dataset containing 224,316 images from 65,240 patients. We remove the lateral images, leaving 191,229 frontal view images. Each image is annotated for five independent diseases, and the task is to classify each image into five separate binary labels: atelectasis, cardiomegaly, consolidation, edema, and pleural effusion. We use the expert-annotated validation set as the test set.

[0128] RSNA Pneumonia Detection: We use the second phase of the RSNA Pneumonia dataset for classification, object detection, and semantic segmentation tasks. The dataset contains approximately 29,700 orthostatic chest X-ray images, each with an image-level pneumonia classification label and a bounding box pneumonia detection label. For classification experiments, we split the dataset into training, validation, and test sets at a ratio of 70% / 15% / 15%. In object detection and semantic segmentation experiments, the dataset is split at a ratio of 60% / 20% / 20%.

[0129] Object CXR: Object CXR contains 9,000 chest X-rays with foreign body detection targets. The original training set is divided into a training set containing 6,400 images and a validation set of 1,600 images. The original development set is used as a test set, containing 1,000 images.

[0130] SIIM-ACR Pneumothorax: The SIIM-ACR pneumothorax dataset contains 12,047 chest X-rays, each with a manually annotated pneumothorax segmentation mask. The present invention divides the dataset into a training set, a validation set, and a test set in a ratio of 70% / 15% / 15%.

[0131] 2. Implementation Details

[0132] The present invention uses BioClinicalBERT as the text encoder and ResNet-50 as the image encoder. The present invention resizes the image to 224×224, sets the dimension of the latent space to 768, and sets the temperature hyperparameters τ1, τ2, and τ3 to 0.01. The loss weight coefficient is set to λ global =λ local =1.0 and λ re =10.0. The optimal weight of μ is found to be 3 through grid search. For the pre-training stage, the batch size is set to 64, the initial learning rate is 1e-5, and the cosine decay strategy is adopted. The first stage is set to 10 epochs, and the second stage is set to 20 epochs. The present invention conducts pre-training experiments on 4 RTX 3090 GPUs.

[0133] The proposed method is compared with other widely used self-supervised methods, including single-modal image self-supervised learning (SSL) methods (MoCo, MoCoV2, SimCLR) and multimodal visual-language pre-training (VLP) methods (ConVIRT, CLIP, GloRIA, LoVT, MGCA, PRIOR). In addition, the present invention also includes a randomly initialized ResNet50 model and a ResNet50 model pre-trained on the ImageNet dataset for comparison. In the single-modal image SSL methods, only the data of the image modality is used, and positive and negative sample pairs are constructed through data augmentation for contrastive learning pre-training. In the multimodal VLP methods, ConVIRT and CLIP do not design a local alignment module to facilitate the extraction of fine-grained pathology information. GloRIA, LoVT, MGCA and PRIOR specifically design their own local alignment modules to extract local pathology information, making them more suitable for medical image-text SSL tasks. Since the pre-training dataset of some experiments is not MIMIC-CXR, for fair comparison, the present invention reproduces all methods on the MIMIC-CXR dataset using their respective official codes and default parameter settings, and then uses the obtained pre-trained models for downstream tasks.

[0134] 3. Medical Image Classification Results

[0135] The present invention conducted medical image classification experiments on CheXpert, RSNA and COVIDx datasets. The present invention added a linear classification head to the image encoder obtained in the pre-training stage. During the downstream task, the present invention froze the weight parameters of the image encoder and only trained the linear classification head to evaluate the migration performance of the pre-trained image encoder. In addition, the present invention evaluated the difference in model performance when fine-tuning using 1%, 10% and 100% of the downstream data, further verifying the data efficiency of the model of the present invention. The classification experiment results are shown in Table 1.

[0136] As shown in Table 1, all methods use ResNet50 as the backbone network. Among them, the method using image self-supervised learning (SSL) outperforms the methods with random initialization and ImageNet initialization. In addition, the method using visual-language pre-training (VLP) outperforms the method using only image SSL. In addition, the method containing a specially designed local alignment module significantly outperforms the standard VLP method. The CM-CGNS method of the present invention achieved the best results in nine different settings on three datasets, surpassing the current most advanced MGCA and PRIOR methods. This shows the effectiveness of increasing the number of negative samples through the CM-CGNS design to enhance the contrastive learning strategy.

[0137] Table 1. Linear classification results obtained by fine-tuning on three datasets using 1%, 10%, and 100% of the data. For CheXpert and RSNA datasets, the area under the ROC curve (AUROC) is used as the evaluation metric; for COVIDx dataset, the accuracy (ACC) is used

[0138] As an evaluation metric, the best result is highlighted in bold and the second best result is highlighted in strokes.

[0139]

[0140] 4. Medical target detection results

[0141] The present invention conducted object detection experiments on the RSNA and Object CXR datasets. For fair comparison, all methods adopted the architecture of ResNet50 combined with YOLOv3 and kept the parameter configuration consistent. The experimental results are shown in Table 2. The present invention observed that methods using VLP (visual language pre-training) generally performed better than methods using only image modality for pre-training. However, some SSL (self-supervised learning) methods that use only image modality, such as MoCoV2, outperform VLP methods that do not contain local alignment modules, such as ConVIRT and CLIP in some cases. This may be because in the pre-trained MIMIC-CXR dataset, medical reports may not provide detailed annotations of the specific location and type of pneumonia, resulting in challenges for VLP methods in extracting and utilizing this information. In contrast, SSL methods rely mainly on the information of the image itself and are therefore less affected by inaccurate reports, thereby showing better performance on the RSNA dataset. However, the method of the present invention achieved the best results in five of the six different experimental settings, showing that the model of the present invention can effectively extract features even when processing complex datasets, demonstrating its robustness.

[0142] Table 2. Object detection results (mAP [%) on RSNA and Object CXR datasets. Each dataset is fine-tuned using 1%, 10%, and 100% of the training data. "-" indicates that the mAP is less than 1%.

[0143]

[0144] 5. Medical semantic segmentation results

[0145] The present invention conducted semantic segmentation experiments on RSNA and SIIM datasets, and all methods adopted the architecture combining ResNet50 and U-Net. As can be seen from Table 3, the methods containing specially designed local alignment modules showed significantly higher performance than those lacking such modules. This observation emphasizes the critical importance of local detail information in medical image segmentation. In addition, the methods using VLP continuously outperformed the methods that relied solely on image modality for pre-training. This suggests that integrating information from image and text modalities during training can enhance the model's ability to extract meaningful image representations. It is worth noting that among all the evaluated methods, the CM-CGNS method proposed in the present invention stood out, achieving the best performance in four of the six experimental settings and taking second place in the remaining two settings. These results confirm the effectiveness of the CGNS module and mask image reconstruction mechanism in enhancing local image feature extraction, which is particularly beneficial for medical image segmentation tasks.

[0146] Table 3. Semantic segmentation results (Dice [%) on SIIM and RSNA datasets. Each dataset is fine-tuned using 1%, 10%, and 100% of the training data, respectively.

[0147]

[0148] Table 4. Ablation study results on the CheXpert classification dataset and SIIM segmentation dataset. CM-CGNS and CM-MIR are the methods proposed in this paper. The best experimental results are highlighted in bold.

[0149]

[0150] 6. Visualization and Ablation Research

[0151] The present invention Figure 4 The attention map between the sentence-level representation and the image representation is visualized in Figure 1. It can be seen that the model of the present invention can well align the sentence representation with its corresponding image region, which helps to obtain more fine-grained features, thereby enhancing the representation ability of the model. In order to further evaluate the representation learning ability of the model proposed in the present invention, the present invention uses the t-SNE visualization method. Using the CheXpert 5x200 dataset, the present invention generates visual representations based on the embedding extracted from the last layer of the image encoder, such as Figure 5 As shown. The findings of the present invention show that in the t-SNE diagram, the points corresponding to different diseases of the method of the present invention are clearly separable, showing a more significant clustering effect. In contrast, the best performing alternative methods tend to generate more isotropic and uniform feature distributions, and therefore have difficulty in effectively distinguishing various disease categories.

[0152] In the ablation study, the baseline only contains a global alignment module and a traditional local alignment module. The proposed CM-CGNS and CM-MIR methods are added to the baseline model to verify the effectiveness of each component. As shown in Table 4, when any method is added to the baseline, the experimental indicators are improved, indicating that all proposed methods are effective. When the two methods are combined, the performance is further improved, which shows that these methods complement each other.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A cross-modal clustering guided image encoder training method, characterized by: The steps include: S1: Receive a dataset The dataset consists of N pairs of medical images and reports. Respectively represent the i-th medical image and the i-th report, and the medical image and the report correspond one to one; preprocess the N pairs of data in G; S2: The N pairs of preprocessed data are used to obtain a batch, which is then passed through the image encoder E I Map a batch of medical images in G to the latent space to obtain local image features and global image features Through the text encoder E R Map the reports in a batch in G to the latent space to obtain local text features and global text features Where p = 1, 2, .. P, m = 1, 2, … M, P represents the number of local image blocks in a medical image, and M represents the number of sentences in a report; S3: Use four multi-layer perceptrons MLPs to and Mapped to the same dimension D; In the current batch: Calculate the global alignment loss from the global medical image to the report When the medical images are matched one by one As the anchor point anchor, at the same time, the corresponding As positive samples, other As negative samples; In the current batch: Calculate the global alignment loss from the global report to the medical image When the medical images are matched one by one As the anchor point anchor, at the same time, the corresponding As positive samples, other As negative samples; S4: and The sum is taken as the total global loss L global ; S5: Compute attention representation from local image to report and reports to the local image attention representation In the current batch: Calculate the local alignment loss of medical images to report When the medical images are matched one by one As an anchor, the medical image corresponding to the anchor and Two cross-modal representations computed via cross-attention and and Represent the cross-modal representation from local image to local text and the cross-modal representation from local text to local image, respectively. As positive samples, other Treated as negative samples; In the current batch: calculate the bidirectional boundary loss L bml When the medical images are matched one by one As an anchor, it will be Most similar cluster center Medical images in the associated cluster are considered false negative samples; In the current batch: Calculate the local alignment loss reported to the medical image S6: L bml and The sum is taken as the local alignment loss L local ; S7: Randomly mask with a preset probability Get the local image features after masking Connect on channel dimension D and use To complete The fused image features are obtained, and the fused image features are decoded and reconstructed through the decoding module, and the image decoding and reconstruction loss L is calculated for the fused image features. re ; S8: L global , L local and L re The weighted sum is used to obtain the overall loss L, and E is updated inversely according to the value of L. I and E R The parameters are returned to S2 until the value of L stops decreasing and the trained E is obtained. I and E R ; L=λ global L global +λ local L local +λ re L re (13) where λ global ,λ local and λ re are the corresponding hyperparameters respectively.

2. The method for training a cross-modal clustering guided image encoder according to claim 1, characterized in that: The local image features are obtained in S2 and global image features The steps are: Use ResNet50 as E I ,Will The representation encoded into P sub-regions is used as the local image feature, denoted as Where p represents the position index of the p-th sub-region, and C I is the dimension of local image features, through Perform self-attention pooling to obtain global image features 3. The method for training a cross-modal clustering guided image encoder according to claim 2, characterized in that: The local text features are obtained in S2 and global text features The steps are: Using BioClinicalBERT as the text encoder R , each report is encoded into M sentence-level representations, denoted as where m represents the sentence index of the mth sentence in the i-th report, and C R is the dimension of local text features, M is not fixed between different texts; Perform self-attention pooling instead of using [CLS] tags to obtain global text features 4. The method for training a cross-modal clustering guided image encoder according to claim 3, characterized in that: The S3 calculation The process is: The global alignment loss from global medical images to reports is: Where B represents the batch size and τ1 represents the temperature hyperparameter.

5. The method for training a cross-modal clustering guided image encoder according to claim 4, characterized in that: The S3 calculation The process is:

6. The method for training a cross-modal clustering guided image encoder according to claim 5, characterized in that: The S5 calculation and The process is: Local features obtained from encoder using criss-cross attention and Get the corresponding cross-modal representation in and For each sentence, by letting Participate in all paired images The attention process is used to obtain cross-modal representation The attention representation based on local image to report is expressed as: where Q,K,V∈R D×D are different learnable matrices, σ represents the sigmoid function; The formula for calculating the report to the local image attention representation is:

7. The method for training a cross-modal clustering guided image encoder according to claim 6, characterized in that: The S5 calculation The process is: from Collect all the sentence representations in the lth mini-batch, denoted as from Collect all the sentence representations in the lth mini-batch, denoted as where j∈[1,M l ],M l represents the number of sentences in the lth mini-batch; The corresponding l-th mini-batch contains all the sentences Cluster into K clusters The cluster center is denoted by for according to With each cluster center Arrange clusters C in descending order of cosine similarity l and cluster center c l ,get and at this time, With cluster center The most similar cluster The elements in Select the closest The cluster center As Difficult negative samples; When computing the local alignment loss for medical images to be reported, The corresponding difficult negative sample set Used as additional negative samples, medical images to report local alignment loss Defined as: where τ3 is the temperature hyperparameter, It is a weighted matrix. The values ​​on the diagonal represent the weights of difficult negative samples. The intermediate variable H(.) has no actual meaning.

8. The method for training a cross-modal clustering guided image encoder according to claim 7, characterized in that: The S5 calculates L bml The process is: will with Most similar cluster center The medical sentence representations in the associated cluster are regarded as false negative samples, and the similarity between the anchor point and the false negative sample should be higher than the hard negative sample but lower than the positive sample: Where Δx l,j is the difference between two cosine similarities, and α and β are two threshold parameters.

9. The method for training a cross-modal clustering guided image encoder according to claim 8, characterized in that: The S5 calculation The process is: Where h represents MLP, S represents the stop gradient operation, and sim calculates the cosine similarity.

10. The method for training a cross-modal clustering guided image encoder according to claim 9, characterized in that: In S7, the fused image features are subjected to image decoding and reconstruction loss L re for: Where F represents the fused image features, D I Represents the image decoder, and head represents the upsampling head.

Images