Active learning method for multi-modal medical images

Abstract

The application discloses an active learning method for multi-modal medical images, comprising the following steps: obtaining an initial unlabeled data set, the initial unlabeled data set comprising a plurality of modality sets, wherein the modality set is a set of a plurality of samples; constructing an encoder, pre-training the encoder through a multi-modal contrast learning method to obtain a pre-trained feature encoder; calculating the point-by-point mutual information between each sample modality in the initial unlabeled data set through the pre-trained feature encoder, and obtaining a valuable sample subset based on the point-by-point mutual information; labeling the valuable sample subset to obtain labeled data; adjusting and training the pre-trained feature encoder according to the labeled data and the valuable sample subset to obtain a task model, and identifying the multi-modal medical images through the task model to obtain a multi-modal medical image recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to an active learning method for multi-modal medical images. BACKGROUND

[0002] Multi-modal medical images can show different characteristics of the same anatomical structure or lesion, and can provide rich complementary information, so they have very wide use in clinical practice. Deep learning has achieved great success in medical image computing, one of the reasons being that it can effectively utilize a large amount of labeled data. Labeling of medical images requires medical knowledge and expert experience, and is very time-consuming, and the differences and changes between different modalities in multi-modal medical images make it more difficult to label, for example, doctors need to compare the imaging results on different modalities several times to determine the exact area of some lesions, thereby further increasing the labeling cost. How to reduce the labeling cost and maintain the best possible performance is of great significance to medical image computing, especially for multi-modal medical image computing tasks.

[0003] Active learning (AL) is a method to reduce the labeling cost, and has been widely studied in medical image segmentation and classification. By selecting the most valuable unlabeled samples and labeling them by experts, AL aims to achieve the best possible performance with a small amount of labeled samples. Due to the universality of multi-modal data, there are active learning works using multi-modal data in both computer vision and medical image computing.

[0004] AL can be divided into iterative AL and one-shot AL according to the number of interactions with experts. In iterative AL, the entire system selects a small number of samples to interact with experts to obtain labels at each iteration, and performs calculations on all data. Iterative AL has high time and computational cost, and assumes that expert labeling can be obtained at any time, which is unrealistic. During the entire AL process, one-shot AL only needs the expert to label the selected samples once, so there is no need to interact with the expert during the training process of the DL model, thereby greatly reducing the labeling time and computational cost, but one-shot AL cannot learn features using labeling information and select valuable samples, so it needs to select appropriate feature learning methods and design completely different sampling strategies from iterative AL.

[0005] Selecting valuable samples is a core issue in algorithmic analysis (AL). Most methods rely on uncertainty to select valuable samples. These methods require a small number of labeled samples to train the initial model, and due to the tendency of neural networks to have excessively high confidence levels, the sample with the highest uncertainty is not necessarily the most valuable. Researchers have proposed various novel sampling strategies to overcome the drawbacks of uncertain sampling, such as representative sampling, core set sampling, and gradient-based sampling. However, for multimodal medical images, further research is needed on how to utilize the multimodal characteristics of the data during the sampling process to further reduce annotation costs. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides an active learning method for multimodal medical images, which can reduce manual annotation in the process of constructing deep learning-based medical image analysis, thereby reducing annotation costs.

[0007] To achieve the above technical objectives, the present invention provides the following technical solution: an active learning method for multimodal medical images, comprising:

[0008] Obtain the initial unlabeled dataset, which includes several modality sets, where each modality set is a collection of several samples;

[0009] Construct an encoder and pre-train it using a multimodal contrastive learning method to obtain a pre-trained feature encoder;

[0010] The pointwise mutual information between each sample modality in the initial unlabeled dataset is calculated by a pre-trained feature encoder, and a subset of valuable samples is obtained based on the pointwise mutual information.

[0011] Label the subset of valuable samples to obtain labeled data;

[0012] The pre-trained feature encoder is adjusted and trained based on labeled data and a subset of valuable samples to obtain a task model. The task model is then used to identify multimodal medical images, resulting in multimodal medical image recognition results.

[0013] Optionally, the process of obtaining an initial unlabeled dataset of multimodal medical images includes:

[0014] Unlabeled multimodal medical images were collected and preprocessed to obtain an initial unlabeled dataset.

[0015] Optionally, the pre-training process for deep neural networks includes:

[0016] The same random cropping and data augmentation are performed on the same sample under different modalities to obtain modal data; the modal data is independently encoded by an encoder to obtain independently encoded data; the dimensionality of the independently encoded data is reduced to obtain sample features;

[0017] The sample features are extracted to generate ordered sample feature combinations, which include positive sample feature pairs and negative sample feature pairs.

[0018] The sample features are stored in a memory bank, and the sample features in the ordered sample combination features are iteratively updated using a momentum update method until all ordered sample feature combinations are traversed. The contrastive learning loss between all two modalities is calculated based on all ordered sample feature combinations during the iterative update process. The multimodal contrastive learning loss is calculated based on the modal contrastive learning loss between the two modalities, and the deep neural network is optimized based on the multimodal contrastive learning loss.

[0019] Optionally, the contrastive learning loss between all two modalities can be calculated based on all ordered sample feature combinations during the iterative update process. include:

[0020]

[0021] in A positive sample feature pair is formed by the i-th sample in the first modality and the i-th sample in the second modality. Let h(·,·) be the negative sample feature pair formed by the i-th sample in the first modality and the j-th negative sample corresponding to the second modality, where h(·,·) is the similarity function, S is the index set of negative samples, N is the number of samples in each modality, and E is the number of samples in each modality. i Let be the expected loss over all samples.

[0022] Optionally, the multimodal contrastive learning loss is calculated. The process includes:

[0023]

[0024] in The comparison learning loss between the m-th mode and the n-th mode is denoted as . Let M be the contrastive learning loss between the nth mode and the mth mode, where M is the number of modes.

[0025] Optionally, the process of calculating pointwise mutual information between each sample modality in the initial unlabeled dataset includes:

[0026] The contrastive learning loss for all two modalities during the pre-training process of the pre-trained feature encoder is used to calculate the pointwise mutual information between each sample modality:

[0027]

[0028] Let k be the pointwise mutual information between sample modes, and k be the number of negative samples in the index set of negative samples.

[0029] The final point-to-point mutual information embedding is obtained by integrating the point-to-point mutual information, where the final point-to-point mutual information embedding is a set containing all point-to-point mutual information.

[0030] Optionally, the process of obtaining a subset of valuable samples includes:

[0031] The final pointwise mutual information embedding is sampled using the k-Center-Greedy algorithm to obtain a subset of valuable samples. The distance between pointwise mutual information in the final pointwise mutual information embedding is measured using L2 distance.

[0032] Optionally, the pre-trained feature encoder can be adjusted and trained based on labeled data and a subset of valuable samples to obtain an optimized network.

[0033] The pre-trained feature encoders are adjusted to obtain the initial network, which includes a segmentation network and a classification network. The segmentation network is a U-shaped segmentation network composed of pre-trained feature encoders, and the classification network is a classification network that is formed by sequentially concatenating all pre-trained feature encoders and connecting them to a fully connected layer.

[0034] The initial network is trained using labeled data and a subset of valuable samples to obtain the task model. During the training process, the classification network is trained using cross-entropy loss.

[0035] The present invention has the following technical effects:

[0036] This application proposes an active learning method for multimodal medical images that utilizes intermodal differences to select valuable samples for annotation. It introduces pointwise mutual information to quantify the differences between different modalities for each sample, employs two sampling strategies based on pointwise mutual information to select samples with more complementary information, uses multimodal contrastive learning to learn feature representations, and uses contrastive learning loss to estimate pointwise mutual information. The active learning method provided in this application has a clear process and can significantly reduce annotation costs while ensuring the performance of the target task, demonstrating both theoretical innovation and practical superiority. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of an unlabeled dataset provided in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of a multimodal contrastive learning pre-trained feature editor provided in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of PMI analysis provided in an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of expert annotation provided for an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the training task model provided in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] The purpose of this invention is to provide an active learning method that can reduce manual annotation in the process of constructing deep learning-based medical image analysis, thereby reducing annotation costs.

[0046] like Figure 1 As shown, to achieve the above objectives, this application provides an active learning method for multimodal medical images, specifically including the following steps:

[0047] 1) Collect a large number of unlabeled multimodal medical images and perform appropriate preprocessing to form an initial unlabeled dataset, such as multiparameter magnetic resonance imaging (mpMRI) or positron emission tomography / X-ray computed tomography (PET / CT).

[0048] 2) A deep neural network is pre-trained using multimodal contrastive learning as a feature encoder, and image patches cropped from the original images of samples of the same modality in the initial unlabeled dataset are used as the input of the feature encoder, i.e., modal data.

[0049] 3) Using a pre-trained feature encoder, estimate the intermodal mutual information of each sample in the initial unlabeled dataset. Based on the labeling budget, select a certain number of valuable samples through a sampling strategy based on the intermodal mutual information to form a subset of valuable samples.

[0050] 4) Have doctors label each sample in the valuable sample subset to obtain the labels for the valuable samples;

[0051] 5) Train and adjust the generation task model based on the pre-trained feature encoder. If the target task is a segmentation task, train a multi-encoder segmentation network; if the target task is a classification task, train a multi-modal classification network.

[0052] Since the number of valuable samples is much smaller than the number of samples in the entire dataset, this application can significantly reduce the number of samples that need to be labeled while ensuring the performance of the target task, thereby reducing the workload of manual labeling and lowering the labeling cost.

[0053] 1. Construct an unlabeled dataset, such as Figure 2 As shown, after collecting a sufficient amount of unlabeled images, necessary preprocessing is performed to form the initial unlabeled dataset.

[0054] An initial unlabeled multimodal medical image dataset can be represented as a set of M modalities X = {V} 1 V 2 , ..., V M Each modality is a set of N samples.

[0055] 2. Use multimodal contrastive learning to pre-train a deep neural network feature encoder, such as... Figure 3 As shown.

[0056] Image patches cropped from the original image are used as input to a feature encoder. The encoder extracts semantic features from these patches, resulting in shorter feature distances between different modalities of the same sample and wider feature distances between different samples. The same random cropping and data augmentation are applied to all modalities of the same sample, denoted as T. i (·) represents the modal data of the same sample, which is then processed using M encoders f. m Each modality is encoded independently, and an independent projection head g is superimposed after each encoder. m To further reduce the dimensionality, we finally obtain Sample features It represents the i-th sample in the m-th mode.

[0057] The core idea of ​​multimodal contrastive learning is to bring positive sample feature pairs closer together. Distance between them and push away k negative sample feature pairs The distance between the two modalities is given by S, where S is the set of indices of the negative samples, and |S| = k. Contrast learning is performed using features from all samples in the unlabeled dataset, i.e., k = N-1. The multimodal contrastive learning loss between the two modalities is:

[0058]

[0059] Where h(·, ·) is the similarity function, and the cosine similarity with temperature τ is used to measure whether two features are similar, defined as follows:

[0060]

[0061] Use a memory bank. Features of all samples are stored according to different modalities. Features in the memory bank are updated using momentum, with the update rule being... Where η is momentum, and η = 0.9 is set to ensure convergence. When M > 2, the contrastive learning of two modalities can be generalized to the contrastive learning of multiple modalities, traversing all... By combining a number of ordered modalities, the final loss, i.e., the multimodal contrastive learning loss, can be obtained:

[0062]

[0063] The feature encoder is trained using the final loss described above.

[0064] 3. Use pointwise mutual information to estimate the differences between modes in each unlabeled sample.

[0065] First, the lower bound of mutual information between any two modalities is estimated using the loss of multimodal contrastive learning. Then, based on this lower bound, pointwise mutual information between the images of any two modalities for each sample is obtained through ordered traversal, forming a pointwise mutual information embedding. Pointwise mutual information (PMI) can quantify modal differences at the sample level, thus valuable samples can be selected using PMI. However, calculating PMI for high-dimensional data is very difficult. This invention estimates MI through the close relationship between multimodal contrastive learning and mutual information (MI), ultimately obtaining an estimate of PMI. Minimizing the loss of two-modal multimodal contrastive learning is equivalent to maximizing the lower bound of MI between the features of the two modalities. Therefore, the lower bound of MI can be used as an estimate of mutual information MI, i.e., ...

[0066]

[0067] The larger k is, the more accurate the MI estimate. The estimate of mutual information MI is... NCE (z m ;z nThe truth value of replacing mutual information can be obtained by the following formula:

[0068]

[0069] Ultimately, MI's point-to-point mutual information As shown in the following formula:

[0070]

[0071] Because the samples have been cropped before being input into the feature encoder, the features... This invention only contains partial information about the samples; it uses the features of each sample in the memory bank as... The representation of Right now In fact The moving average during training, and due to the randomness of the clipping, we can consider... It contains all the information for the sample. Each sample can yield a total... Intermodal PMI estimation, Abbreviated as This ultimately constitutes point-to-point mutual information embedding I i ,Right now

[0072]

[0073] 4. Two sampling strategies based on point-by-point mutual information are used to select unlabeled samples with labeling value.

[0074] Able to reflect and The amount of information shared Smaller values ​​indicate less shared information, reflecting more unique information for each modality. Since the unique information of each modality complements each other, samples with more complementary information are more valuable for various downstream tasks. Therefore, the following two sampling strategies are used:

[0075] like Figure 4 As shown, (1) the average mutual information is minimized (AvgPMI-min). The core idea of ​​this strategy is that a large modal difference represents more complementary information, and more complementary information is more helpful for downstream tasks. This strategy first obtains the pointwise mutual information embedding I of all samples through the loss of multimodal contrastive learning. i Then, the average mutual information embedding of each sample in the pointwise mutual information embedding is calculated, and finally, the q samples with the smallest average pointwise mutual information are selected and sent to experts for annotation.

[0076] (2) Mutual Information Diversity Sampling (PMI-diversity). This method increases the diversity in the sampling process while minimizing the average mutual information. The method first calculates the mutual information embedding of all samples in the pointwise mutual information embedding, and finally uses the k-Center-Greedy algorithm to sample q samples between the mutual information embeddings of the samples. The distance between two mutual information embeddings is measured by L2 distance.

[0077] 5. Experts label valuable samples.

[0078] like Figure 5 As shown, a subset of valuable samples is sent to experts for annotation. If the target task is segmentation, the annotation is pixel-by-pixel segmentation annotation; if the target task is classification, the annotation is category annotation.

[0079] 6. Training the task model

[0080] like Figure 6 As shown, the task model is trained based on a pre-trained feature encoder. After obtaining expert annotations for valuable samples, the task model is constructed. First, an initial network is built with a certain structure using a pre-trained encoder. The initial network is then trained and optimized using a subset of valuable samples to generate the task model. If the target task is segmentation, the initial network is constructed as a multi-encoder segmentation network. The encoder uses a pre-trained encoder, and a structurally symmetrical decoder is added after the encoder to form a multi-encoder U-shaped segmentation network. After loading the pre-trained weights from multimodal contrastive learning, the encoder is trained using a subset of valuable samples and a combination of Dice and cross-entropy loss. If the target task is classification, the initial network is constructed as a multi-encoder classification network. The outputs of all encoders are concatenated and input into a fully connected layer to generate classification prediction results. The classification network is trained using cross-entropy loss. The final task model is then used to recognize multimodal images, completing the recognition task for multimodal images.

[0081] This application has the following beneficial effects:

[0082] This application proposes an active learning method based on multimodal medical images that utilizes intermodal differences to select valuable samples for annotation. It introduces pointwise mutual information to quantify the differences between different modalities for each sample, employs two sampling strategies based on pointwise mutual information to select samples with more complementary information, uses multimodal contrastive learning to learn feature representations, and uses contrastive learning loss to estimate pointwise mutual information. The active learning method provided in this application has a clear process and can significantly reduce annotation costs while ensuring the performance of the target task, demonstrating both theoretical innovation and practical superiority.

[0083] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An active learning method for multimodal medical images, characterized in that, include: Obtain an initial unlabeled dataset, which includes several modality sets, wherein each modality set is a set of several samples; Construct an encoder and pre-train it using a multimodal contrastive learning method to obtain a pre-trained feature encoder; The pointwise mutual information between each sample modality in the initial unlabeled dataset is calculated by a pre-trained feature encoder, and a subset of valuable samples is obtained based on the pointwise mutual information. Label the subset of valuable samples to obtain labeled data; The pre-trained feature encoder is adjusted and trained based on labeled data and a subset of valuable samples to obtain a task model. The task model is then used to identify multimodal medical images to obtain multimodal medical image recognition results. The process of pre-training a deep neural network includes: The same random cropping and data augmentation are performed on the same sample under different modalities to obtain modal data; the modal data is independently encoded by an encoder to obtain independently encoded data; the dimensionality of the independently encoded data is reduced to obtain sample features; The sample features are extracted to generate ordered sample feature combinations, which include positive sample feature pairs and negative sample feature pairs. The sample features are stored in a memory bank, and the sample features in the ordered sample combination are iteratively updated and extracted using a momentum update method until all ordered sample feature combinations are traversed. The contrastive learning loss between all two modalities is calculated based on all ordered sample feature combinations in the iterative update and extraction process. The multimodal contrastive learning loss is calculated based on the modal contrastive learning loss between the two modalities. The deep neural network is optimized based on the multimodal contrastive learning loss. Calculate the contrastive learning loss between all two modalities based on all ordered sample feature combinations during the iterative update process. include: in A positive sample feature pair is formed by the i-th sample in the first modality and the i-th sample in the second modality. The negative sample feature pair consists of the i-th sample in the first mode and the j-th negative sample corresponding to the second mode. For similarity function, Let N be the set of indices for negative samples, and E be the number of samples in each modality. i Let be the expected loss over all samples; The multimodal contrastive learning loss is calculated. The process includes: ； in The contrastive learning loss between the m-th modality and the n-th modality; The contrastive learning loss between the nth mode and the mth mode is denoted by M, where M is the number of modes. The process of calculating the pointwise mutual information between each sample mode in the initial unlabeled dataset includes: The pointwise mutual information between each sample mode is calculated based on the contrastive learning loss of all two modalities during the pre-training process of the pre-trained feature encoder. Let k be the pointwise mutual information between sample modes, and k be the number of negative samples in the index set of negative samples. The final point-to-point mutual information embedding is obtained by integrating the point-to-point mutual information, wherein the final point-to-point mutual information embedding is a set containing all point-to-point mutual information.

2. The active learning method for multimodal medical images according to claim 1, characterized in that: The process of obtaining the initial unlabeled dataset of multimodal medical images includes: Unlabeled multimodal medical images were collected and preprocessed to obtain an initial unlabeled dataset.

3. The active learning method for multimodal medical images according to claim 1, characterized in that: The process of obtaining a subset of valuable samples includes: The final pointwise mutual information embedding is sampled using the k-Center-Greedy algorithm to obtain a subset of valuable samples, wherein the distance between pointwise mutual information in the final pointwise mutual information embedding is measured using L2 distance.

4. The active learning method for multimodal medical images according to claim 1, characterized in that: The pre-trained feature encoder is adjusted and trained based on labeled data and a subset of valuable samples to obtain an optimized network; The pre-trained feature encoder is adjusted to obtain an initial network, which includes a segmentation network and a classification network. The segmentation network is a U-shaped segmentation network composed of pre-trained feature encoders, and the classification network is a classification network that is formed by sequentially concatenating all pre-trained feature encoders and connecting them to a fully connected layer. The initial network is trained using labeled data and a subset of valuable samples to obtain the task model. During the training process, the classification network is trained using cross-entropy loss.

Images