A multi-modal collaborative learning gastroscope image semi-supervised classification method

Abstract

The present application relates to a kind of multi-modal collaborative learning gastroscope image semi-supervised classification method, belong to image processing technical field.The method, the training process of unlabeled sample is separated from the training process of labeled sample, and a kind of self-supervised pre-training strategy based on consistency learning, recovery learning and alignment learning collaborative learning is proposed.In order to realize multi-modal feature fusion on labeled sample, a kind of fine-tuning strategy based on feature distribution drift is proposed.In order to verify the effectiveness of the method of the present application, extensive experiments are carried out on two public gastroscope image datasets Kvasir and Kvasir v2.The method of the present application can have good classification performance for the classification of gastroscope image using only a small amount of labeled data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a gastroscope image semi-supervised classification method based on multi-modal collaborative learning. BACKGROUND

[0002] With the rapid development of medical imaging technology, medical images play an increasingly important role in clinical diagnosis and treatment. However, single modal medical image information is limited, and it is often difficult to fully reflect the characteristics of the lesion, thereby affecting the accuracy of diagnosis. Multi-modal medical images provide more diverse lesion information by fusing different imaging technologies, which can provide more reliable basis for clinical diagnosis and disease analysis. For example, during a clinical gastroscopy, doctors of the department of gastroenterology usually first use a white light source to explore the patient's intestines and stomach to roughly judge the suspicious area. White light endoscopy imaging uses all visible waves to generate red, green and blue images, which is more sensitive in terms of checking morphology, mucosa and blood vessel distribution, and helps doctors to find gastric ulcers and tumors, which is the standard mode of current endoscopic imaging. When a suspicious area is found, the doctor will switch to narrow-band imaging to further determine the possible disease type. Narrow-band imaging uses a specific filter to generate blue and green wavelength light, which can enhance the mucosal features and blood vessel structure of the suspicious area, and help to locate early gastric cancer.

[0003] Multi-modal visual tasks have received extensive attention from researchers in recent years, and multi-modal representation effectively utilizes the feature information of different modalities to improve the performance of downstream tasks. For example, ALBEF aligns the feature representations of images and texts through cross-modal attention. CCR-Net reconstructs remote sensing data across modalities to learn more compact feature fusion representations. DeepGuide uses the knowledge learned in the superior modality to guide the use of the inferior modality, effectively improving the diagnostic performance of the low-level modality. FusionM4Net adopts a two-stage multi-modal learning method to effectively fuse the feature-level representations of clinical images and dermatoscope images. GiMP considers modeling high-order correlations and proposes a group multi-head self-attention gene encoder to capture the global structure of gene expression. FusAtNet simultaneously uses multispectral images and hyperspectral images to realize land cover classification. MSAN focuses on modality specificity and uses two attention modules to focus on fundus images and OCT images to guide eye disease classification.

[0004] However, due to the difference and redundancy between the modal images, how to effectively fuse the multi-modal image information has become a difficult problem to be solved. In recent years, deep learning technology has made significant progress in medical image analysis, especially in feature extraction and pattern recognition. As a data-driven end-to-end method, the deep learning model needs a large amount of labeled data support. However, these data labeled by professional doctors are difficult to obtain and have low labeling efficiency. Especially in multi-modal tasks, doctors need to repeatedly switch between multiple modalities to ensure the accuracy of data labeling.

[0005] In summary, the challenges of multi-modal based medical image tasks are as follows: (1) multi-modal medical images provide multiple lesion information, but it is still difficult to effectively fuse features of different modalities; (2) in multi-modal medical image tasks, it is more time-consuming and laborious to obtain professional physician labeled data than single modal.

[0006] In addition, the current semi-supervised classification method relies on the consistency prediction of strong and weak enhancement of unlabeled images. Although common strong enhancement methods such as RandAugment, CTAugment, etc. have been successful in perturbing natural images (e.g., cats, airplanes), these combined enhancement methods can easily destroy the lesion features of medical images and it is difficult to find a suitable enhancement combination. Inspired by these semi-supervised classification methods, we believe that when processing medical images, one modality of the same disease type can be the best enhancement sample for another modality and predict consistency. In addition, the scale of multi-modal semi-supervised image classification data is usually small, and the supervised learning of labeled samples and the consistency prediction of unlabeled samples are easy to affect each other, thereby affecting the classification performance. SUMMARY

[0007] The purpose of the present application is to provide a multi-modal collaborative learning gastroscopy image semi-supervised classification method, which can have good classification performance for gastroscopy image classification using only a small amount of labeled data.

[0008] To achieve the above purpose, the technical scheme of the present application is: a multi-modal collaborative learning gastroscopy image semi-supervised classification method, comprising:

[0009] Pre-training stage: a self-supervised pre-training strategy based on consistency learning, recovery learning and alignment learning collaborative learning is constructed, and a pre-training stage model is trained;

[0010] Dictionary construction stage: a drift vector dictionary is constructed according to the feature representation learned by the pre-training stage model;

[0011] The multimodal fusion fine-tuning stage: the multimodal fusion fine-tuning model is initialized by using the pre-training weight, and the distribution drift generated by the drift vector dictionary is used to provide additional training samples for the multimodal fusion fine-tuning model.

[0012] Further, the pre-training stage is implemented as follows:

[0013] For the input paired modal images , is the white light image, is the narrowband light image. The white light image features and the narrowband image features are extracted by the initially set encoder and the encoder respectively; the white light image features and the narrowband image features are aligned in the low-dimensional feature space through consistency learning, and the consistency loss is calculated; through restoration learning, the paired modal images are randomly masked to obtain , is the randomly masked white light image, is the randomly masked narrowband light image, features and are extracted by the initially set encoder and the encoder respectively, and are reconstructed by the unified decoder to calculate the reconstruction loss ; the alignment learning projects and to the high-dimensional latent feature space through a global embedding and calculates the alignment loss ; finally, the pre-training stage loss function composed of the consistency loss, the reconstruction loss and the alignment loss is constructed, and the trained pre-training encoder and the pre-training encoder are obtained.

[0014] Further, the pre-training stage loss function is represented as follows:

[0015]

[0016] wherein , and represent the weighting coefficients of the corresponding loss.

[0017] Further, the drift vector dictionary is constructed as follows:

[0018] Based on the input paired modal images According to the pre-trained encoder and the pre-trained encoder Extract the unlabeled sample features:

[0019]

[0020]

[0021] Wherein represents the total number of training set samples, and respectively represent the pre-trained encoder and the pre-trained encoder extracted white light image features and narrowband light image features, and respectively represent the first white light image and the first narrowband light image in the training set;

[0022] The extracted unlabeled sample features are clustered by K-means to generate clusters:

[0023]

[0024] Wherein is the clustering label, , represents the number of classification categories, is the prototype vector of each cluster, is an indicator function;

[0025] For each cluster Calculate the sample mean to update the prototype is:

[0026]

[0027] Wherein, represents the value when , is the number of samples in cluster , and the sample covariance matrix in cluster is represented as:

[0028]

[0029] According to the covariance matrix Calculate the drift vector dictionary, that is, the mean is , covariance P drift vectors are sampled in the multivariate Gaussian distribution , the drift vector is defined as:

[0030]

[0031] Where p represents the drift vector index taken, P represents the P drift vectors sampled in the multivariate Gaussian distribution; the drift vector dictionary of all clusters is , represents the dimension of the original input feature.

[0032] Further, the multimodal fusion fine-tuning stage is implemented as follows:

[0033] Data with class labels , through the initially set encoder and the encoder , extract the corresponding modal features and , through the multimodal encoder fusion feature representation to get multimodal fusion feature , and According to the drift vector dictionary, the prototype and the corresponding drift vector and , and the drift vector and are added element by element to get the distribution drift feature based on knowledge transfer and ; the loss function of the multimodal fusion fine-tuning stage is defined as follows:

[0034]

[0035] Where represents the class label, represents the classification head, represents the cross-entropy function.

[0036] Further, the multimodal fusion fine-tuning stage also introduces a kind of dynamic confidence fusion method TMC for multi-view classification, to optimize the loss function of the multimodal fusion fine-tuning stage, and the optimized loss function of the multimodal fusion fine-tuning stage is defined as follows:

[0037]

[0038] is the fusion loss of white light samples and narrowband light samples, is the fusion sample loss.

[0039] Further, are defined as follows:

[0040]

[0041] wherein, and are the predictive probabilities of the white light image and the narrowband light image, and are the mean values of the Dirichlet distribution of the white light image and the narrowband light image respectively, and respectively represent the Dirichlet distribution after the true value label is replaced by 1 for the white light image class and the narrowband light image class, and further avoid the Dirichlet parameter of the basic true value class being 1, represents the probability density of calculating the Dirichlet distribution, represents the KL divergence.

[0042] Further, are defined as follows:

[0043]

[0044] wherein, is a parameter for controlling the KL divergence, preventing the parameter space from outputting a flat uniform distribution due to excessive emphasis on the KL divergence in the network training process, is the mean value of the fused modal Dirichlet distribution, represents the Dirichlet distribution after the true value label is replaced by 1 for the fused modal.

[0045] The application further provides a computer readable storage medium, which stores computer program instructions capable of being executed by a processor, and when the processor executes the computer program instructions, the steps of the method according to any one of the above embodiments can be implemented.

[0046] Compared with the prior art, the present application has the following beneficial effects: the method of the present application separates the training process of the unlabeled sample from the training process of the labeled sample, and proposes a self-supervised pre-training method based on consistency learning, recovery learning and alignment learning collaborative learning. In order to realize multi-modal feature fusion on the labeled sample, a fine-tuning method based on feature distribution drift is also proposed. Specifically, the present application uses a self-supervised training method in the pre-training stage. Consistency learning regards multiple modalities as different enhanced sample predictions to learn the consistency of the unlabeled sample feature representation. Recovery learning extracts features from different encoders to a unified decoder to reconstruct the image to focus on the local detail representation of different modalities after randomly masking the original image. Alignment learning forces two modalities to approach in the high-dimensional feature space through instance-level contrastive learning, further improving the representation similarity of the paired images. In the fine-tuning stage, the present application proposes a multi-modal encoder structure to fuse the feature representations of different modalities. In addition, in order to prevent the overfitting risk caused by too few labeled samples in the fine-tuning stage, the present application uses the pre-training weight to construct a base class dictionary for modality clustering. The present application takes the feature output by the multi-modal encoder as the prototype, and obtains the perturbation vector from the base class dictionary. Subsequently, the prototype feature generates a perturbation feature close to the initial modality according to the perturbation vector, and fills the gap between the initial modality and the fused modality in the feature space through distribution drift. In order to verify the effectiveness of the proposed method, the present application carries out extensive experiments on two public gastroscope image datasets Kvasir and Kvasirv2. The experimental results show that the MICS proposed by the present application can have good classification performance for gastroscope image classification using only a small amount of labeled data. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The overall framework of the collaborative learning pre-training of the present application.

[0048] Figure 2 The overall framework of the multi-modal fusion fine-tuning of the present application.

[0049] Figure 3 The white light image, Figure 3 In (a), a normal pylorus sample, Figure 3 In (b), a sample of a normal pylorus sample after random enhancement, Figure 3 In (c), an esophagitis sample, Figure 3 In (d), a sample of a normal pylorus sample and a polyp image after mixed enhancement.

[0050] Figure 4 The overall process of dynamic evidence fusion.

[0051] Figure 5Sample examples for Kvasir and Kvasirv2 datasets, where the first row is the true white light image and the second row is the paired narrow-band light image generated by WtNGAN.

[0052] Figure 6 Feature visualization results on Kvasir v2 for each algorithm trained with only 5% of the labeled data. DETAILED DESCRIPTION

[0053] The technical solutions of the present application will be described in detail below with reference to the drawings.

[0054] The present application provides a multi-modal collaborative learning endoscopic image semi-supervised classification method, comprising:

[0055] Pre-training stage: a self-supervised pre-training strategy based on consistency learning, recovery learning and alignment learning collaborative learning is constructed, and a pre-training stage model is trained;

[0056] Dictionary construction stage: a shift vector dictionary is constructed according to the feature representation learned by the pre-training stage model;

[0057] Multi-modal fusion fine-tuning stage: the pre-training weight is used to initialize the multi-modal fusion fine-tuning model, and the distribution shift generated by the shift vector dictionary is used to provide additional training samples for the multi-modal fusion fine-tuning model.

[0058] The following is the specific implementation process of the present application.

[0059] The training process of the multi-modal collaborative learning endoscopic image semi-supervised classification method MICS proposed by the method of the present application is divided into a pre-training stage, a dictionary construction stage and a multi-modal fusion fine-tuning stage. The pre-training stage includes consistency learning, recovery learning and alignment learning. The dictionary construction stage constructs a Shift Vector dictionary (shift vector dictionary) according to the feature representation learned by the pre-training stage model. The multi-modal fusion fine-tuning stage initializes the model using the pre-training weight, and provides additional training samples for the fusion model according to the distribution shift generated by the Shift Vector dictionary.

[0060] 1. Collaborative learning pre-training (i.e. pre-training stage)

[0061] The proposed collaborative learning pre-training framework is shown in Figure 1 During the pre-training stage, features of different modalities are extracted by single-modal encoders. For the input paired modal images , is the white light image, is the narrow-band light image, the collaborative learning sets the encoder and to extract white light image features and narrowband image features . These latent features are first aligned in a low-dimensional feature space by consistency learning, computing a consistency loss . Then the original images are randomly masked , and are the white light image and narrowband light image after random masking respectively. Similarly, features and are extracted by the encoder and are reconstructed by a unified decoder to compute a reconstruction loss . Finally, to align the high-dimensional feature representations of different modalities, and are projected to a high-dimensional latent feature space by a global embedding and a alignment loss is computed.

[0062] 1.2 Consistency learning

[0063] Inspired by image-text contrastive learning, the present invention proposes an alignment method before the fusion of different modalities. Similar to the momentum contrast learning method MOCO, a number of image pairs of features are stored in two different queues as negative samples, is the negative sample of white light image features in the queue, is the negative sample of narrowband light image features in the queue. During training, this number of image feature pairs are mapped to a low-dimensional space by a mapping head and their similarity is computed:

[0064]

[0065]

[0066] where and are the image features extracted by the momentum model maintained by the Exponential Moving Average (EMA), , , denote the temperature. denotes the feature concatenation of channel dimension, the superscript denotes the matrix transpose operation, denotes after transposition and a white light image feature negative sample formed by concatenating the channel dimension, denotes after transposition and a narrowband light image feature negative sample formed by concatenating the channel dimension.

[0067] Then, a target similarity matrix is constructed and the target similarity is calculated:

[0068]

[0069]

[0070] wherein, denotes the similarity matrix of the white light image feature in the low-dimensional space and the narrowband light image feature, denotes the similarity matrix of the narrowband light image feature in the low-dimensional space and the white light image feature, is a hyperparameter used to control the ratio of the similarity matrix and the target similarity matrix , and denotes a function that maps a numerical value to a value between 0 and 1.

[0071] Therefore, the similarity of the feature of the current image pair and the feature in the momentum queue is

[0072]

[0073]

[0074] wherein, denotes the total number of sample pairs, denotes the th sample feature of a batch of data, denotes the th feature in the momentum queue, denotes the loss of matching the target similarity of the narrowband light image using the logarithmic probability of the white light image similarity, denotes the loss of matching the target similarity of the white light image using the logarithmic probability of the narrowband light image similarity.

[0075] The final consistency loss is defined as

[0076]

[0077] 1.3 Recovery learning

[0078] Masked autoencoder is considered as an effective self-supervised learning method, so a generative learning method is set in the collaborative learning pre-training to focus on the local detail representation between different modalities. Unlike the masked autoencoder, a unified decoder is set in the multimodal task for modal feature interaction, that is, whether it is an encoder or an encoder extracted image features, the decoder can reconstruct the original image. This is similar to the process of adversarial learning. The present application hopes that different modalities will only align the key region information in the feature space rather than be completely consistent, so a unified decoder helps different encoders to retain the representation ability of corresponding modal features. Specifically, the masked feature and are reconstructed into the original image by the decoder and the reconstruction loss is calculated . The process of obtaining the masked feature is represented as:

[0079]

[0080] wherein, represents random masking, represents the masking ratio. Therefore, the reconstruction loss is defined as:

[0081]

[0082] 1.4 Alignment learning

[0083] For medical images, high-dimensional vectors are beneficial to the model to richly represent features to learn small and difficult-to-detect lesions. Specifically, a global embedding is set to project and to high-dimensional latent feature vectors and . Alignment learning aligns the representations through instance-level contrastive learning with the InfoNCE loss optimization objective. The similarity between the embedding vectors of the white light image and the narrowband light image is calculated as follows:

[0084]

[0085]

[0086] wherein, and represent the white light image features and the narrowband light image features, and Let represent the high-dimensional latent feature space vectors obtained after global embedding of white light image features and narrowband light image features, respectively. Represents cosine similarity. Indicates temperature. and Indicates positive instance pairs, Represents the first positive instance White light image features, and Indicates the first positive instance The narrowband light image features, and and They represent the first negative instance, respectively. White light image features and narrowband light image features This represents the total number of samples in a batch of data. This represents the first similarity metric calculated at the current time. One sample, Let represent an exponential function. Therefore, the optimization objective of alignment learning is defined as follows:

[0087]

[0088] in Represents the cross-entropy function. This represents the one-hot label of an instance-level sample.

[0089] 1.5 Loss Function in the Pre-training Stage

[0090] In summary, the overall optimization objective loss function of the proposed method in the pre-training phase is expressed as follows:

[0091]

[0092] in , and This represents the weighting coefficient for the corresponding loss.

[0093] 2. Drift vector dictionary

[0094] Explicit enhancement methods such as Mixup and RandAugment are prone to producing unreliable enhanced samples when processing medical data. Figure 3 As shown, this invention provides an example of a normal white light image. In common semi-supervised classification methods, typically... Figure 3 The weakly enhanced version of the prediction in (a) is used as a pseudo-label and forces the model to predict... Figure 3 (b) are of the same category, which is true for this pair. Figure 3The prediction in (c) has an impact. The samples augmented by RandAugment are intuitively closer to the real Esophagitis, and even the well-trained ViT-B / 16 was misclassified. However, the Mixup with label information, which mixes normal-pylorus and polyps, was identified as Esophagitis by the model.

[0095] Therefore, this invention considers designing an implicit enhancement scheme in an unsupervised manner. This avoids introducing potentially ambiguous semantic information while transferring transferable knowledge from existing data. This invention proposes an implicit feature enhancement scheme that relies on the drift of the base class dictionary distribution. This invention is applicable to multimodal tasks, guiding the fused features towards the initial modality by generating drift components through the sample distribution of the base class dictionary.

[0096] Specifically, this invention utilizes a fully trained encoder. and Because the intermediate layers of the encoder operate independently during the pre-training phase, therefore and The ability to map the original modality distribution is preserved. First, the encoder extracts features from unlabeled samples:

[0097]

[0098]

[0099] in, and The features of unlabeled white light images and unlabeled narrowband light images are described respectively. Indicates the first A sample of unlabeled white light images. Indicates the first A sample of a standard-free narrowband light image. This represents the total number of samples in the training set. Then, the extracted features are generated using K-means clustering. Clusters:

[0100]

[0101] in For clustering labels, . Indicates the number of categories. For each cluster, the prototype vector, For indicator functions, express The th cluster Clusters, when clustering labels With the When the labels of the clusters are the same, then ,otherwise Then, for each cluster Calculate sample mean to update prototype for:

[0102]

[0103] in, Indicates when time value, For clusters The number of samples in the cluster. Therefore, the cluster The sample covariance matrix in the data can be represented as:

[0104]

[0105] Finally, based on the covariance matrix Calculate the drift vector dictionary, i.e., from the mean value... Covariance is Sampling P drift vectors in a multivariate Gaussian distribution drift vector Defined as:

[0106]

[0107] Where p represents the index of the selected drift vector, and P represents the number of drift vectors sampled in the multivariate Gaussian distribution; the drift vector dictionary for all clusters is... , This represents the dimension of the original input features.

[0108] 3. Multimodal fusion fine-tuning

[0109] Multimodal fusion fine-tuning, such as Figure 2 As shown, the fine-tuning phase includes the initial modality encoder provided by the co-learning pre-training. and Multimodal encoder Using a small amount of data with category labels. The multimodal fusion fine-tuning model in the fine-tuning phase. Specifically, and Extract corresponding modal features and Then by Multimodal fusion features are obtained from the fusion feature representation. .Then, and The prototype is randomly searched based on the constructed initial modal drift vector dictionary. and the corresponding drift vector and Finally, with the drift vector and element-wise addition to obtain the distribution drift feature based on knowledge transfer and Therefore, the loss function of the proposed method in the multi-modal fusion fine-tuning stage is defined as follows:

[0110]

[0111] wherein represents the class label, represents the classification head.

[0112] The present application hopes to realize the information interaction between modalities and remove redundant features through the cross attention of the multi-modal fusion encoder in the fine-tuning stage. In addition, the proposed implicit data augmentation method based on distribution drift avoids the influence of ambiguous semantic information on model training on the one hand, and does not introduce additional parameter calculation on the other hand, expands the training data and alleviates the overfitting risk caused by the lack of labeled samples.

[0113] 4. Evidence fusion based on uncertainty

[0114] Inspired by multi-view classification, the present application considers that, in addition to feature fusion, evidence fusion based on uncertainty estimation can improve the reliability of multi-modal classification. As shown in Figure 4 , the present application introduces a dynamic confidence fusion method Trusted Multi-view Classification (TMC) for multi-view classification in the fine-tuning stage of gastroscope multi-modal classification. It is worth noting that the present application does not modify the theory of TMC, but extends it from the multi-view classification task to the field of gastroscope multi-modal classification, providing new insights for subsequent multi-modal classification tasks.

[0115] Specifically, the present application obtains the concentrated parameters of the Dirichlet distribution of the fine-tuning model for each modal sample through TMC:

[0116]

[0117] Therefore, the Dirichlet distribution is represented as , The mean of the Dirichlet distribution is . Then, the confidence quality and the uncertainty of the class are defined as:

[0118]

[0119]

[0120] where is the number of classification categories. For samples of two modalities, TMCis generated for white light images and is generated for narrow-band light images and So, the probability mass distribution for different modalities is defined as:

[0121]

[0122]

[0123] Given the above mass of fine-tuning models for each modality, these confidences and uncertainties from different modalities are fused:

[0124]

[0125] The fine-tuning model is provided with a decision layer-based fusion, and the additional modal complementarity optimization objective at the representation layer can promote information interaction between modalities. Therefore, the final fine-tuning loss is defined as follows:

[0126]

[0127] where as shown in equation (15). is the fusion loss of white light samples and narrow-band light samples, defined as follows:

[0128]

[0129] where, and are the predicted probabilities of white light images and narrow-band light images, and are the average values of the Dirichlet distribution of white light images and narrow-band light images, respectively, and represent the Dirichlet distribution after replacing the true value label with 1 for white light image classes and narrow-band light image classes, respectively, thereby avoiding the punishment of the Dirichlet parameter of the basic true value class being 1, represents the probability density of calculating the Dirichlet distribution, represents the KL divergence.

[0130]

[0131] where, the average value of the Dirichlet distribution of the fusion modal, represents the Dirichlet distribution after the fusion modal replaces the true value label with 1.

[0132] 5、Experiment

[0133] Since there is currently a lack of publicly available and paired multi-modal medical image datasets, the present application uses the publicly available gastroscope white light image dataset Kvasir and Kvasir v2, and the paired narrow band light image of each image is generated by WtNGAN and the multi-modal data used by all comparison methods are the same. An example of the dataset is shown in Figure 5 As shown in the figure, the Kvasir dataset contains three types of normal stomach dissection structures and three types of common stomach disease images, and the paired NBI images generated by the algorithm can clearly show the blood vessels and mucosal details. The unified image resolution of the present application is 256x256, and the training set, validation set and test set are divided according to the ratio of 6:2:2. The detailed composition of the used dataset is shown in Table 1.

[0134] Table 1 Detailed composition of the used dataset

[0135]

[0136] In addition, the comparison algorithms used are as follows:

[0137] FixMatch consistency regularization is applied to the pseudo-label method, and a fixed confidence threshold is set and the consistency of different enhanced samples of the unlabeled sample is predicted.

[0138] FlexMatch dynamically adjusts the pseudo-label confidence threshold of each class of samples according to the learning state of the model for the unlabeled samples.

[0139] Dash selects a subset of training samples to screen out false pseudo-label samples based on dynamic adjustment of the threshold during the iteration process.

[0140] FreeMatch sets a lower pseudo-label threshold in the early training stage to accelerate model convergence, and increases the threshold in the later training stage to exclude false pseudo-label predictions.

[0141] SoftMatch focuses on the quality-quantity trade-off problem of pseudo-label samples, and maintains a large number of high-quality pseudo-labels during the training process to effectively use the unlabeled samples.

[0142] HABIT proposes a consistency-aware momentum genetic algorithm to reduce the bias in pseudo-label sample selection.

[0143] SIAVC proposes a sample super-enhancement module and a cross-set sample enhancement module, so that the model optimization benefits from the samples that have been sufficiently learned.

[0144] 5.1 Implementation details

[0145] In the pre-training stage, the initial learning rate of the model is set to and is gradually decayed to using the cosine schedule. The AdamW optimizer with a weight decay coefficient of 0.02 is used in the present invention, with a momentum parameter set to 0.995, and 100 rounds of training. The parameter settings in the fine-tuning stage remain similar to those in the pre-training stage, except that the present invention sets the minimum learning rate to and the decay coefficient to 0.999. All methods are trained for 300 rounds to ensure model convergence. All methods are implemented in Python 3.9.13 and Pytorch 1.12.1. A computer equipped with Intel xenon silver 4208 CPU and seven NVIDIA GeForce RTX 3090 GPUs is used for model training and testing.

[0146] 5.2 Quantitative comparison

[0147] The quantitative experimental results are shown in Tables 2 and 3. The present invention conducted two groups of experiments under four label proportions. The results show that replacing the strong enhanced images with weak enhanced images of other modalities can effectively improve the semi-supervised classification performance, verifying the hypothesis of the present invention. It is worth noting that FixMatch based on multi-modal training significantly improves the accuracy rate using only 5% of the labeled data, which may be because it relies on rich unlabeled samples for optimization, avoids the wrong feature representation caused by excessive strong enhancement, and filters the wrong pseudo-labels using a high threshold. Other threshold-based methods show an accuracy improvement of between 0.17% and 18.50%, while the composite enhancement method shows a decrease in accuracy of between 0.17% and 5.83% under certain label proportions. Among all the results, the proposed MICS achieves the best classification effect using multi-modal data, reaching an accuracy of 70.25% on the Kvasir v2 dataset using only 5% of the labeled data.

[0148] Table 2 Quantitative experimental results on the Kvasir dataset

[0149]

[0150] Table 3 Quantitative experimental results on the Kvasir v2 dataset

[0151]

[0152] 5.3 Qualitative analysis

[0153] Qualitative experimental results as follows Figure 6 As shown, this invention provides UMAP feature visualization results for a comparative method trained on the Kvasir v2 dataset using only 5% of the labeled data. Experimental results show that almost all classification methods confuse normal cecum, ulcers, and polyps. On the one hand, the cecum is part of the colon, and ulcers and polyps are more likely to occur in the colonic region; on the other hand, due to the limited amount of labeled data, these methods fail to focus on lesion features and instead memorize the context in which the lesion occurs, which is easier to achieve than the former. Notably, in the feature visualization results of the method of this invention, normal cecum samples are clearly separated from ulcer or polyp samples in the feature space, indicating that the MICS proposed in this invention can distinguish the lesion location and the lesion itself well. MICS, SIAVC, and Dash can also distinguish normal pyloric structures and Z-lines. The pylorus and Z-lines have high structural similarity, and sensitivity to the pore-like black structures may cause the model to confuse the two, resulting in low discriminative power in the feature space. Finally, when labeled data is scarce, all methods cannot distinguish between normal Z-lines and esophagitis. From an anatomical perspective, the Z-line is located at the junction of the stomach and esophagus, and esophagitis is one of the most common pathological manifestations of the Z-line. Therefore, when esophagitis images show low-grade inflammation or low contrast, models trained with limited annotations have difficulty distinguishing between normal esophagus and mild esophagitis.

[0154] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions executable by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of any of the methods described above.

[0155] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A semi-supervised classification method for gastroscopy images using multimodal collaborative learning, characterized in that, include: Pre-training phase: Construct a self-supervised pre-training strategy based on the collaborative learning of consistency learning, restorative learning, and alignment learning to train the pre-trained model; the pre-training phase is implemented as follows: For the input paired modality image , A white light image. For narrowband light images, Through the initially set encoder and encoder Extracting features from white light images respectively and narrowband image features White light image features and narrowband image features After consistency learning, the representations are aligned in the low-dimensional feature space, and the consistency loss is calculated. ; By restorative learning, randomly masked paired modal images are obtained. get , The image shows the white light after random occlusion. This is a narrowband light image after random occlusion. Each is generated by the encoder initially set. and encoder Feature extraction and , and After a unified decoder Perform reconstruction to calculate reconstruction losses ; Alignment learning will and After a global embedding Project onto a high-dimensional latent feature space and compute alignment loss Finally, a pre-training stage loss function consisting of consistency loss, reconstruction loss, and alignment loss is constructed, resulting in the pre-trained encoder. and pre-trained encoder ; Dictionary construction phase: Construct the drift vector dictionary based on the feature representations learned by the model in the pre-training phase; the drift vector dictionary is constructed as follows: Input-based paired modality images According to the pre-trained encoder and pre-trained encoder Extract features from unlabeled samples: in This represents the total number of samples in the paired modality image training set. and These represent the pre-trained encoders. and pre-trained encoder Extracted white light image features and narrowband light image features, and These represent the first and second images in the paired modality image training set, respectively. The first white light image and the first A narrowband light image; The extracted unlabeled sample features are then generated using K-means clustering. Clusters: in For clustering labels, , Indicates the number of categories. For each cluster, the prototype vector, For indicator functions; For each cluster Calculate sample mean and update prototype vector for: in, Indicates when time value, For clusters Number of samples in the cluster The sample covariance matrix in the data is represented as follows: Based on the covariance matrix Calculate the drift vector dictionary, i.e., from the mean value... Covariance is Sampling P drift vectors in a multivariate Gaussian distribution drift vector Defined as: Where p represents the index of the selected drift vector, and P represents the number of drift vectors sampled in the multivariate Gaussian distribution; the drift vector dictionary for all clusters is... , The dimension representing the original input features; Multimodal fusion fine-tuning stage: Initialize the multimodal fusion fine-tuning model using pre-trained weights, and provide additional training samples for the multimodal fusion fine-tuning model based on the distribution drift generated by the drift vector dictionary.

2. The semi-supervised classification method for gastroscopy images based on multimodal collaborative learning according to claim 1, characterized in that, The loss function for the pre-training phase is expressed as follows: in , and This represents the weighting coefficient for the corresponding loss.

3. The semi-supervised classification method for gastroscopy images based on multimodal collaborative learning according to claim 1, characterized in that, The multimodal fusion fine-tuning stage achieves the following: Data with category labels Through the initially set encoder and encoder Extract corresponding modal features and After passing through a multi-mode encoder Multimodal fusion features are obtained from the fusion feature representation. , and Randomly find the prototype based on the drift vector dictionary. and the corresponding drift vector and , With drift vector and Element-wise addition yields distribution drift characteristics based on knowledge transfer. and The loss function for the multimodal fusion fine-tuning stage is defined as follows: in Indicates category label, Indicates the category header, This represents the cross-entropy function.

4. The semi-supervised classification method for gastroscopy images based on multimodal collaborative learning according to claim 3, characterized in that, In the multimodal fusion fine-tuning stage, a dynamic credibility fusion method (TMC) for multi-view classification is introduced to optimize the loss function of the multimodal fusion fine-tuning stage. The optimized loss function of the multimodal fusion fine-tuning stage is defined as follows: The fusion loss between white light samples and narrowband light samples, This is the loss from fused samples.

5. The semi-supervised classification method for gastroscopy images based on multimodal collaborative learning according to claim 4, characterized in that, The definition is as follows: in, and To predict the probabilities for white light images and narrowband light images, To control the parameters of KL divergence, and , respectively, are the average values ​​of the Dirichlet distribution for the white light image and the narrowband light image. and Let represent the Dirichlet distributions for the white light image class and the narrowband light image class after replacing the ground truth labels with 1, respectively, to avoid penalizing the Dirichlet parameter of the base ground truth class, which is 1. This indicates the calculation of the probability density function of the Dirichlet distribution. This represents the KL divergence.

6. The semi-supervised classification method for gastroscopy images based on multimodal collaborative learning according to claim 4, characterized in that, The definition is as follows: in, To control the parameters of KL divergence, The average value of the Dirichlet distribution for the fusion modes. This represents the Dirichlet distribution after the fusion mode has replaced the truth labels with 1. This indicates the calculation of the probability density function of the Dirichlet distribution. This represents the KL divergence.

7. A computer-readable storage medium having stored thereon computer program instructions executable by a processor, wherein when the processor executes the computer program instructions, it is able to implement the steps of the method as described in any one of claims 1-6.

Images