Medical Image Recognition Method and Device Based on Self-Supervised and Few-Shot Learning

Abstract

The present invention relates to the field of image recognition. In a specific embodiment, a medical image recognition method, device, computer device, and storable medium based on self-supervised and few-shot learning are provided. The same pCLE image is randomly enhanced twice using image enhancement technology to obtain different instances, and the two instances are respectively input into a siamese neural network to obtain corresponding image feature embeddings. Then, the two image feature embeddings are feature-mixed, and the siamese neural network is trained in an unsupervised paradigm using the pCLE image dataset. The obtained pre-trained model is combined with a classifier to complete the recognition of pCLE images of common diseases. The pre-trained model obtained in self-supervised learning is combined with metric-based meta-learning to further fine-tune the siamese neural network to achieve the recognition of pCLE images under rare diseases, which has good generalization and effectively realizes the accurate recognition of pCLE images in two scenarios of common category diseases and rare category diseases.

Description

Technical Field

[0001] The present invention relates to the fields of image recognition and computer vision, and particularly to a medical image recognition method, device, computer device, and storage medium based on self-supervised and few-shot learning. Background Art

[0002] Early cancer screening is a key measure to reduce the incidence and mortality of cancer. Therefore, accurate early diagnosis and treatment of tumors have important scientific value. Confocal Laser Endomicroscopy (CLE) is an endoscopic technique that generates in-vivo real-time pathological optical sections. By means of the confocal principle, it can magnify tissues by 1000 times, providing clinicians with real-time in-vivo pathological-level image information, thereby diagnosing early minute lesions, especially tumor lesions, improving the accuracy of biopsies, and reducing the missed diagnosis rate. However, as one of the emerging early digestive disease diagnosis techniques, current doctors still lack the knowledge of diagnosing corresponding diseases based on CLE images, which easily leads to misdiagnosis and missed diagnosis.

[0003] In recent years, computer-aided diagnosis techniques have been widely applied to medical image processing for disease recognition, mainly due to the rapid development of supervised paradigm neural networks and the emergence of large-scale paired datasets. However, as an emerging early gastrointestinal disease diagnosis technique, confocal endoscopy lacks large-scale and complete annotated data. Therefore, training a supervised deep model for confocal endoscopy medical image recognition has become a major challenge. Studying how to learn discriminative representations in the absence of large-scale and complete annotated data to solve confocal endoscopy medical image recognition has great practical significance.

[0004] A method for confocal endoscopy medical image recognition based on self-supervised learning and few-shot learning is currently blank and has not been reported publicly. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a medical image recognition method, device, computer device, and storage medium based on self-supervised and few-shot learning.

[0006] In a first aspect, the present invention provides a medical image recognition method based on self-supervised and few-shot learning, including:

[0007] Using image enhancement technology to perform two random enhancements on probe-based confocal laser endoscopy pCLE images to obtain two different instances, and inputting the two different instances into a siamese neural network respectively to obtain corresponding image feature embeddings;

[0008] Fuse the two obtained image feature embeddings and optimize the pCLE image dataset through regularization;

[0009] Use the pCLE image dataset to perform unsupervised training on the siamese neural network to obtain a pre-trained model and a classifier, and the pre-trained model and the classifier are used for rough recognition of pCLE images;

[0010] Use few-shot learning to train the pre-trained model to obtain a few-shot classifier, and the few-shot classifier is used for fine recognition of pCLE images.

[0011] As an optional solution, the using the image enhancement technology to perform two random enhancements on the probe-type confocal laser endoscopy pCLE image to obtain two different instances, and inputting the two different instances into the siamese neural network respectively to obtain corresponding image feature embeddings, includes:

[0012] Use the image enhancement technology to perform two random enhancements on the same pCLE image X, and obtain the first enhanced image X1 and the second enhanced image X2 respectively,

[0013] Input the first enhanced image X1 and the second enhanced image X2 into the siamese neural network f respectively to obtain the first parameter Z1 and the second parameter Z2;

[0014] Input the first parameter Z1 and the second parameter Z2 into the predictor module composed of three fully connected layers FC to obtain the first category parameter P1 and the second category parameter P2, where ReLU activation function and batch sample normalization BN are included between the fully connected layers;

[0015] Based on the fact that the second parameter Z2 and the second category parameter P2 come from the same branch second parameter Z2, define that the similarity between the second parameter Z2 and the second category parameter P2 is greater than the first parameter Z 1和 the first category parameter P1, get Z2P2>Z1P2, and construct Z based on the following formula (1) 1_mix and Z 2_mix :

[0016]

[0017] where, Z2P2>Z1P2 = S, deduce (1 - λ mix )(Z2P2 - S)<=0, get λ mix >1.

[0018] As an optional solution, the fusing the two obtained image feature embeddings and optimizing the pCLE image dataset through regularization, includes:

[0019] The first parameter Z1 and the second parameter Z2 are mixed using a feature mixing technique to obtain Z 1_mix and Z 2_mix ;

[0020] The cosine distance loss function is used to measure the distances between the first parameter Z1 and the second class parameter P2, the second parameter Z2 and the first class parameter P1, Z 1_mix and P2, and Z 2_mix and P1.

[0021] As an alternative, the unsupervised training of the siamese neural network using the pCLE image dataset to obtain a pre-trained model and a classifier includes:

[0022] Based on the distances, the Adam optimizer is used to optimize the parameters of the siamese neural network, where Z1, Z2, Z 1_mix and Z 2_mix branches stop the gradient and do not update the parameters;

[0023] The parameter optimization is continuously performed until the siamese neural network converges;

[0024] A linear classifier or a knn classifier is used to perform coarse classification on the pCLE images.

[0025] As an alternative, the training of the pre-trained model using few-shot learning to obtain a few-shot classifier for fine recognition of pCLE images includes:

[0026] The prototype centers P of each class in the support set S are obtained using the pre-trained model c ,

[0027]

[0028] where S c represents the number of each class in the support set S, f represents the pre-trained model obtained in the self-supervised learning stage, {(x1, y1), (x2, y2), …, (x n , y n )}, indicates that there are n classes in the few-shot learning process, and each class only includes a small number of samples. During the training stage, C classes are randomly selected from the dataset D, and each class has K data to form the support set S;

[0029] Based on the Euclidean nearest neighbor algorithm, the distances between the prototype centers P of each class calculated from the support set S and the samples in the query set Q are calculated, d(f(q), P c , c );

[0030] Calculate the distance distribution based on Softmax, and the distance distribution is expressed as:

[0031]

[0032] Optimize the distance distribution to obtain a few-shot classifier, which is used to identify pCLE images in rare category scenarios.

[0033] As an optional solution, the image enhancement technology includes at least one of random cropping and random rotation.

[0034] In a second aspect, the present invention provides a medical image recognition device based on self-supervised and few-shot learning, including:

[0035] An image enhancement unit, configured to perform two random enhancements on the probe-based confocal laser endoscopy pCLE image using the image enhancement technology to obtain two different instances, and input the two different instances into the siamese neural network respectively to obtain corresponding image feature embeddings;

[0036] A feature fusion unit, configured to fuse the two obtained image feature embeddings and optimize the pCLE image dataset by regularization;

[0037] A pre-training unit, configured to perform unsupervised training on the siamese neural network using the pCLE image dataset to obtain a pre-trained model and a classifier, and the pre-trained model and the classifier are used to perform rough recognition on pCLE images;

[0038] A few-shot learning unit, configured to train the pre-trained model using few-shot learning to obtain a few-shot classifier, and the few-shot classifier is used to perform fine recognition on pCLE images.

[0039] In a third aspect, the present invention provides a computer device, including:

[0040] At least one processor; and

[0041] A memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned medical image recognition method based on self-supervised and few-shot learning.

[0043] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause the computer to execute the above-mentioned medical image recognition method based on self-supervised and few-shot learning.

[0044] In the embodiments of the present invention, a medical image recognition method, device, computer device, and storable medium based on self-supervised and few-shot learning are provided. The image enhancement technology is used to perform two random enhancements on the same pCLE image to obtain different instances, and the two instances are respectively input into the siamese neural network to obtain corresponding image feature embeddings. Then, the two image feature embeddings are subjected to feature mixing, and the siamese neural network is trained in an unsupervised paradigm using the pCLE image dataset. The obtained pre-trained model is combined with a classifier to complete the recognition of pCLE images of common diseases. The pre-trained model obtained in self-supervised learning is combined with metric-based meta-learning to further fine-tune the siamese neural network to achieve the recognition of pCLE images under rare diseases, which has good generalization and effectively realizes the accurate recognition of pCLE images in two scenarios of common category diseases and rare category diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a medical image recognition method based on self-supervised and few-shot learning provided in the embodiments of the present invention;

[0046] Figure 2 It is a schematic diagram of the model in the self-supervised learning stage of a medical image recognition method based on self-supervised and few-shot learning provided in the embodiments of the present invention;

[0047] Figure 3 It is a partial category schematic diagram of the SSL_pCLE image dataset in a medical image recognition method based on self-supervised and few-shot learning provided in the embodiments of the present invention;

[0048] Figure 4 It is a partial category schematic diagram of the FS_pCLE image dataset in a medical image recognition method based on self-supervised and few-shot learning provided in the embodiments of the present invention;

[0049] Figure 5 It is a schematic structural diagram of a medical image recognition device based on self-supervised and few-shot learning provided in the embodiments of the present invention;

[0050] Figure 6 It is a structural block diagram of a computer device provided in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0052] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order different from that shown or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0053] Combined with Figure 1 As shown, an image recognition method for medical images based on self-supervised and few-shot learning is provided in an embodiment of the present invention, including:

[0054] S101. Use image enhancement technology to perform two random enhancements on probe-based confocal laser endomicroscopy (pCLE) images to obtain two different instances, and input the two different instances into a siamese neural network respectively to obtain corresponding image feature embeddings.

[0055] S102. Perform feature fusion on the two obtained image feature embeddings and optimize the pCLE image dataset through regularization.

[0056] S103. Use the pCLE image dataset to perform unsupervised training on the siamese neural network to obtain a pre-trained model and a classifier, and the pre-trained model and the classifier are used to perform rough recognition on pCLE images. The rough recognition mentioned here can be pCLE images in common category disease scenarios.

[0057] S104. Use few-shot learning to train the pre-trained model to obtain a few-shot classifier, and the few-shot classifier is used to perform fine recognition on pCLE images. The fine recognition mentioned here can be pCLE images in rare category disease scenarios.

[0058] Specifically, the same pCLE image is randomly enhanced twice using image enhancement techniques to obtain different instances, and the two instances are respectively input into a siamese neural network to obtain corresponding image feature embeddings; then, the two image feature embeddings are feature-mixed. Feature mixing can introduce more task-related features through regularization methods to address the large intra-class variance problem in the pCLE dataset and avoid overfitting the mutual information between the two instances. The siamese neural network is trained in an unsupervised paradigm using the pCLE image dataset, and the obtained pre-trained model is combined with a classifier to complete the rough recognition of pCLE images of common diseases.

[0059] Since CLE, as an emerging early diagnosis technology for gastrointestinal diseases, has only a small number of pCLE annotated images for certain diseases, common supervised learning or unsupervised learning has not been able to accurately identify it; therefore, the embodiments of the present invention propose to further fine-tune the siamese neural network by combining the pre-trained model obtained in self-supervised learning with metric-based meta-learning to achieve the recognition of pCLE images under rare diseases, and can accurately recognize pCLE images of both common category diseases and rare diseases.

[0060] Specifically, in S101, the probe-based confocal laser endoscopy pCLE image is randomly enhanced twice using image enhancement techniques to obtain two different instances, and the two different instances are respectively input into a siamese neural network to obtain corresponding image feature embeddings, including:

[0061] The same pCLE image X is randomly enhanced twice using image enhancement techniques to obtain a first enhanced image X1 and a second enhanced image X2 respectively.

[0062] The first enhanced image X1 and the second enhanced image X2 are respectively input into the siamese neural network f to obtain a first parameter Z1 and a second parameter Z2.

[0063] The first parameter Z1 and the second parameter Z2 are input into a predictor module composed of three fully connected layers FC to obtain a first category parameter P1 and a second category parameter P2, where ReLU activation functions and batch sample normalization BN are included between the fully connected layers.

[0064] Based on the fact that the second parameter Z2 and the second category parameter P2 come from the same branch, the second parameter Z2, it is defined that the similarity between the second parameter Z2 and the second category parameter P2 is greater than the first parameter Z 1和 The first category parameter P1, resulting in Z2P2 > Z1P2. Based on the following formula (1), Z 1_mix and Z 2_mix :

[0065]

[0066] Among them, Z2P2 > Z1P2 = S, and it is deduced that (1 - λ mix )(Z2P2 - S) <= 0, and λ mix > 1 is obtained.

[0067] As an alternative solution, the feature fusion of the two obtained image features by embedding and optimizing the pCLE image dataset in a regularization manner includes:

[0068] Using a feature mixing technique to mix the first parameter Z1 and the second parameter Z2 to obtain Z 1_mix and Z 2_mix ;

[0069] Using the cosine distance loss function to measure the distance between the first parameter Z1 and the second category parameter P2, the second parameter Z2 and the first category parameter P1, Z 1_mix and P2, and the distance between Z 2_mix and P1.

[0070] As an alternative solution, the unsupervised training of the siamese neural network using the pCLE image dataset to obtain a pre-trained model and a classifier includes:

[0071] Based on the distance, using the Adam optimizer to optimize the parameters of the siamese neural network, where Z1, Z2, Z 1_mix and Z 2_mix branches stop the gradient and do not update the parameters;

[0072] Continuously optimize the parameters until the siamese neural network converges;

[0073] Using a linear classifier or a knn classifier to perform rough classification on the pCLE image.

[0074] As an alternative solution, the training of the pre-trained model using few-shot learning to obtain a few-shot classifier for fine recognition of pCLE images includes:

[0075] Using the pre-trained model to obtain the prototype center P of each category in the support set S c ,

[0076]

[0077] where S c represents the number of each category in the support set S, f represents the pre-trained model obtained in the self-supervised learning stage, {(x1, y1), (x2, y2), …, (xn , y n )}, which means that there are n categories in the few-shot learning process, and each category only includes a small number of samples. During the training stage, C categories are randomly selected from the dataset D, and each category has K data to form the support set S;

[0078] Calculate the prototype center P of each category obtained by calculating the support set S based on the Euclidean nearest neighbor algorithm c , the distance from the sample in the query set Q, d(f(q), P c );

[0079] Calculate the distance distribution based on Softmax, and the distance distribution is expressed as:

[0080]

[0081] Optimize the distance distribution to obtain a few-shot classifier, which is used to recognize pCLE images in rare category scenarios.

[0082] It can be understood that the metric-based meta-learning method can be called a C-way K–shot task, D = {(x1, y1), (x2, y2), …, (x n , y n )}, which means that there are n categories in the few-shot learning process, and each category only includes a small number of samples (generally dozens or several). During the training stage, C categories are randomly selected from the dataset D, and each category has K data to form the support set S (support set), and the remaining m data of each category in the dataset D are used to construct the query set Q (queryset) for the meta-learning task.

[0083] As an optional solution, the image enhancement technology includes at least one of random cropping and random rotation, and those of ordinary skill in the art can flexibly select it without limitation.

[0084] To better understand the medical image recognition method based on self-supervised and few-shot learning provided by the present invention, the innovation points are described below:

[0085] Combined with Figure 2As shown, self-supervised learning (SSL) can be considered a special case of unsupervised learning, which directly learns powerful representations from the data itself without the need for large-scale labeled datasets. One of the most common strategies for SSL to learn powerful representations is to design various pretext tasks, such as image jigsaw puzzles, predicting the relative positions of patches, image inpainting, image colorization, and image rotation. Contrastive learning is used to obtain invariant representations and achieve state-of-the-art performance. Contrastive learning methods are usually based on instance discrimination, which aims to reduce the distance between different views of the same instance and increase the distance between different instances to achieve similar attraction and different repulsion. Recently, there has been increasing attention to non-contrastive learning methods, which are based only on two different views generated from the same image (without constructing negative instances). However, non-contrastive learning methods usually lead to model collapse or dimensional collapse, and various methods have been proposed to prevent model collapse or dimensional collapse, including stopping gradients, cross-correlation matrices, and whitening, and all of these methods have shown satisfactory results. In addition to natural images, self-supervised learning has also been widely applied in medical images. Before contrastive learning, pretext tasks mainly included image jigsaw puzzles or predicting the relative positions of patches. However, existing contrastive and non-contrastive methods have failed to effectively solve the problem of small inter-class variance and large intra-class variance in the pCLE dataset, as Figure 1 shown, we can see that the first and third pictures belong to the same class, but they show great visual differences between them, while the second and third pictures belong to the same class, but they are visually almost close. We believe that the above methods are learned on large datasets and are not suitable for scenarios involving a small amount of pCLE data. Therefore, this method follows the principle of non-contrastive learning methods and introduces more task-related information through feature mixing at the feature level, avoiding overfitting the minimum mutual information between instances.

[0086] To facilitate the understanding of the self-supervised learning stage, corresponding pseudo-code is provided as an example in this embodiment.

[0087]

[0088] The method provided by the present invention mainly includes two stages, namely the self-supervised learning stage and the few-shot learning stage. Few-shot learning is an active subfield of machine learning. It learns a robust embedding representation from a base learner and then applies it to classes with supervised information and only a few training samples, enabling it to classify unseen data of new classes. Few-shot learning methods can be divided into two major categories: metric-based meta-learning methods and gradient-based meta-learning methods. Metric-based meta-learning methods are usually used to compare data samples. In few-shot classification problems, it classifies test samples based on the similarity between test samples and training samples. Gradient-based meta-learning methods include a base learner and a meta-learner, where the base learner is trained on a large-scale sample to obtain a pre-trained model. Then, before the meta-learner updates its parameters, it can be trained to adapt to small and unseen classes of data. The core of this task is to enable the model to quickly adapt to new learning tasks with limited training data without overfitting. The essence of this process is how to make the pre-trained model adapt to data changes. Therefore, we believe that the pre-trained model obtained in the self-supervised learning stage can quickly adapt to previously unseen classes to achieve performance gains in few-shot learning.

[0089] The formula for negative cosine similarity is as follows:

[0090]

[0091] Let the feature embedding generated by each image after the encoder f and the predictor be p. Negative cosine similarity is a method for measuring the similarity between two images. By calculating the negative cosine similarity and using gradient descent, it can better guide the model to learn discriminative features.

[0092] The formula for Top-1 accuracy is as follows:

[0093]

[0094] Among them, TP represents a correct prediction and the actual class is positive; TN represents a correct prediction and the actual class is negative; FP represents an incorrect prediction and the actual class is positive; FN represents an incorrect prediction and the actual class is negative. This formula means that the predicted label takes the one with the largest probability in the probability vector as the prediction result. If the class with the largest probability in this prediction result is the same as the actual class, the prediction is correct. Otherwise, the prediction is incorrect. By using Top-1 accuracy, it can better guide the classification of the pCLE image model.

[0095] The formula for the 95% confidence interval is as follows:

[0096]

[0097] Among them, μ represents the mean, σ represents the standard deviation, and n is the number of test samples. The 95% confidence interval is mainly used to indicate that there is a 95% probability that the classification accuracy of the model will fall within the interval of the true result, providing a more reliable estimate of the classification accuracy of the model.

[0098] Combined with Figure 3 and 4 As shown, below, in combination with a specific usage scenario, for the medical image recognition method provided by the present invention based on self-supervised and few-shot learning, specifically, the experimental objects include the SSL_pCLE image dataset and the FS_pCLE image dataset. The SSL_pCLE image dataset is collected from 86 patients, including six categories: intestinal metaplasia (405 images), low-grade intraepithelial neoplasia (277 images), high-grade intraepithelial neoplasia (167 images), atrophic gastritis (568 images), normal gastric antrum (627 images), and normal gastric body (325 images), a total of 2369 digestive tract disease pictures. During the experiment, in order to avoid information leakage, we randomly selected the data of 70 patients as the training set to obtain the model parameters in the self-supervised stage, and the remaining 16 patients as the test set, as Figure 4 shown. Further, the FS_pCLE image dataset has a total of 14 categories, with approximately 40 pictures in each category, as Figure 5 shown.

[0099] For the encoder, ResNet18 (removing the last fully connected layer) is selected as the backbone network, followed by a projection head (FC-BN-ReLU-FC-BN-ReLU-FC-BN) composed of multiple fully connected layers, with input and output dimensions of 512 and 2048 respectively. Finally, for the predictor, we adopted batch normalization BN in the hidden layer and set the dimension to 512. There is neither BN nor ReLU in the output layer of the predictor.

[0100] Obtained by training on the SSL_pCLE dataset, since part of the pCLE dataset is inconsistent with ImageNet, training is restarted. First, perform augmentation operations, scale the pCLE images to 256×256 and use random cropping (224×224), random vertical flipping, random horizontal flipping, and Gaussian blur. Select SGD as the optimizer of the model, with a learning rate of 0.02 and a weight decay value of 5e-4, and use the cosine gradient descent strategy to adjust the learning rate. This siamese neural network model is trained for a total of 800 epochs, with a batch size set to 128, and the hyperparameters α and λ are set to 2.0 and 0.7 respectively. All experiments are carried out on the Nvidia RTX5000 GPU.

[0101] Next, the performance of the model in the self-supervised learning stage is verified through a linear evaluation protocol.

[0102] First, freeze all the features before the global average pooling layer of ResNet18, and then train a linear classifier to evaluate the performance of the model in the self-supervised learning stage, where the Top-1 accuracy and 95% confidence interval are used as the measurement tools. Set the learning rate and batch size of this linear classifier to 0.01 and 128 respectively, and train for a total of 100 epochs, with the learning rate halved at the 60th and 80th epochs. Use SGD as the optimizer, and set the momentum and weight decay to 0.9 and 0 respectively. In addition, experiments were also conducted using two linear layers (FC-ReLU-BN-FC) as the classifier, with other settings remaining unchanged.

[0103] Table 1 shows the results of classification experiments of various state-of-the-art methods and the proposed method on the SSL_pCLE dataset. It can be seen from the table that the proposed method outperforms SimSiam by 6% and 5% under one and two evaluations respectively, where one means using one fully connected layer in the linear evaluation, and two means using two fully connected layers in the linear evaluation. In addition, the method proposed in the present invention shows competitive performance with the supervised method under one and two evaluations. This shows that the present invention can effectively classify pCLE images of various digestive tract diseases and reduce the labor cost consumed by labeled data.

[0104] Table 1 Comparison with Other State-of-the-Art Methods on the SSL_pCLE Dataset

[0105]

[0106] One means using one fully connected layer in the linear evaluation.

[0107] Two means using two fully connected layers in the linear evaluation.

[0108] Next, experiments in the few-shot learning stage are carried out on the FS_pCLE dataset.

[0109] Use the pre-trained model obtained in the self-supervised learning stage (delete all fully connected layers) for few-shot learning. First, randomly split the FS_pCLE dataset, where the data of 8 categories are used for training and the data of 6 categories are used for testing. It should be noted that the data between any two categories are not collected from the same patient. Then, select Adam as the optimizer of the model and set the learning rate to 0.0005. There are 10 query images for each category in each episode during training and testing, and a total of 50 epochs are trained, with the learning rate halved every ten epochs. We conduct 5-way 1-shot and 5-way 5-shot tasks on the FS_pCLE dataset respectively to verify the proposed method and compare it with the state-of-the-art methods.

[0110] Table 2 shows the results of few-shot learning of various methods and the proposed method on the FS_pCLE dataset. From the table, it can be seen that the method proposed in the present invention outperforms other methods in both the 5-way 1-shot and 5-way 5-shot tasks. This indicates that the performance of few-shot learning highly depends on an effective pre-trained model. At the same time, it also shows that the present invention can effectively classify pCLE images of digestive tract diseases with few-shot categories, which helps to improve the classification of pCLE images lacking large-scale annotation data at different stages of tumor development.

[0111] Accuracy of Few-shot Classification on the FS_pCLE Dataset

[0112]

[0113] The medical image recognition method based on self-supervised and few-shot learning provided in the embodiments of the present invention uses image enhancement technology to perform two random enhancements on the same pCLE image to obtain different instances, and inputs the two instances into a siamese neural network respectively to obtain corresponding image feature embeddings; then, performs feature mixing on the two image feature embeddings, uses the pCLE image dataset to train the siamese neural network in an unsupervised paradigm, combines the obtained pre-trained model with a classifier to complete the recognition of pCLE images of common diseases, and uses the pre-trained model obtained in self-supervised learning in combination with metric-based meta-learning to further fine-tune the siamese neural network to achieve the recognition of pCLE images under rare diseases, which has good generalization and effectively realizes the accurate recognition of pCLE images in two scenarios of common category diseases and rare category diseases.

[0114] Combined Figure 5 As shown, an apparatus for medical image recognition based on self-supervised and few-shot learning provided in the embodiments of the present invention includes:

[0115] An image enhancement unit 601, configured to use image enhancement technology to perform two random enhancements on a probe-based confocal laser endomicroscopy (pCLE) image to obtain two different instances, and input the two different instances into a siamese neural network respectively to obtain corresponding image feature embeddings;

[0116] A feature fusion unit 602, configured to perform feature fusion on the two obtained image feature embeddings and optimize the pCLE image dataset by means of regularization;

[0117] A pre-training unit 603, configured to use the pCLE image dataset to perform unsupervised training on the siamese neural network to obtain a pre-trained model and a classifier, and the pre-trained model and the classifier are used to perform rough recognition on pCLE images;

[0118] A few-shot learning unit 604 is used to train the pre-trained model using few-shot learning to obtain a few-shot classifier, and the few-shot classifier is used for fine recognition of pCLE images.

[0119] In the embodiment of the present invention, the medical image recognition device based on self-supervised and few-shot learning uses image enhancement technology to perform two random enhancements on the same pCLE image to obtain different instances, and inputs the two instances into the siamese neural network respectively to obtain corresponding image feature embeddings; then, performs feature mixing on the two image feature embeddings, uses the pCLE image dataset to train the siamese neural network in an unsupervised paradigm, combines the obtained pre-trained model with a classifier to complete the recognition of pCLE images of common diseases, and uses the pre-trained model obtained in self-supervised learning combined with metric-based meta-learning to further fine-tune the siamese neural network to achieve the recognition of pCLE images under rare diseases, which has good generalization and effectively realizes the accurate recognition of pCLE images in two scenarios of common category diseases and rare category diseases.

[0120] Correspondingly, according to the embodiment of the present invention, the present invention also provides a computer device, a readable storage medium, and a computer program product.

[0121] Figure 6 It is a schematic structural diagram of a computer device 12 provided in the embodiment of the present invention. Figure 6 It shows a block diagram of an exemplary computer device 12 suitable for implementing the embodiments of the present invention. Figure 6 The shown computer device 12 is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0122] As Figure 6 shown, the computer device 12 is presented in the form of a general-purpose computing device. The computer device 12 is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0123] The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0124] Bus 18 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor bus, or a local bus using any of the several bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0125] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including both volatile and nonvolatile media, removable and non-removable media.

[0126] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory. Computer device 12 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, storage system 34 can be used for reading from and writing to non-removable, nonvolatile magnetic media ( Figure 6 not shown and typically called a "hard disk drive"). Although Figure 6 not shown in the figures, a disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from and writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) can be provided. In these instances, each drive can be connected to bus 18 by one or more data media interfaces. Memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of the embodiments of the present invention.

[0127] A program / utility 40 having a set (at least one) of program modules 42 can be stored, for example, in memory 28, and such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a networking environment. Program modules 42 typically carry out the functions and / or methods of the embodiments described herein.

[0128] The computer device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the computer device 12, and / or communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the computer device 12 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the computer device 12 through the bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0129] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, for example, implementing the medical image recognition method based on self-supervised and few-shot learning provided by the embodiments of the present invention.

[0130] Embodiments of the present invention also provide a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein when the program is executed by a processor, it is the medical image recognition method based on self-supervised and few-shot learning provided by all the embodiments of the present application.

[0131] The computer storage medium of the embodiments of the present invention can adopt any combination of one or more computer-readable media. The computer-readable media can be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0132] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0133] The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing. The computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or, can be connected to an external computer (e.g., via an Internet service provider through the Internet).

[0134] An embodiment of the present invention also provides a computer program product, including a computer program, which when executed by a processor implements the medical image recognition method based on self-supervised and few-shot learning as described above.

[0135] It should be understood that various forms of the flow shown above can be used, reordering, adding, or deleting steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and no limitation is made herein.

[0136] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A medical image recognition method based on self-supervised and few-shot learning, characterized in that, Including: Using image enhancement technology to perform two random enhancements on the probe-based confocal laser endoscopy pCLE image to obtain two different instances, and inputting the two different instances into a Siamese neural network respectively to obtain corresponding image feature embeddings; including: Using image enhancement technology to perform two random enhancements on the same pCLE image X, respectively obtaining a first enhanced image X1 and a second enhanced image X2, Input the first enhanced image X1 and the second enhanced image X2 into the siamese neural network respectively f to obtain a first parameter Z1 and a second parameter Z2; Inputting the first parameter Z1 and the second parameter Z2 into a predictor module composed of three fully connected layers FC to obtain a first class parameter P1 and a second class parameter P2, where ReLU activation function and batch sample normalization BN are included between the fully connected layers; Based on the fact that the second parameter Z2 and the second category parameter P2 come from the same branch's second parameter Z2, it is defined that the similarity between the second parameter Z2 and the second category parameter P2 is greater than that between the first parameter Z1 and the first category parameter P1, resulting in Z2P2 > Z1P2. Based on the following formula (1), construct Z 1_mix and Z 2_mix : （1） Among them, Z2P2 > Z1P2 = S, which is deduced to (1 - λ mix ) (Z2P2 - S) <= 0, and λ mix > 1 is obtained; Performing feature fusion on the two obtained image feature embeddings and optimizing the pCLE image dataset by regularization; Using the pCLE image dataset to perform unsupervised training on the Siamese neural network to obtain a pre-trained model and a classifier, and the pre-trained model and the classifier are used for rough recognition of pCLE images; Using few-shot learning to train the pre-trained model to obtain a few-shot classifier, and the few-shot classifier is used for fine recognition of pCLE images.

2. The medical image recognition method based on self-supervised and few-shot learning according to claim 1, characterized in that, The performing feature fusion on the two obtained image feature embeddings and optimizing the pCLE image dataset by regularization includes: The first parameter Z1 and the second parameter Z2 are mixed by using a feature mixing technique to obtain Z 1_mix and Z 2_mix ; Use the cosine distance loss function to measure the distances between the first parameter Z1 and the second category parameter P2, the second parameter Z2 and the first category parameter P1, Z 1_mix and P2, as well as Z 2_mix and P1.

3. The medical image recognition method based on self-supervised and few-shot learning according to claim 2, wherein, The using the pCLE image dataset to perform unsupervised training on the Siamese neural network to obtain a pre-trained model and a classifier includes: Based on the distance, use the Adam optimizer to optimize the parameters of the Siamese neural network, where Z1, Z2, Z 1_mix and Z 2_mix branches stop the gradient and do not update the parameters; Continuously performing parameter optimization until the Siamese neural network converges; Using a linear classifier or a knn classifier to perform rough classification on pCLE images.

4. The medical image recognition method based on self-supervised and few-shot learning according to claim 3, wherein, The using few-shot learning to train the pre-trained model to obtain a few-shot classifier, and the few-shot classifier is used for fine recognition of pCLE images includes: Obtain the prototype center P of each category in the support set S by using the pre-trained model c , where S c represents the number of each category in the support set S, f represents the pre-trained model obtained in the self-supervised learning stage, In the few-shot learning process, there are n categories, and each category only includes a small number of samples. During the training stage, C categories are randomly selected from the dataset D, and each category has K data to form the support set S; The prototype center P of each category calculated by computing the support set S based on the Euclidean nearest neighbor algorithm c , the distance to the samples in the query set Q ; Calculating the distance distribution based on Softmax, and the distance distribution is expressed as: Optimizing the distance distribution to obtain a few-shot classifier, and the few-shot classifier is used for recognizing pCLE images in rare category scenarios.

5. The medical image recognition method based on self-supervised and few-shot learning according to claim 1, wherein The image enhancement technology includes at least one of random cropping and random rotation.

6. A medical image recognition device based on self-supervised and few-shot learning, characterized in that, Including: An image enhancement unit, configured to use image enhancement technology to perform two random enhancements on the probe-based confocal laser endoscopy pCLE image to obtain two different instances, and input the two different instances into a Siamese neural network respectively to obtain corresponding image feature embeddings; including: Using image enhancement technology to perform two random enhancements on the same pCLE image X, respectively obtaining a first enhanced image X1 and a second enhanced image X2, Input the first enhanced image X1 and the second enhanced image X2 into the siamese neural network respectively f to obtain a first parameter Z1 and a second parameter Z2; Inputting the first parameter Z1 and the second parameter Z2 into a predictor module composed of three fully connected layers FC to obtain a first class parameter P1 and a second class parameter P2, where ReLU activation function and batch sample normalization BN are included between the fully connected layers; Based on the fact that the second parameter Z2 and the second category parameter P2 come from the same branch of the second parameter Z2, it is defined that the similarity between the second parameter Z2 and the second category parameter P2 is greater than that between the first parameter Z1 and the first category parameter P1, resulting in Z2P2 > Z1P2. Based on the following formula (1), construct Z 1_mix and Z 2_mix : （1） Among them, Z2P2 > Z1P2 = S, and it is deduced that (1 - λ mix ) (Z2P2 - S) <= 0, and λ mix > 1 is obtained; A feature fusion unit, configured to perform feature fusion on the two obtained image feature embeddings and optimize the pCLE image dataset by regularization; A pre-training unit for unsupervised training of the siamese neural network using the pCLE image dataset to obtain a pre-trained model and a classifier, where the pre-trained model and the classifier are used for rough recognition of pCLE images; A few-shot learning unit for training the pre-trained model using few-shot learning to obtain a few-shot classifier, where the few-shot classifier is used for fine recognition of pCLE images.

7. A computer device, characterized in that, Comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the medical image recognition method based on self-supervised and few-shot learning according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the medical image recognition method based on self-supervised and few-shot learning according to any one of claims 1 to 5.

Images