Self-supervised recognition method, system, device and medium based on esophageal endoscopic image

Abstract

This invention discloses a self-supervised recognition method, system, device, and medium based on esophageal endoscopy images, which addresses the problem of low classification accuracy of esophageal endoscopy images caused by traditional self-supervised learning training in existing technologies. In the second stage of training, unlabeled image sample data undergoes two identical random data augmentations before being input into an online encoder and a target encoder, respectively. In the online encoder, the backbone teacher network T and the backbone student network S1 perform knowledge distillation and self-supervised learning. The output of the backbone student network S1 is sequentially passed through a projection module and a prediction module, with the output of the prediction module serving as the output of the online encoder. In the target encoder, the backbone teacher network T and the backbone student network S2 perform knowledge distillation and self-supervised learning. The output of the backbone student network S2 is input into a projection module, with the output of the projection module serving as the output of the target encoder. Finally, constraints are applied to the feature map differences between the outputs of the online encoder and the target encoder.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to the identification and classification of esophageal cancer lesions, and more particularly to a self-supervised identification method, system, device and medium based on esophageal endoscopic images. Background Technology

[0002] Currently, artificial intelligence technology has developed rapidly and is widely used in esophageal cancer screening. Liu et al. proposed a two-stream esophageal classification method. This method divides the network into two different streams, named O-Net and P-Net. The input of O-Net is the original image, mainly focusing on extracting color features and global features. The processed image is input to P-Net to extract texture and more complex details. The features extracted from the two streams are fused, and then the esophageal image is classified. This method uses approximately 1200 labeled images. Due to the relatively small dataset, its effectiveness in real-world medical scenarios will be significantly reduced. Wu et al. introduced a multi-task method for localization, classification, and segmentation of esophageal lesions in white light endoscopic images. They processed the reflection points of these data. First, a faster R-CNN is used to localize the lesions in the image; then, a two-stream network is used to perform four-class classification on the localized points; finally, the U-Net algorithm is used to segment abnormal images. This method was mainly validated on esophageal images under white light endoscopy, but its effectiveness on other types of staining, such as narrow-band imaging and iodine staining images, has not yet been determined. Considering the limited size of the endoscopic image dataset, Wang et al. implemented three data augmentation strategies to increase the dataset size. They then input these three types of images into the network separately for feature extraction and fusion, ultimately classifying the data based on the fused features. In contrast, Zhou et al. proposed a method that establishes a deep multimodal convolutional neural network structure to classify esophageal cancer subtypes using dynamic CT and WSI images. This method primarily focuses on a feature enhancement module, which assigns different weights to feature maps using local and global perception at different scales. Furthermore, the evolutionary algorithm employed in this method alleviates the manual construction problem in previous generations of esophageal lesion identification and segmentation tasks. Pedro E et al. proposed a knowledge distillation method to detect polyps, tumors, and NBDE lesions, with category-aware loss playing a crucial role. These methods all use labeled esophageal images when training the neural network model, neglecting unlabeled esophageal endoscopy images, or failing to consider the features of unlabeled images. Therefore, the potential of unlabeled esophageal endoscopy images remains to be explored and studied.

[0003] In fact, researchers have already conducted studies on feature learning for unlabeled images. Azizi et al. used Multi-Instance Contrast Learning (MICLe) for medical image classification, whose core framework is Simple Contrast Learning (SimCLR). MICLe reduces the need for a large number of pairs in self-supervised learning, allowing fine-tuning of the pre-trained model using a small amount of labeled data. However, the core of this method, SimCLR, requires a large batch size during retraining, placing significant demands on computational resources. In another study, Ouyang et al. proposed the SSL-ALPNet framework to further investigate the capabilities of self-supervised learning, aiming to circumvent the necessity of labels during training. These studies demonstrate the significant potential of self-supervised learning in AI-based esophageal cancer screening.

[0004] Based on the aforementioned existing technologies, self-supervised methods using contrastive learning have shown great potential in feature learning from unlabeled images. Furthermore, unlabeled esophageal endoscopy images are relatively easy to collect. Currently, most methods for training models using unlabeled images for self-supervised learning directly input unlabeled data into a pre-trained model, allowing the model to undergo self-supervised learning and training. However, these traditional methods result in poor self-supervised learning performance, affecting the classification accuracy of esophageal endoscopy images. Summary of the Invention

[0005] The purpose of this invention is to address the problem of low classification accuracy of esophageal endoscopy images caused by traditional self-supervised learning training in the prior art, and to provide a self-supervised recognition method, system, device, and medium based on esophageal endoscopy images. In the self-supervised learning process, a novel knowledge distillation strategy is adopted, which can extract features from the supervised learning stage, enhance feature extraction of unlabeled esophageal endoscopy images, and improve the classification accuracy of esophageal endoscopy images.

[0006] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0007] A self-supervised recognition method based on esophageal endoscopy images includes the following steps:

[0008] Step S1: Obtain sample data;

[0009] Acquire esophageal endoscopy image samples, including labeled image sample data and unlabeled image sample data;

[0010] Step S2: Construct an image recognition and classification model;

[0011] An image recognition and classification model is constructed, which includes a backbone teacher network T, a backbone student network S1, a backbone student network S2, a projection module, and a prediction module.

[0012] Step S3: Train the image recognition and classification model;

[0013] The image recognition and classification model constructed in step S2 is trained using the sample data obtained in step S1, specifically including:

[0014] Phase 1: Supervised learning training of the backbone teacher network T using labeled image sample data;

[0015] Phase Two: The supervised learning-trained backbone teacher network T is replicated to obtain two backbone teacher networks T. One backbone teacher network T, together with the backbone student network S1, the projection module, and the prediction module, forms an online encoder. The other backbone teacher network T, together with the backbone student network S2, and the projection module, forms a target encoder. Unlabeled image sample data is input into the online encoder and the target encoder after undergoing two identical random data augmentations. In the online encoder, the backbone teacher network T and the backbone student network S1 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S1 is passed through the projection module and the prediction module in sequence, and the output of the prediction module becomes the output of the online encoder. In the target encoder, the backbone teacher network T and the backbone student network S2 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S2 is input into the projection module, and the output of the projection module becomes the output of the target encoder. Finally, constraints are applied to the feature map differences between the outputs of the online encoder and the target encoder.

[0016] Phase 3: Supervised learning training of the backbone student network S1 is performed using labeled image sample data to obtain the final image recognition and classification model;

[0017] Step S4: Real-time classification of esophageal endoscopy images;

[0018] The acquired real-time esophageal endoscopy images are input into step S3 to obtain the final image recognition and classification model, which then outputs the classification results.

[0019] Furthermore, in step S2, the backbone teacher network T, backbone student network S1, and backbone student network S2 have the same structure, all using ResNet networks, and both the projection module and the prediction module are fully connected layers.

[0020] Furthermore, in step S3, the distillation loss of the backbone teacher network T and the backbone student network S1 in the online encoder... for:

[0021] ;

[0022] Distillation loss of backbone teacher network T and backbone student network S2 in the target encoder for:

[0023] ;

[0024] Total distillation loss for:

[0025] ;

[0026] in, Indicates the number of feature maps selected. Indicates a teacher network, Indicates student network, This represents the i-th feature map in the online encoder. This represents the i-th feature map in the target encoder. This represents the i-th feature map of the teacher network in the online encoder. This represents the i-th feature map of the student network in the online encoder. This represents the i-th feature map of the teacher network in the target encoder. Let i represent the i-th feature map of the student network in the target encoder.

[0027] Furthermore, the complete distillation loss function Represented as:

[0028] ;

[0029] in, This represents the symmetrical loss due to distillation.

[0030] Furthermore, in step S3, the self-supervised loss of the online encoder and the target encoder is... for:

[0031] ;

[0032] in, This represents the parameters of the backbone network in the online encoder. This represents the mapping between the projection layer and the prediction layer in an online encoder. This represents the mapping of the backbone network in the online encoder. This represents the output of the online encoder. This represents the mapping of the backbone network in the target encoder. This represents the output of the target encoder.

[0033] Furthermore, complete self-monitored loss Represented as:

[0034] ;

[0035] in, Indicates self-monitoring loss Symmetrical loss.

[0036] A self-supervised recognition system based on esophageal endoscopy images, comprising:

[0037] The sample data acquisition module is used to acquire esophageal endoscopy image samples, which include labeled image sample data and unlabeled image sample data.

[0038] The image recognition and classification model building module is used to build an image recognition and classification model. The image recognition and classification model includes a backbone teacher network T, a backbone student network S1, a backbone student network S2, a projection module, and a prediction module.

[0039] The image recognition and classification model training module is used to train the image recognition and classification model constructed by the image recognition and classification model construction module using sample data obtained by the sample data acquisition module. Specifically, it includes:

[0040] Phase 1: Supervised learning training of the backbone teacher network T using labeled image sample data;

[0041] Phase Two: The supervised learning-trained backbone teacher network T is replicated to obtain two backbone teacher networks T. One backbone teacher network T, together with the backbone student network S1, the projection module, and the prediction module, forms an online encoder. The other backbone teacher network T, together with the backbone student network S2, and the projection module, forms a target encoder. Unlabeled image sample data is input into the online encoder and the target encoder, respectively. In the online encoder, the backbone teacher network T and the backbone student network S1 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S1 is passed through the projection module and the prediction module in sequence, and the output of the prediction module becomes the output of the online encoder. In the target encoder, the backbone teacher network T and the backbone student network S2 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S2 is input into the projection module, and the output of the projection module becomes the output of the target encoder. Finally, constraints are applied to the feature map differences between the outputs of the online encoder and the target encoder.

[0042] Phase 3: Supervised learning training of the backbone student network S1 is performed using labeled image sample data to obtain the final image recognition and classification model;

[0043] The real-time esophageal endoscopy image classification module is used to input the acquired real-time esophageal endoscopy images into the image recognition classification model training module to obtain the final image recognition classification model, and the image recognition classification model outputs the classification result.

[0044] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method described above.

[0045] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-described method.

[0046] The beneficial effects of this invention are as follows:

[0047] 1. Compared to traditional machine learning, this invention is not limited by the amount of labeled data. Traditional medical AI models rely heavily on the number of labels, while this invention primarily uses readily available unlabeled medical images.

[0048] 2. Compared to ordinary self-supervised contrastive learning, this invention can learn more features from unlabeled data. In ordinary contrastive learning, the convergence of self-supervised loss depends on batch size, the degree of data augmentation, etc., and convergence is not easy. The distillation module added in this invention provides a new direction for the loss function, which can promote the convergence of the loss function.

[0049] 3. Compared with conventional knowledge distillation, this invention uses knowledge distillation to enhance feature extraction from unlabeled data rather than model compression, which is another new approach to knowledge distillation.

[0050] 4. Compared to specific tasks, this invention aims to provide a self-supervised learning framework. The resulting model can be used on any medical image-related task, such as classification, detection, and segmentation. This invention provides a pre-trained model that can be transferred to other digestive endoscopy tasks.

[0051] 5. This invention combines knowledge distillation with self-supervised learning, adding knowledge distillation to ordinary self-supervised comparative learning, thereby improving the classification accuracy of esophageal endoscopy images.

[0052] 6. This invention employs a unique knowledge distillation process. Traditional knowledge distillation is used for model compression, while the knowledge distillation in this invention is used to extract features from unlabeled images.

[0053] 7. This invention adopts an innovative loss function structure, which incorporates hyperparameter balancing self-supervised loss and distillation loss into the loss function, thereby improving the learning and training efficiency and effectiveness of the model.

[0054] 8. This invention determines that using knowledge distillation simultaneously on both the target encoder and the online encoder yields the best results. Additionally, it also considers the logarithm of the feature maps selected by knowledge distillation.

[0055] 9. Conventional self-supervised learning typically involves two stages: first, self-supervised learning is performed, and then the network is fine-tuned using labeled images. However, this invention uses three stages, employing the same batch of labeled data for supervised training both before and after self-supervised learning. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the present invention;

[0057] Figure 2 This is a schematic diagram of model training in this invention;

[0058] Figure 3 This is a schematic diagram illustrating knowledge distillation and self-supervised learning between the teacher network and the student network in this invention. Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0060] Therefore, all other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0061] Example 1

[0062] This embodiment provides a self-supervised recognition method based on esophageal endoscopy images, such as... Figure 1 As shown, the specific steps include:

[0063] Step S1: Obtain sample data;

[0064] Esophageal endoscopic image samples were acquired, including labeled and unlabeled image samples. In this embodiment, the sample data all came from West China Hospital of Sichuan University, and the labels were all added by professional doctors at the hospital.

[0065] Step S2: Construct an image recognition and classification model;

[0066] An image recognition and classification model is constructed, which includes a backbone teacher network T, a backbone student network S1, a backbone student network S2, a projection module, and a prediction module.

[0067] The backbone teacher network T, backbone student network S1, and backbone student network S2 have the same structure and all use ResNet networks, with ResNet-50 being the preferred choice. Both the projection module and the prediction module are fully connected layers.

[0068] Step S3: Train the image recognition and classification model;

[0069] The image recognition and classification model constructed in step S2 is trained using the sample data obtained in step S1, such as... Figure 2 As shown, it specifically includes:

[0070] Phase 1: Supervised learning training of the backbone teacher network T using labeled image sample data;

[0071] Phase Two: The supervised learning-trained backbone teacher network T is replicated to obtain two backbone teacher networks T. One backbone teacher network T, together with the backbone student network S1, the projection module, and the prediction module, forms an online encoder. The other backbone teacher network T, together with the backbone student network S2, and the projection module, forms a target encoder. Unlabeled image sample data is input into the online encoder and the target encoder after undergoing two identical random data augmentations. In the online encoder, the backbone teacher network T and the backbone student network S1 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S1 is passed through the projection module and the prediction module in sequence, and the output of the prediction module becomes the output of the online encoder. In the target encoder, the backbone teacher network T and the backbone student network S2 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S2 is input into the projection module, and the output of the projection module becomes the output of the target encoder. Finally, constraints are applied to the feature map differences between the outputs of the online encoder and the target encoder.

[0072] Phase 3: Supervised learning training of the backbone student network S1 is performed using labeled image sample data to obtain the final image recognition and classification model.

[0073] In this embodiment, the model training comprises three stages. First, in the first stage, the model undergoes supervised training using labeled esophageal endoscopy images, forming the "teacher" model (backbone network T). In the second stage, two branches are established: an online encoder and a target encoder. Unlabeled esophageal endoscopy images, after undergoing the same random data augmentation processing, are input into both branches. The output differences between the two encoders are optimized. Simultaneously, through knowledge distillation, the "teacher" model (backbone network T) in the first stage teaches the "student" models (backbone networks S1 and S2) in the second stage, thereby better learning the features of the unlabeled images. In the third stage, the student model (backbone network S1), after self-supervised learning and feature distillation, further learns, trains, and refines the labeled data. Compared to models pre-trained on natural images, the model in this embodiment is expected to learn more features from esophageal endoscopy images. Figure 3 As shown, in the second stage, a distillation component is added, utilizing a pre-trained supervised model T to predict features of unlabeled esophageal endoscopy images and optimize the loss. Specifically, the data-augmented unlabeled data is simultaneously input into the "teacher" model T (backbone network T) trained in the first stage and the new "student" models Sj (backbone networks S1 and S2). Feature maps from n "teacher" models are extracted, denoted as... , and The feature maps of the corresponding "student" models are denoted as follows: , and The loss between the feature maps corresponding to the teacher network and the student network is denoted as... The sum of the losses among all selected corresponding feature maps is denoted as the distillation loss. In this embodiment, knowledge distillation can learn from unlabeled data in a self-supervised learning manner. Furthermore, the online encoder branch uses traditional backpropagation to update model parameters, while the target encoder branch uses the momentum update method proposed by He to update parameters. At the end of the second stage, neither the parameters of the target encoder nor the projection and predictor of the online encoder are saved; only the "student" model from the online encoder branch is retained. The parameters.

[0074] Distillation loss of backbone teacher network T and backbone student network S1 in online encoder for:

[0075] ;

[0076] Distillation loss of backbone teacher network T and backbone student network S2 in the target encoder for:

[0077] ;

[0078] Total distillation loss for:

[0079] ;

[0080] in, Indicates the number of feature maps selected. Indicates a teacher network, Indicates student network, This represents the i-th feature map in the online encoder. This represents the i-th feature map in the target encoder. This represents the i-th feature map of the teacher network in the online encoder. This represents the i-th feature map of the student network in the online encoder. This represents the i-th feature map of the teacher network in the target encoder. Let i represent the i-th feature map of the student network in the target encoder.

[0081] Complete (distillation) loss Represented as:

[0082] ;

[0083] in, The symmetric loss, representing the distillation loss, is the distillation loss obtained by swapping the sample data input to backbone network S1 and backbone network S2. .

[0084] Self-supervised loss of online encoders and target encoders for:

[0085] ;

[0086] in, This represents the parameters of the backbone network in the online encoder. This represents the mapping between the projection layer and the prediction layer in an online encoder. This represents the mapping of the backbone network in the online encoder. This represents the output of the online encoder. This represents the mapping of the backbone network in the target encoder. This represents the output of the target encoder.

[0087] Complete self-monitored loss Represented as:

[0088] ;

[0089] in, Indicates self-monitoring loss Symmetrical loss.

[0090] in, Symmetric loss representing self-supervised loss The self-supervised loss obtained by swapping the sample data input to backbone network S1 and backbone network S2 is the symmetric loss. .

[0091] Introducing hyperparameters To balance the full self-monitoring loss and complete (distillation) loss The overall loss function of the image recognition and classification model is expressed as:

[0092] .

[0093] Step S4: Real-time classification of esophageal endoscopy images;

[0094] The acquired real-time esophageal endoscopy images are input into step S3 to obtain the final image recognition and classification model, which then outputs the classification results.

[0095] Example 2

[0096] This embodiment provides a self-supervised recognition system based on esophageal endoscopy images, such as... Figure 1 As shown, the specific steps include:

[0097] The sample data acquisition module is used to acquire esophageal endoscopy image samples, including labeled and unlabeled image samples. In this embodiment, the sample data all come from West China Hospital of Sichuan University, and the labels were all added by professional doctors at the hospital.

[0098] The image recognition and classification model building module is used to build an image recognition and classification model. The image recognition and classification model includes a backbone teacher network T, a backbone student network S1, a backbone student network S2, a projection module, and a prediction module.

[0099] The backbone teacher network T, backbone student network S1, and backbone student network S2 have the same structure and all use ResNet networks, with ResNet-50 being the preferred choice. Both the projection module and the prediction module are fully connected layers.

[0100] The image recognition and classification model training module is used to train the image recognition and classification model constructed by the image recognition and classification model construction module using sample data obtained by the sample data acquisition module. Figure 2 As shown, it specifically includes:

[0101] Phase 1: Supervised learning training of the backbone teacher network T using labeled image sample data;

[0102] Phase Two: The supervised learning-trained backbone teacher network T is replicated to obtain two backbone teacher networks T. One backbone teacher network T, together with the backbone student network S1, the projection module, and the prediction module, forms an online encoder. The other backbone teacher network T, together with the backbone student network S2, and the projection module, forms a target encoder. Unlabeled image sample data is input into the online encoder and the target encoder after undergoing two identical random data augmentations. In the online encoder, the backbone teacher network T and the backbone student network S1 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S1 is passed through the projection module and the prediction module in sequence, and the output of the prediction module becomes the output of the online encoder. In the target encoder, the backbone teacher network T and the backbone student network S2 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S2 is input into the projection module, and the output of the projection module becomes the output of the target encoder. Finally, constraints are applied to the feature map differences between the outputs of the online encoder and the target encoder.

[0103] Phase 3: Supervised learning training of the backbone student network S1 is performed using labeled image sample data to obtain the final image recognition and classification model.

[0104] In this embodiment, the model training comprises three stages. First, in the first stage, the model undergoes supervised training using labeled esophageal endoscopy images, forming the "teacher" model (backbone network T). In the second stage, two branches are established: an online encoder and a target encoder. Unlabeled esophageal endoscopy images, after undergoing the same random data augmentation processing, are input into both branches. The output differences between the two encoders are optimized. Simultaneously, through knowledge distillation, the "teacher" model (backbone network T) in the first stage teaches the "student" models (backbone networks S1 and S2) in the second stage, thereby better learning the features of the unlabeled images. In the third stage, the student model (backbone network S1), after self-supervised learning and feature distillation, further learns, trains, and refines the labeled data. Compared to models pre-trained on natural images, the model in this embodiment is expected to learn more features from esophageal endoscopy images. Figure 3 As shown, in the second stage, a distillation component is added, utilizing a pre-trained supervised model T to predict features of unlabeled esophageal endoscopy images and optimize the loss. Specifically, the data-augmented unlabeled data is simultaneously input into the "teacher" model T (backbone network T) trained in the first stage and the new "student" models Sj (backbone networks S1 and S2). Feature maps from n "teacher" models are extracted, denoted as... , and The feature maps of the corresponding "student" models are denoted as follows: , and The loss between the feature maps corresponding to the teacher network and the student network is denoted as... The sum of the losses among all selected corresponding feature maps is denoted as the distillation loss. In this embodiment, knowledge distillation can learn from unlabeled data in a self-supervised learning manner. Furthermore, the online encoder branch uses traditional backpropagation to update model parameters, while the target encoder branch uses the momentum update method proposed by He to update parameters. At the end of the second stage, neither the parameters of the target encoder nor the projection and predictor of the online encoder are saved; only the "student" model from the online encoder branch is retained. The parameters.

[0105] Distillation loss of backbone teacher network T and backbone student network S1 in online encoder for:

[0106] ;

[0107] Distillation loss of backbone teacher network T and backbone student network S2 in the target encoder for:

[0108] ;

[0109] Total distillation loss for:

[0110] ;

[0111] in, Indicates the number of feature maps selected. Indicates a teacher network, Indicates student network, This represents the i-th feature map in the online encoder. This represents the i-th feature map in the target encoder. This represents the i-th feature map of the teacher network in the online encoder. This represents the i-th feature map of the student network in the online encoder. This represents the i-th feature map of the teacher network in the target encoder. Let i represent the i-th feature map of the student network in the target encoder.

[0112] Complete (distillation) loss Represented as:

[0113] ;

[0114] in, The symmetric loss, representing the distillation loss, is the distillation loss obtained by swapping the sample data input to backbone network S1 and backbone network S2. .

[0115] Self-supervised loss of online encoders and target encoders for:

[0116] ;

[0117] in, This represents the parameters of the backbone network in the online encoder. This represents the mapping between the projection layer and the prediction layer in an online encoder. This represents the mapping of the backbone network in the online encoder. This represents the output of the online encoder. This represents the mapping of the backbone network in the target encoder. This represents the output of the target encoder.

[0118] Complete self-monitored loss Represented as:

[0119] ;

[0120] in, Indicates self-monitoring loss Symmetrical loss.

[0121] in, Symmetric loss representing self-supervised loss The self-supervised loss obtained by swapping the sample data input to backbone network S1 and backbone network S2 is the symmetric loss. .

[0122] Introducing hyperparameters To balance the full self-monitoring loss and complete (distillation) loss The overall loss function of the image recognition and classification model is expressed as:

[0123] .

[0124] The real-time esophageal endoscopy image classification module is used to input the acquired real-time esophageal endoscopy images into the image recognition classification model training module to obtain the final image recognition classification model, and the image recognition classification model outputs the classification result.

[0125] Example 3

[0126] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform steps of a self-supervised recognition method based on esophageal endoscopy images.

[0127] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0128] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D-interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the memory may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the self-supervised recognition method based on esophageal endoscopy images. In addition, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0129] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the computer device. In this embodiment, the processor is used to run program code stored in the memory or process data, for example, to run the program code of the self-supervised recognition method based on esophageal endoscopy images.

[0130] Example 4

[0131] A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform steps of a self-supervised recognition method based on esophageal endoscopy images.

[0132] The computer-readable storage medium stores an interface display program that can be executed by at least one processor to cause the at least one processor to perform the steps of the self-supervised recognition method based on esophageal endoscopy images as described above.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the self-supervised recognition method based on esophageal endoscopy images described in the embodiments of this application.

Claims

1. A self-supervised recognition method based on esophageal endoscopic images, characterized in that, Includes the following steps: Step S1: Obtain sample data; Acquire esophageal endoscopy image samples, including labeled image sample data and unlabeled image sample data; Step S2: Construct an image recognition and classification model; An image recognition and classification model is constructed, which includes a backbone teacher network T, a backbone student network S1, a backbone student network S2, a projection module, and a prediction module. Step S3: Train the image recognition and classification model; The image recognition and classification model constructed in step S2 is trained using the sample data obtained in step S1, specifically including: Phase 1: Supervised learning training of the backbone teacher network T using labeled image sample data; Phase Two: The supervised learning-trained backbone teacher network T is replicated to obtain two backbone teacher networks T. One backbone teacher network T, together with the backbone student network S1, the projection module, and the prediction module, forms an online encoder. The other backbone teacher network T, together with the backbone student network S2, and the projection module, forms a target encoder. Unlabeled image sample data is input into the online encoder and the target encoder, respectively. In the online encoder, the backbone teacher network T and the backbone student network S1 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S1 is passed through the projection module and the prediction module in sequence, and the output of the prediction module becomes the output of the online encoder. In the target encoder, the backbone teacher network T and the backbone student network S2 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S2 is input into the projection module, and the output of the projection module becomes the output of the target encoder. Finally, constraints are applied to the feature map differences between the outputs of the online encoder and the target encoder. Phase 3: Supervised learning training of the backbone student network S1 is performed using labeled image sample data to obtain the final image recognition and classification model; Self-supervised loss of online encoders and target encoders for: ； in, This represents the parameters of the backbone network in the online encoder. This represents the mapping between the projection layer and the prediction layer in an online encoder. This represents the mapping of the backbone network in the online encoder. This represents the output of the online encoder. This represents the mapping of the backbone network in the target encoder. This represents the output of the target encoder; Step S4: Real-time classification of esophageal endoscopy images; The acquired real-time esophageal endoscopy images are input into step S3 to obtain the final image recognition and classification model, which then outputs the classification results.

2. The self-supervised recognition method based on esophageal endoscopy images as described in claim 1, characterized in that: In step S2, the backbone teacher network T, backbone student network S1, and backbone student network S2 have the same structure, all using ResNet networks, and both the projection module and the prediction module are fully connected layers.

3. The self-supervised recognition method based on esophageal endoscopy images as described in claim 1, characterized in that: In step S3, the distillation loss of the backbone teacher network T and the backbone student network S1 in the online encoder is... for: ； Distillation loss of backbone teacher network T and backbone student network S2 in the target encoder for: ； Total distillation loss for: ； in, Indicates the number of feature maps selected. Indicates a teacher network, Indicates student network, This represents the i-th feature map in the online encoder. This represents the i-th feature map in the target encoder. This represents the i-th feature map of the teacher network in the online encoder. This represents the i-th feature map of the student network in the online encoder. This represents the i-th feature map of the teacher network in the target encoder. Let i represent the i-th feature map of the student network in the target encoder.

4. The self-supervised recognition method based on esophageal endoscopy images as described in claim 3, characterized in that: Complete distillation loss function Represented as: ； in, This represents the symmetrical loss due to distillation.

5. The self-supervised recognition method based on esophageal endoscopy images as described in claim 1, characterized in that: Complete self-monitored loss Represented as: ； in, Indicates self-monitoring loss Symmetrical loss.

6. A self-supervised recognition system based on esophageal endoscopic images, characterized in that, include: The sample data acquisition module is used to acquire esophageal endoscopy image samples, which include labeled image sample data and unlabeled image sample data. The image recognition and classification model building module is used to build an image recognition and classification model. The image recognition and classification model includes a backbone teacher network T, a backbone student network S1, a backbone student network S2, a projection module, and a prediction module. The image recognition and classification model training module is used to train the image recognition and classification model constructed by the image recognition and classification model construction module using sample data obtained by the sample data acquisition module. Specifically, it includes: Phase 1: Supervised learning training of the backbone teacher network T using labeled image sample data; Phase Two: The supervised learning-trained backbone teacher network T is replicated to obtain two backbone teacher networks T. One backbone teacher network T, together with the backbone student network S1, the projection module, and the prediction module, forms an online encoder. The other backbone teacher network T, together with the backbone student network S2, and the projection module, forms a target encoder. Unlabeled image sample data is input into the online encoder and the target encoder, respectively. In the online encoder, the backbone teacher network T and the backbone student network S1 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S1 is passed through the projection module and the prediction module in sequence, and the output of the prediction module becomes the output of the online encoder. In the target encoder, the backbone teacher network T and the backbone student network S2 undergo knowledge distillation and self-supervised learning. The output of the backbone student network S2 is input into the projection module, and the output of the projection module becomes the output of the target encoder. Finally, constraints are applied to the feature map differences between the outputs of the online encoder and the target encoder. Phase 3: Supervised learning training of the backbone student network S1 is performed using labeled image sample data to obtain the final image recognition and classification model; Self-supervised loss of online encoders and target encoders for: ； in, This represents the parameters of the backbone network in the online encoder. This represents the mapping between the projection layer and the prediction layer in an online encoder. This represents the mapping of the backbone network in the online encoder. This represents the output of the online encoder. This represents the mapping of the backbone network in the target encoder. This represents the output of the target encoder; The real-time esophageal endoscopy image classification module is used to input the acquired real-time esophageal endoscopy images into the image recognition classification model training module to obtain the final image recognition classification model, and the image recognition classification model outputs the classification result.

7. A computer device, characterized in that: It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The device stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 5.

Images