Stomach representative image classification method and system based on depth-assisted contrast learning

Abstract

The invention provides a stomach representative image classification method and system based on depth-assisted contrast learning, and relates to the technical field of medical image processing, and the method comprises the steps: obtaining a depth estimation map of a frame image; obtaining a pose transformation matrix between adjacent frames; performing image reconstruction by taking the depth estimation map of the frame image and the pose transformation matrix between the adjacent frames as an intermediate quantity, performing special transformation and image reconstruction on a translation vector, respectively obtaining positive and negative samples, extracting features, and obtaining projection vectors of the features of the positive and negative samples; calculating and constructing a loss function by maximizing the projection vector similarity between the positive samples and minimizing the projection vector similarity between the negative samples; and inputting the endoscopic image with the mark whether the image is the representative stomach image into a comparative learning encoder to extract features, weighting the output features of the encoder by using the depth estimation map, and inputting the weighted features into a classifier to complete classification. According to the invention, the classification accuracy of the representative stomach image is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular, relates to a method and system for classifying representative stomach images based on deep assisted contrast learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Gastric cancer (GC) is one of the most common cancers worldwide and ranks third in cancer-related deaths. Due to the mild symptoms of GC in the early stage, the 5-year survival rate of patients is only 30%, and they are usually diagnosed in the late stage. On the other hand, if GC is found and diagnosed in the early stage, curative resection can be achieved, and the 5-year survival rate may increase to 95%. Therefore, early detection is one of the best ways to reduce the mortality rate of GC. Gastrointestinal metaplasia (GIM) is a common precancerous lesion associated with the development of early GC (EGC) and is also a key stage for gastric cancer to progress from chronic gastritis to adenocarcinoma. Patients with severe GIM have a tenfold increased risk of GC. Therefore, it is very important to identify the severity of GIM. In recent years, the Endoscopic Grading of Gastric Intestinal Metaplasia (EGGIM) system has been introduced to evaluate GIM in the antrum, notch, and body of the stomach and predict the risk of EGC through high-quality endoscopic images. EGGIM has an accuracy of 98% in grading the severity of GIM. EGGIM has been widely recognized and validated and is recommended by European guidelines. EGGIM grades GIM based on the proportion of GIM in the field of view in endoscopic images: no GIM is grade 0, GIM proportion ≤30% is grade 1, and GIM proportion >30% is grade 2. Images taken at inappropriate distances or angles can lead to significant errors in estimating the proportion of GIM in the field of view in endoscopic images. For example, an image that is actually GIM grade 1 but is too close to the image can easily be judged as GIM grade 2, while an image that is actually GIM grade 2 but is too far from the image can easily be judged as GIM grade 1. In summary, the classification of representative images of the stomach is crucial for GIM grading and EGC prediction.

[0004] However, it is difficult to obtain high-quality endoscopic images of the stomach, which means not only a clear and unobstructed field of view, but also that the distance between the endoscope and the gastrointestinal tissue needs to be moderate to fully include one site and not include other sites. In addition, the human stomach is a flexible, shape-changing organ that needs to be inflated during endoscopy to fully expand the gastric mucosal folds for clear observation of the gastric mucosa. This expansion can cause significant changes in various parts of the stomach, making it more difficult to capture and retain high-quality endoscopic images, especially in the greater and lesser curvatures of the gastric body, the greater and lesser curvatures of the gastric antrum, and the notch. We collectively refer to high-quality endoscopic images of these five parts as representative images of the stomach. Even experienced endoscopists cannot guarantee that the images they capture are necessarily representative images of the stomach.

[0005] Recently, deep learning-based methods have shown great advantages in computer-aided diagnosis. Current deep learning methods rely on a large amount of labeled data to achieve better performance. However, labeled data for representative and non-representative images of the stomach are limited, because reviewing endoscopic videos and labeling representative and non-representative images of the stomach is an extremely time-consuming and expensive task, which usually requires annotation by experienced endoscopists. Recently, self-supervised learning has emerged as a promising alternative to manually labeled data. It aims to learn representations from the structure of unlabeled data instead of relying on supervised losses involving manual labeling. As a key method in self-supervised learning, contrastive learning has shown impressive results on a range of image classification tasks. In contrastive learning, general image representations are initially learned from an unlabeled dataset and can subsequently be fine-tuned with a small set of labeled images to improve performance on specific tasks.

[0006] However, there are the following technical problems in the prior art: First, traditional contrastive learning often uses data augmentation to construct positive samples, using other images and augmented images in the same batch as negative samples of the input image. The positive and negative samples obtained in this way may belong to the same category, which is difficult to provide sufficient category discrimination information, making it impossible for the model to fully learn the distinguishable features between the two categories.

[0007] Secondly, if the positive and negative samples are explicitly constructed with the help of some auxiliary information that can help distinguish different categories, the category distinction between positive and negative samples can be enhanced. However, the number of explicitly constructed negative samples is usually small, which makes it easy for the contrastive learning model to close the distance between positive samples, and then map the samples to a single point or very close points in the feature space, causing the model to crash.

[0008] Finally, the depth estimation map of the labeled image can represent the shooting distance between the endoscope lens and the stomach, which is an important reference for classifying representative images of the stomach. However, the fine-tuning method in contrastive learning only uses the output features of the pre-trained encoder for image classification, lacking the use of the depth estimation map of the labeled image. Summary of the invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a representative stomach image classification method and system based on depth-assisted contrast learning. The present invention performs a specific transformation on the depth estimation map and the pose transformation matrix through the proposed positive and negative sample construction method based on the depth estimation map and posture, and then uses them as intermediate quantities for image reconstruction to obtain positive and negative samples, thereby increasing the category distinction of positive and negative samples.

[0010] In addition, the encoder is replaced with ConvNeXt by the proposed adjacent frame contrast learning method based on ConvNeXt encoder to enhance the feature extraction capability; the negative sample set is expanded and diversified by merging negative samples of adjacent frames, and the similarity between negative samples is incorporated into the loss function and minimized, thereby improving the encoder's category distinction ability and preventing model collapse. Finally, in the fine-tuning stage, the depth estimation map of the labeled image is transformed into a multi-scale and the encoder features are weighted by the proposed feature weighting method based on the multi-scale depth estimation map. The weighted features can make full use of the depth information of the labeled image.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: A first aspect of the present invention provides a method for classifying representative stomach images based on depth-assisted contrast learning.

[0012] Representative stomach image classification methods based on deep assisted contrast learning include: Acquire endoscopic video frames, construct and pre-train a depth estimation network, input the video frames into the pre-trained depth estimation network, and obtain a depth estimation map of the frame image; Obtain adjacent video frame pairs of the endoscope, construct and pre-train a pose estimation network, input the adjacent video frame pairs into the pre-trained pose estimation network, and obtain the pose transformation matrix between adjacent frames; The depth estimation map of the frame image and the pose transformation matrix between adjacent frames are used as intermediate quantities to reconstruct the image and obtain positive samples; the translation vectors in the depth estimation map and the pose transformation matrix are specially transformed to obtain negative samples after image reconstruction; Construct and train a contrastive learning encoder: extract positive and negative sample features through the ConvNeXt encoder, and obtain the projection vectors of the positive and negative sample features through the projection head; calculate the similarity between the positive sample projection vectors, the cross similarity between the positive and negative sample projection vectors, and the similarity between the negative sample projection vectors, and construct a loss function by maximizing the similarity between the positive sample projection vectors and minimizing the similarity between the negative sample projection vectors; The endoscopic image marked with whether it is a representative image of the stomach is input into the contrastive learning encoder to extract features, and is also input into the pre-trained depth estimation network to obtain a depth estimation map; the encoder output features are weighted using the depth estimation map, and the weighted features are input into the classifier to complete the classification.

[0013] A second aspect of the present invention provides a representative stomach image classification system based on deep assisted contrast learning.

[0014] A representative stomach image classification system based on deep assisted contrast learning, including: The depth estimation module is configured to obtain an endoscopic video frame, construct and pre-train a depth estimation network, input the video frame into the pre-trained depth estimation network, and obtain a depth estimation map of the frame image; A pose estimation module is configured to obtain adjacent video frame pairs of the endoscope, construct and pre-train a pose estimation network, input the adjacent video frame pairs into the pre-trained pose estimation network, and obtain a pose transformation matrix between adjacent frames; The positive and negative sample construction module is configured to reconstruct the image using the depth estimation map of the frame image and the pose transformation matrix between adjacent frames as intermediate quantities to obtain a positive sample; perform a special transformation on the translation vector in the depth estimation map and the pose transformation matrix to obtain a negative sample after image reconstruction; The adjacent frame contrast learning module is configured to construct and train a contrast learning encoder: extract positive and negative sample features through the ConvNeXt encoder, and obtain the projection vectors of the positive and negative sample features through the projection head; respectively calculate the similarity between the positive sample projection vectors, the cross similarity between the positive and negative sample projection vectors, and the similarity between the negative sample projection vectors, and construct a loss function by maximizing the similarity between the projection vectors of the positive samples and minimizing the similarity between the projection vectors of the negative samples; The fine-tuning module is configured to input the endoscopic image marked as a representative image of the stomach into a contrastive learning encoder to extract features, and at the same time input it into a pre-trained depth estimation network to obtain a depth estimation map; use the depth estimation map to weight the encoder output features, and input the weighted features into the classifier to complete classification.

[0015] The third aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a method as described in the first aspect of the present invention are implemented.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a method as described in the first aspect of the present invention.

[0017] One or more of the above technical solutions have the following beneficial effects: The present invention obtains a depth estimation map and a pose transformation matrix of an endoscopic image through depth estimation, and obtains positive and negative samples with different depths and perspectives through specific transformation and image reconstruction, thereby improving the category distinction of positive and negative samples and overcoming the problems of constructing positive and negative samples and separating classification tasks.

[0018] The adjacent frame contrast learning method based on ConvNeXt encoder proposed in the present invention makes up for the defect of insufficient number of negative samples of display construction by merging negative sample sets of adjacent frames and incorporating the similarity between negative samples into the loss function, which can effectively prevent model collapse.

[0019] The feature weighting method based on multi-scale depth estimation graph proposed in this invention makes full use of the distance information between the endoscope and the stomach by weighting the encoder output features in the fine-tuning stage with depth information, thereby improving the classification accuracy of representative images of the stomach. In clinical applications, it not only reduces the burden on endoscopists, but also reduces their missed diagnosis rate.

[0020] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0022] Figure 1 This is a network structure diagram of a network model in a method for classifying representative stomach images based on deep assisted contrast learning in Example 1 of the present invention; Figure 2 This is a structural diagram of feature weighting in a representative stomach image classification method based on deep assisted contrast learning according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0023] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0024] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0025] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0026] Embodiment 1 This embodiment discloses a method for classifying representative stomach images based on deep assisted contrast learning. Figure 1 To more clearly illustrate this embodiment, a method for classifying representative stomach images based on deep assisted contrast learning can be specifically described as follows: Step S101: construct and pre-train a depth estimation network, input a video frame into the pre-trained depth estimation network, and obtain a depth estimation map of the frame image.

[0027] The depth estimation network includes a depth encoder and a depth decoder. The video frame is input into the pre-trained depth encoder for feature encoding, and then the pre-trained depth decoder is used to decode the encoded features to obtain a depth estimation map of the frame image.

[0028] Specifically, the following detailed steps are included: Step S1011: Use the video processing library to open the video file (specifically, the endoscope video) and obtain the basic properties of the video, such as the total number of frames, frame rate, etc. We specify the sampling interval per second as the frame rate divided by five and rounded up. Then traverse all frames of the video, selectively read and save frame images according to the sampling interval, and assign a unique file name to each saved frame image.

[0029] Step S1012: We build a deep encoder and a deep decoder based on the Lite-Mono model and pre-train them using endoscopic videos.

[0030] The deep encoder includes three stages: initial convolution, continuous dilated convolution, and local-global feature exchange. In the initial convolution stage, a 3×3 convolution kernel is used with a first-layer step size of 2 for dimensionality reduction, followed by two layers of 3×3 convolutions with a step size of 1 to deepen feature extraction. In the continuous dilated convolution stage, 3×3 convolutions with a step size of 2 are used to further reduce the dimensionality, and then dilated convolutions with different dilation rates are introduced to increase the receptive field and enrich the feature representation. In the local-global feature exchange module stage, a self-attention layer is introduced to capture the long-distance dependency between features, which is integrated with the features of the previous stage to improve the sensitivity of the model.

[0031] A deep decoder consisting of an upsampling block and a prediction head. The upsampling block uses bilinear upsampling to increase the spatial dimension and uses a convolutional layer to concatenate the encoder features. The prediction head is built with 1×1 convolution, bilinear upsampling, and a sigmoid function.

[0032] Step S1013: In the deep encoder, downsampling and continuous dilated convolution are first used to extract rich multi-scale local features. For input features of size H×W×C, the output of continuous dilated convolution is as follows: (1) in represents a point-by-point convolution operation, then G is the GELU activation function, BN is a batch normalization layer, It is a 3×3 depth-dilated convolution with a dilation rate of r.

[0033] Step S1014: Encode the global context information into the previously extracted local features through the self-attention mechanism. Given an input feature map of dimension H×W×C, it is linearly projected to a query map of the same dimension. ,key ,value ,in , ,and is the weight matrix. The cross-covariance attention mechanism is used to enhance the input X: (2) in, represents the Softmax activation function, express The transpose of .

[0034] Step S1015: After feature extraction and global information encoding, the depth decoder uses bilinear upsampling to increase the spatial dimension and uses a convolutional layer to connect the features of the encoder. The bilinear upsampling is followed by a prediction head to output a depth estimation map.

[0035] Step S102: Acquire adjacent pairs of endoscopic video frames, construct and pre-train a pose estimation network, input the adjacent video frame pairs into the pre-trained pose estimation network, and obtain a pose transformation matrix between adjacent frames.

[0036] The pose estimation network includes a pose encoder and a pose decoder. Adjacent video frame pairs are input into the pre-trained pose encoder for feature encoding, and then the pre-trained pose decoder is used to decode the encoded features to obtain the pose transformation matrix between adjacent frames.

[0037] Specifically, the following detailed steps are included: Step S1021: For the N video frames obtained in step S1011, the first N-1 video frames are used as the source frame sequence, and the last N-1 video frames are used as the target frame sequence. The frames in the source frame sequence and the target frame sequence correspond to each other one by one to form adjacent frame pairs.

[0038] Step S1022: We use the ResNet18 network without the classification head as the pose encoder and construct a pose decoder using four convolutional layers with a kernel size of 3×3. The number of channels in the first three layers is 256 and the number of channels in the last layer is 6. ReLU function is used as the activation function between each convolutional layer.

[0039] Step S1023: Input the adjacent frame pairs into the pose encoder for feature extraction. After that, the pose encoder features enter the pose decoder for upsampling decoding. Finally, the pose decoder outputs a 4×4 pose transformation matrix as the 6-DoF relative pose between adjacent frames, including a 3×3 rotation matrix and a 3×1 translation vector.

[0040] In this embodiment, the depth estimation network and the pose estimation network are self-supervised pre-trained using endoscopic video, and a depth estimation map and a pose transformation matrix are output respectively. The output depth estimation map and pose transformation matrix are used for image reconstruction, and the depth estimation network and the pose estimation network are updated by minimizing the image reconstruction loss and the edge-aware smoothing loss, so as to obtain a pre-trained depth encoder and depth decoder that can output an accurate depth estimation map, and a pose encoder and pose decoder that can accurately output a pose transformation matrix.

[0041] The pre-training of the pose estimation network and the depth estimation network is carried out simultaneously, where the input of the depth estimation network is the video frame, and the input of the pose estimation network is the adjacent frame pairs. The pre-training uses unlabeled endoscopic video frames to train the two networks, and optimizes the two networks by minimizing the image reconstruction loss and edge-aware smoothing loss. The specific construction process and pre-training process refer to the Lite-Mono model. Our method specifically uses the two pre-trained networks to directly generate more accurate depth estimation maps and pose transformation matrices.

[0042] Step S103: reconstruct the image using the depth estimation map of the frame image and the pose transformation matrix between adjacent frames as intermediate quantities to obtain a positive sample; then perform a special transformation on the depth estimation map and the translation vector in the pose transformation matrix to obtain a negative sample after image reconstruction.

[0043] Specifically, the depth estimation map of the frame image and the pose transformation matrix between adjacent frame pairs are used as intermediate quantities to reconstruct the image to obtain positive samples, and random rotation is used to amplify the positive samples; then, a special transformation is performed on the depth estimation map and the translation vector in the pose transformation matrix, and the depth estimation map and pose transformation matrix that have undergone different transformations are used as intermediate quantities to reconstruct the image to obtain negative samples.

[0044] The detailed steps include: Step S1031: In the positive sample construction, each frame image in the input video First, it passes through the pre-trained depth estimation network, and then forms adjacent frame pairs with adjacent frame images and passes through the pose estimation network. Then, the outputs of the two networks are used for normal reconstruction: (3) in and The source images are and the pixel coordinates of the reconstructed image, K is the camera intrinsic parameter matrix. We use the input frame as the source image and the adjacent frame as the target image of the reconstructed image. We calculate the projection relationship between the source image and the reconstructed image through the above formula, and use bilinear sampling to reconstruct the source image to obtain the reconstructed image. . Afterwards and Perform random rotation to obtain and , a total of three positive samples, achieving positive sample amplification.

[0045] Step S1032: Randomly increase or decrease the X-axis translation vector in the posture transformation matrix within a certain range: (4) in represents the translation vector, is a random number between [-0.01, -0.005] ∨ [0.005, 0.01], which is obtained from multiple experiments. >0, it means the camera moves to the right. When <0, it means the camera moves to the left. Adjust the depth estimation image according to the relationship between the depth estimation image and the camera perspective: (5) in represents the depth estimation map, Indicates the focal length of the camera in the horizontal component. Use and After image reconstruction, we get a negative sample image with left and right viewing angle changes. and .

[0046] Step S1033: Randomly increase or decrease the Y-axis translation vector in the posture transformation matrix within a certain range: (6) in represents the translation vector, is a random number between [-0.01, -0.005] ∨ [0.005, 0.01], which is obtained from multiple experiments. >0, it means the camera moves upward. When <0, it means the camera moves downward. Adjust the depth estimation image according to the relationship between the depth estimation image and the camera viewing angle: (7) in represents the depth estimation map, Indicates the focal length of the camera in the vertical component, using and After image reconstruction, negative sample images with up and down viewing angles are obtained. and .

[0047] Step S1034: Randomly increase or decrease the Z-axis translation vector in the posture transformation matrix within a certain range: (8) in represents the translation vector, is a random number between [-0.01, -0.005] ∨ [0.005, 0.01], which is obtained from multiple experiments. >0, it means the camera moves inward. When <0, it means the camera moves outward. Adjust the depth estimation image according to the relationship between the depth estimation image and the camera viewing angle: (9) in Represents the depth estimation map, using and After image reconstruction, a negative sample image with varying depth and size is obtained. and .

[0048] Step S104: Construct and train a contrastive learning encoder: extract positive and negative sample features through the ConvNeXt encoder, and obtain the projection vectors of the positive and negative sample features through the projection head, and then add the negative sample projection vector of the frame after the input frame to the negative sample projection vector set of the input frame; calculate the similarity between the positive sample projection vectors, the cross similarity between the positive and negative sample projection vectors, and the similarity between the negative sample projection vectors, and construct a loss function by maximizing the similarity between the positive sample projection vectors and minimizing the similarity between the negative sample projection vectors.

[0049] Specifically, the following detailed steps are included: Step S1041: Select the ConvNeXt network without the classification head as the encoder, and input the positive and negative samples X of all images in the batch data into the encoder in sequence Then h passes through the network projection head The vector z is obtained as shown in the following formula: (10) in and They are the first and second layers of the multilayer perceptron, is the ReLU activation function.

[0050] Step S1042: For the three positive sample vectors of each image, we take the average value of the cosine similarities between them as the positive sample similarity, as shown in the following formula: (11) in, is the temperature parameter, n1 is the number of positive samples 3, represents a positive sample vector pair, Represents the cosine similarity of two vectors, as shown in the following formula: (12) Step S1043: Assuming the batch size is N, for the first N-1 images, the negative sample vectors are the set of negative sample vectors of the current image and the next frame, which is 12 in number; for the Nth image, the negative sample vector is only the set of negative sample vectors of the current image, which is 6 in number. We calculate the cross similarity between negative samples and positive samples Similarity with negative samples , as shown in the following formula: (13) (14) Among them, n1 is the number of positive samples 3, for the first N-1 images, n2 is 12, and for the Nth image, n2 is 6.

[0051] Step S1044: We calculate the loss of each image , as shown in the following formula: (15) The overall loss L of a batch of data is shown in the following formula: (16) Where N is the batch size. The network updates the encoder and projection head of the corresponding part by continuously minimizing L, where the projection head is only used for self-supervised training and not for fine-tuning tasks.

[0052] Step S105: Input the endoscopic image marked as a representative image of the stomach into the contrastive learning encoder to extract features, and input it into the pre-trained depth estimation network to obtain a depth estimation map; use the depth estimation map to weight the encoder output features, input the weighted features into the classifier to complete classification, and implement fine-tuning of the ConvNext encoder.

[0053] Specifically, the following detailed steps are included: Step S1051: The labeled image I of size 224×224 is first passed through the ConvNeXt encoder trained in step S104 to generate a feature x of shape [64,2048,1,1], where 64 is the batch size, 2048 is the number of channels, and the feature height and width are 1. The labeled image is feature encoded by the deep encoder pre-trained in step S101, and then decoded by the deep decoder pre-trained in step S101, and after decoding, a depth estimation map of size 224×224 is output.

[0054] Step S1052: Figure 2 As shown, the depth estimation map of shape [1,224,224] The input is sent to two average pooling layers for 1 / 2 and 1 / 4 reduction, where 1 is the number of channels, and the depth estimation map size is 224×224, resulting in depth estimation maps with shapes of [1,112,112] and [1,56,56] respectively. and The depth estimation images of the three scales enter a convolution layer to increase the number of channels and obtain three features with 256 channels. The three features are then downsampled to 1×1 through a global average pooling layer.

[0055] Step S1053: Figure 2As shown in the figure, the three depth estimation features and the encoder output features are concatenated and fused in the channel dimension to obtain the fused feature F. F first passes through the convolution layer to reduce the number of channels to 2048, which is consistent with the number of channels of the output feature, and then the normalized weight α is obtained using the Sigmoid function. The weight α is applied to the encoder output feature to obtain the weighted feature w.

[0056] Step S1054: The weighted feature w is passed to the classifier, first through a flattening layer to reach a shape of [64, 2048], and then through a fully connected layer to obtain the classification result.

[0057] In order to speed up the convergence of the network, the weights and biases of each layer of the above network are randomized to a normal distribution with a mean of 0 and a variance of ,in is the number of upper-layer features. The cross entropy function is used to define the loss function for the fine-tuning task. At the same time, the Adam optimizer is used in PyTorch to adjust the connection weights and biases. The maximum number of iterations of the self-supervised pre-trained network is set to 100, the maximum number of iterations of the final fine-tuned network is set to 40, and the loss threshold is set to 0.0001.

[0058] In this embodiment, step S104 is to use unlabeled video frame images for self-supervised training, and step S105 is to use labeled endoscopic images, i.e., whether they are representative images of the stomach, to fine-tune the constructed contrast learning encoder and implement classification.

[0059] Embodiment 2 This embodiment discloses a stomach representative image classification system based on deep assisted contrast learning, including: The depth estimation module is configured to obtain an endoscopic video frame, construct and pre-train a depth estimation network, input the video frame into the pre-trained depth estimation network, and obtain a depth estimation map of the frame image; A pose estimation module is configured to obtain adjacent video frame pairs of the endoscope, construct and pre-train a pose estimation network, input the adjacent video frame pairs into the pre-trained pose estimation network, and obtain a pose transformation matrix between adjacent frames; The positive and negative sample construction module is configured to reconstruct the image using the depth estimation map of the frame image and the pose transformation matrix between adjacent frames as intermediate quantities to obtain a positive sample; perform a special transformation on the translation vector in the depth estimation map and the pose transformation matrix to obtain a negative sample after image reconstruction; The adjacent frame contrast learning module is configured to construct and train a contrast learning encoder: extract positive and negative sample features through the ConvNeXt encoder, and obtain the projection vectors of the positive and negative sample features through the projection head; respectively calculate the similarity between the positive sample projection vectors, the cross similarity between the positive and negative sample projection vectors, and the similarity between the negative sample projection vectors, and construct a loss function by maximizing the similarity between the projection vectors of the positive samples and minimizing the similarity between the projection vectors of the negative samples; The fine-tuning module is configured to input the endoscopic image marked as a representative image of the stomach into a contrastive learning encoder to extract features, and at the same time input it into a pre-trained depth estimation network to obtain a depth estimation map; use the depth estimation map to weight the encoder output features, and input the weighted features into the classifier to complete classification.

[0060] It should be noted that the depth estimation module corresponds to step S101 in the method, the pose estimation module corresponds to step S102 in the method, the positive and negative sample construction module corresponds to step S103 in the method, the adjacent frame comparison learning module corresponds to step S104 in the method, and the fine-tuning module corresponds to step S105 in the method. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the method.

[0061] Embodiment 3 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0062] Embodiment 4 The purpose of this embodiment is to provide a computer-readable storage medium.

[0063] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0064] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0065] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0066] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A representative stomach image classification method based on deep assisted contrast learning, include: Acquire endoscopic video frames, construct and pre-train a depth estimation network, input the video frames into the pre-trained depth estimation network, and obtain a depth estimation map of the frame image; Obtain adjacent video frame pairs of the endoscope, construct and pre-train a pose estimation network, input the adjacent video frame pairs into the pre-trained pose estimation network, and obtain the pose transformation matrix between adjacent frames; The depth estimation map of the frame image and the pose transformation matrix between adjacent frames are used as intermediate quantities to reconstruct the image and obtain positive samples; the translation vectors in the depth estimation map and the pose transformation matrix are specially transformed to obtain negative samples after image reconstruction; Construct and train a contrastive learning encoder: extract positive and negative sample features through the ConvNeXt encoder, and obtain the projection vectors of the positive and negative sample features through the projection head; calculate the similarity between the positive sample projection vectors, the cross similarity between the positive and negative sample projection vectors, and the similarity between the negative sample projection vectors, and construct a loss function by maximizing the similarity between the positive sample projection vectors and minimizing the similarity between the negative sample projection vectors; The endoscopic image marked with whether it is a representative image of the stomach is input into the contrastive learning encoder to extract features, and is also input into the pre-trained depth estimation network to obtain a depth estimation map; the encoder output features are weighted using the depth estimation map, and the weighted features are input into the classifier to complete the classification.

2. A method for classifying representative stomach images based on deep assisted contrast learning as claimed in claim 1, It is characterized in that The construction and pre-training of the depth estimation network specifically comprises: the depth estimation network includes a depth encoder and a depth decoder, the depth encoder and the depth decoder are constructed based on the Lite-Mono model, and the endoscopic video is used to perform self-supervisory pre-training on the depth estimation network; The video frame is input into the pre-trained depth estimation network to obtain the depth estimation map of the frame image. The specific steps are as follows: In the deep encoder, downsampling and continuous dilated convolutions are used to extract multi-scale local features; Encode global context information into the extracted local features through a self-attention mechanism; After feature extraction and global information encoding, the deep decoder uses bilinear upsampling to increase the spatial dimension and uses a convolutional layer to connect the features of the encoder. The upsampling block is followed by a prediction head to output a depth estimation map.

3. A method for classifying representative stomach images based on deep assisted contrast learning as claimed in claim 1, It is characterized in that The pose estimation network includes a pose encoder and a pose decoder; Use the pre-trained ResNet18 as the pose encoder and take a pair of adjacent frames as input for feature extraction; A four-layer convolutional network is used as a pose decoder to estimate the corresponding 6-DoF relative poses between adjacent images.

4. The method for classifying representative stomach images based on deep assisted contrast learning as claimed in claim 1, It is characterized in that The depth estimation image and the translation vector in the pose transformation matrix are specially transformed, and negative samples are obtained after image reconstruction, specifically: The X-axis translation vector in the pose transformation matrix is ​​randomly increased or decreased within a certain range, and the depth estimation map is adjusted according to the relationship between the depth estimation map and the camera perspective. The transformed pose transformation matrix and depth estimation map are used for image reconstruction to obtain a negative sample image with left and right perspective changes. and ; The Y-axis translation vector in the pose transformation matrix is ​​randomly increased or decreased within a certain range, and the depth estimation map is adjusted according to the relationship between the depth estimation map and the camera perspective. The transformed pose transformation matrix and the depth estimation map are used for image reconstruction to obtain a negative sample image with up and down perspective changes. and ; The Z-axis translation vector in the pose transformation matrix is ​​randomly increased or decreased within a certain range, and the depth estimation map is adjusted according to the relationship between the depth estimation map and the camera perspective. The transformed pose transformation matrix and the depth estimation map are used for image reconstruction to obtain a negative sample image with a changed depth size. and .

5. The method for classifying representative stomach images based on deep assisted contrast learning as claimed in claim 1, It is characterized in that The image reconstruction uses a video frame as a source image, uses an adjacent frame as a target image for reconstructing an image, calculates a projection relationship between the source image and the reconstructed image, and uses bilinear sampling to achieve reconstruction of the source image; The calculation formula is as follows: in and are the pixel coordinates of the source image and the reconstructed image, respectively. K is the camera intrinsic parameter matrix. T and D are the transformed pose transformation matrix and depth estimation map, respectively.

6. A method for classifying representative stomach images based on deep assisted contrast learning as claimed in claim 1, It is characterized in that The positive and negative sample features are extracted by the ConvNeXt encoder, and the projection vectors of the positive and negative sample features are obtained by the projection head, specifically: Select the ConvNeXt network without the classification head as the encoder, and input the positive and negative samples X of all images in the batch data into the encoder in sequence to obtain the representation vector h; The vector h passes through a projection head containing two layers of perceptrons to obtain the vector z; After obtaining the projection vector of the feature, the negative sample projection vector of the adjacent frame is added to the negative sample projection vector set of the input frame.

7. A method for classifying representative stomach images based on deep assisted contrast learning as claimed in claim 1, It is characterized in that The similarity between the positive sample projection vectors, the cross similarity between the positive and negative sample projection vectors, and the similarity between the negative sample projection vectors are calculated respectively, and the loss function is constructed by maximizing the similarity between the positive sample projection vectors and minimizing the similarity between the negative sample projection vectors, specifically: For the three positive sample vectors of each image, we take the average of their cosine similarities as the positive sample similarity: in, is the temperature parameter, n1 is the number of positive samples 3, represents a positive sample vector pair, Represents the cosine similarity of two vectors; Calculate the cross similarity between negative samples and positive samples Similarity with negative samples : Among them, n1 is the number of positive samples 3, for the first N-1 images, n2 is 12, and for the Nth image, n2 is 6; Calculate the loss for each image : in, represents the similarity of positive samples, represents the cross similarity between negative samples and positive samples, Represents the similarity of negative samples.

8. The method for classifying representative stomach images based on deep assisted contrast learning as claimed in claim 1, It is characterized in that The endoscopic image with a mark of whether it is a representative image of the stomach is input into a contrastive learning encoder to extract features, and is simultaneously input into a pre-trained depth estimation network to obtain a depth estimation map; the encoder output features are weighted using the depth estimation map, and the weighted features are input into a classifier to complete classification, specifically: The labeled image I of size 224×224 is first passed through the pre-trained encoder to produce features x of shape [64,2048,1,1], where 64 is the batch size, 2048 is the number of channels, and the feature height and width are 1; The labeled image is feature encoded by a pre-trained deep encoder, and then decoded by bilinear upsampling and convolutional layers. After decoding, a prediction head is used to output a depth estimation map of size 224×224; The encoder output feature x with a shape of [64, 2048, 1, 1] is weighted using the depth estimation graph to obtain a weighted feature w with the same shape; The weighted feature w is passed to the classifier, first through a flattening layer to reach the shape of [64, 2048], and then through a fully connected layer to obtain the classification result.

9. A method for classifying representative stomach images based on deep assisted contrast learning as claimed in claim 8, It is characterized in that The encoder output feature x with a shape of [64, 2048, 1, 1] is weighted using the depth estimation graph to obtain a weighted feature w with the same shape, specifically: Depth estimation map with shape [1,224,224] The input is sent to two average pooling layers for 1 / 2 and 1 / 4 reduction, where 1 is the number of channels, and the depth estimation map size is 224×224, resulting in depth estimation maps with shapes of [1,112,112] and [1,56,56] respectively. and ; The depth estimation images of the three scales enter a convolution layer to increase the number of channels and obtain three features with a channel number of 256. Then, the three features are downsampled to 1×1 through a global average pooling layer. The three depth estimation map features and the encoder output features are concatenated and fused in the channel dimension to obtain the fused feature F; F first passes through the convolution layer to reduce the number of channels to 2048, which is consistent with the number of channels of the output feature, and then uses the Sigmoid function to obtain the normalized weight α; Apply weight α to the encoder output feature to obtain the weighted feature w.

10. A representative stomach image classification system based on deep assisted contrast learning, include: The depth estimation module is configured to obtain an endoscopic video frame, construct and pre-train a depth estimation network, input the video frame into the pre-trained depth estimation network, and obtain a depth estimation map of the frame image; A pose estimation module is configured to obtain adjacent video frame pairs of the endoscope, construct and pre-train a pose estimation network, input the adjacent video frame pairs into the pre-trained pose estimation network, and obtain a pose transformation matrix between adjacent frames; The positive and negative sample construction module is configured to reconstruct the image using the depth estimation map of the frame image and the pose transformation matrix between adjacent frames as intermediate quantities to obtain a positive sample; perform a special transformation on the translation vector in the depth estimation map and the pose transformation matrix to obtain a negative sample after image reconstruction; The adjacent frame contrast learning module is configured to construct and train a contrast learning encoder: extract positive and negative sample features through the ConvNeXt encoder, and obtain the projection vectors of the positive and negative sample features through the projection head; respectively calculate the similarity between the positive sample projection vectors, the cross similarity between the positive and negative sample projection vectors, and the similarity between the negative sample projection vectors, and construct a loss function by maximizing the similarity between the projection vectors of the positive samples and minimizing the similarity between the projection vectors of the negative samples; The fine-tuning module is configured to input the endoscopic image marked as a representative image of the stomach into a contrastive learning encoder to extract features, and at the same time input it into a pre-trained depth estimation network to obtain a depth estimation map; use the depth estimation map to weight the encoder output features, and input the weighted features into the classifier to complete classification.

Images