Colon polyp size detection method and system under colonoscope

Abstract

The invention discloses a method and a system for detecting the size of a colon polyp under a colonoscope. According to the method, the YOLOv5 colon polyp detection model and the DepthAnything depth estimation model are combined, the polyp in the colonoscopy image can be accurately detected, the accurate depth map can be estimated, then the distance between the intestinal wall and the lens in the enteroscopy can be determined by using the polyp bounding box and the depth map, and the accuracy of the colonoscopy is improved. And reconstructing a three-dimensional point cloud picture of the colonoscope image in the real world, calculating the size of the polyp in the point cloud, converting the size into the actual physical size of the polyp, and displaying the actual physical size in real time in the operation process. According to the method, accurate positioning and size measurement of the polyp are achieved, and measurement errors caused by lack of reference substances in a traditional method are avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a colonoscopic polyp size detection method and system. Background Art

[0002] Colonoscopy is an important tool in modern medicine for detecting and diagnosing colon diseases. By inserting a flexible tube with a camera into the rectum, doctors can observe the inside of the colon in real time and form a continuous video stream. This technology is crucial for detecting abnormal growths in the colon, such as polyps, because timely detection and removal of these tissues can greatly reduce the risk of colon cancer.

[0003] After colon polyps are detected, they are divided into small polyps ≤5 mm, small polyps >5~10 mm, large polyps >10~20 mm, and giant polyps >20 mm according to their diameter. Different sizes of polyps are treated differently. The size detection of colon polyps mainly depends on the doctor's experience and visual assessment. The doctor will make an estimate based on the appearance of the polyp on the screen and refer to instruments or colon walls of known sizes. However, this method is highly subjective and has errors, especially when there is a lack of suitable reference objects in the video, it is difficult to provide accurate measurement results, and incorrect estimation of the size of colon polyps may lead to waste of medical devices.

[0004] In order to solve these problems, research in recent years has begun to explore automated methods to assist in the detection of polyp size. Based on the perspective phenomenon of polyps in colonoscopy images, some early methods tried to use a preset distance to shoot images, and use a fixed zoom factor multiplied by the size of the polyp in the picture at a certain distance to predict the true size of the polyp. Or instead of shooting at a preset distance, a laser rangefinder lens is installed on the colonoscope lens to measure the distance between the intestinal wall in the center of the image and the lens in real time, and a virtual scale is superimposed on the screen with the measured distance. The doctor uses the virtual scale to measure the true size of the polyp. There are also reference objects of a certain size, such as biopsy forceps and water jets, and deep learning methods are used to automatically compare polyps and reference objects to predict the true size of polyps. However, from the perspective of practical application, some of these methods are inconvenient to operate, and some require the purchase of additional lenses. They still need to be further improved. Summary of the invention

[0005] In view of the problems existing in the above-mentioned background technology, the present invention proposes a method and system for detecting the size of colon polyps under colonoscopy. The present invention combines the YOLOv5 colon polyp detection model and the depth estimation model based on DepthAnything, which can not only accurately detect polyps in colonoscopic images, but also estimate accurate depth maps. Subsequently, the polyp bounding box and depth map can be used to determine the distance between the intestinal wall and the lens during colonoscopy, and reconstruct the three-dimensional point cloud map of the colonoscopic image in the real world, calculate the size of the polyp in the point cloud, convert it into the actual physical size of the polyp, and display it in real time during the operation.

[0006] The method of the present invention specifically comprises the following steps:

[0007] Step 1: Train the YOLOv5 colon polyp detection model

[0008] The present invention uses a data set of colon polyps to train a YOLOv5 target detection model, which is widely used in the field of target detection due to its high efficiency and accuracy.

[0009] Step 1.1: Preparation and preprocessing of colon polyp detection dataset

[0010] The colon polyp detection dataset consists of a large number of colonoscopy video frames, which contain a large number of various types of colon polyps and are professionally annotated to mark the presence and location of polyps. The annotation format is ,in is the offset of the center of the bounding box relative to the upper left corner of the image, are the ratios of the bounding box width and height relative to the image width and height, respectively.

[0011] The original video frames are normalized by cropping to remove the black edges, then filling the square with a grayscale value of 128 and scaling it to L×L using a bilinear interpolation method, where L represents the side length of the input image, to meet the requirements of the YOLOv5 model. Finally, the dataset is randomly divided into three parts: training set, validation set, and test set. The training set is used for model training, the validation set is used for hyperparameter adjustment, and the test set is used to evaluate model performance. At the same time, image enhancement techniques such as brightness adjustment, rotation, and mapping are applied to the training set to enhance the robustness of the model.

[0012] Step 1.2: YOLOv5 colon polyp detection model training process

[0013] First, load the pre-trained weights provided by YOLOv5 as the initial state of the model, and train it using the training set described in step 1.1. The model divides the feature map into a chessboard grid. blocks, each of which is associated with the output of the model indivual vectors, each of which represents a prediction box. Respectively represent the offset of the prediction box relative to the upper left corner of the block in the x and y directions, Represents the length and width of the prediction box, Indicates the probability that the predicted box contains the target, Represents the unique hot encoding of the predicted value of the prediction box category, with a total of categories. Loss function:

[0014]

[0015] in Represents the output feature map. The difference between different feature maps lies in the number of blocks. different; Used to balance the weights of each output feature map; Indicates The feature map Block Whether the predicted box is a positive sample (matching a true value), if it is a positive sample, it is 1, otherwise it is 0; is the weight of the corresponding loss function.

[0016]

[0017] is the complete intersection loss, Represent the predicted box and the true box respectively; Represents the distance between the center point of the predicted box and the real box, is the diagonal distance between the predicted box and the minimum enclosing rectangle of the real box; w G ,h G Represents the width and height of the prediction box, w P ,h P Represent the width and height of the real frame respectively.

[0018]

[0019] is the target confidence loss, where represents the binary cross entropy loss function, is the probability of prediction box prediction, is the predicted box and the corresponding real box value, Represents the weight of the positive sample.

[0020]

[0021] is the classification confidence loss, where is the cross entropy loss function, is the category predicted by the prediction box, For the real category, Represents the weight of the true category.

[0022] In each epoch, the model predicts the images in the training set and calculates the loss value according to the loss function, and then back-propagates to update the weights. In addition, the performance on the validation set is used to adjust the learning rate and other hyperparameters. After training is completed, the best model weights are saved and the model performance is evaluated on the test set, including indicators such as accuracy, recall, and mAP score.

[0023] Step 2: Train a DepthAnything-based depth estimation model

[0024] The present invention uses the colonoscopy depth estimation dataset to train the DepthAnything depth estimation model, and uses both labeled virtual images and unlabeled real images for semi-supervised training. By training on synthetic data with real labels, the model can learn the scale information of image depth and physical size; self-supervised learning in real clinical data can further optimize the model, improve the model's ability to understand complex conditions in the real environment, enhance the stability of depth estimation, and make it perform better in the real clinical environment. Specifically:

[0025] Step 2.1: Preparation and preprocessing of colonoscopy depth estimation dataset

[0026] The colonoscope depth estimation dataset includes a labeled virtual image dataset and an unlabeled real image dataset. The virtual image dataset is a public dataset, which generally includes images taken by an in vitro prosthesis or computer-simulated images. The dataset will provide the lens intrinsic parameter matrix and lens distortion coefficient The real image dataset is a series of unlabeled colonoscopy videos obtained from hospitals.

[0027] For real clinical data sets, manual calibration is required to obtain the endoscope lens intrinsic parameter matrix. and lens distortion coefficient First, prepare a black and white chessboard with known size and number of corner points, then place this calibration pattern at different positions and angles, use the endoscope lens to be calibrated to take multiple images containing the calibration pattern from multiple perspectives, and finally use the OpenCV library to solve the endoscope lens intrinsic parameter matrix And the lens distortion coefficient .

[0028] The data preprocessing part includes: if the data is a video, filter out the blurred frames of the video, retain the relatively high-quality parts, and then split it into pictures, and use the lens intrinsic parameter matrix and lens distortion coefficient to correct the pictures with fisheye lens. The depth truth value of the virtual image is distributed in mm, divided by To zoom to Finally, the true depth values ​​of the dedistorted input image and the virtual image are uniformly scaled to Pixel.

[0029] Step 2.2: Train the DepthAnything depth estimation model on a dataset of labeled virtual images

[0030] The DepthAnything depth estimation model is trained on a labeled virtual image dataset and is called the Teacher model. Model input image The dimension is , model output The dimension is , loss function:

[0031]

[0032] Step 2.3: Train the DepthAnything lighting adjustment model

[0033] The present invention uses a traditional pinhole camera model. The camera is located at the origin of the world coordinate system and looks in the negative direction of the Z axis, the X axis is horizontal to the left, and the Y axis is vertically pointing to the ground. The camera intrinsic parameter matrix is ​​defined as , define the point in the world coordinate system as and through Mapping to pixel coordinates .

[0034] First, the parameters obtained by lens calibration are used to remove the distortion of the input frame, and the processed image is scaled to size, and then use the trained Teacher model to predict its depth map , and then use the following formula to convert The pixels representing the depth values ​​are restored to the point cloud in the world coordinate system:

[0035]

[0036] Among them, the pixel coordinates are Pixels, represents its world coordinates, Indicates its distance from the camera (depth value).

[0037] After obtaining the point cloud, the per-pixel shading (PPS) of the image is calculated according to the following formula:

[0038]

[0039]

[0040] in is the coordinate of the co-location light of the endoscope, is the direction of light, is the angular attenuation coefficient of the illumination. It shows how bright a colonoscopy image should be when illuminated only by the co-located light from the lens.

[0041] Training the DepthAnything illumination adjustment model, grayscale image of the real image With its depth map The splicing result in the channel dimension is ,Will The side length of bilinear interpolation is downsampled by four times to As the model input, the model predicts a correction to the grayscale image of the real image , and then upsample four times by bilinear interpolation to get , loss function:

[0042]

[0043] After training is completed, the trained DepthAnything illumination adjustment model is used to predict the grayscale image of the real image. , get and save the result For subsequent model training.

[0044] Step 2.4: Train the DepthAnything depth estimation model using both virtual and real image datasets

[0045] The DepthAnything depth estimation model is trained using both virtual and real image datasets. This model is called the Student model. Model input image The dimension is , model output The dimension is , when the input is a virtual image, the loss function is:

[0046]

[0047] When the input is a real image, the loss function is:

[0048]

[0049] in yes enter The formula is obtained.

[0050] Step 3: Use the trained YOLOv5 colon polyp detection model and Student model to detect polyp size

[0051] The present invention jointly uses the YOLOv5 colon polyp detection model trained in step 1 and the Student model trained in step 2 to detect the size of polyps.

[0052] For the input image frame, first use the endoscope lens intrinsic parameter matrix obtained by step 2.1 lens calibration And the lens distortion coefficient Perform lens correction and divide by To zoom to In the range, bilinear interpolation is used to scale to Pixel, denoted as .

[0053] Will Input the YOLOv5 colon polyp detection model to get the colon polyp bounding box matrix in the image , which includes Bounding box vector of a colon polyp , these four values ​​are in Among them is the offset of the center of the bounding box relative to the upper left corner of the image, are the ratios of the bounding box width and height relative to the image width and height, respectively.

[0054] Will Input the Student model to get the depth map of the image , using the formula Will The pixels representing the depth value in the image are restored to the point cloud in the world coordinate system. , where the superscript express Three coordinate axes, Indicates a total of points.

[0055] for For each colon polyp in the bounding box vector, calculate the midpoint coordinates of the four sides of the bounding box , then remove exist Coordinates in .calculate of ,in and Respectively exist The maximum and minimum values ​​in the direction, and Respectively exist The maximum and minimum values ​​in the direction, and finally the size of the polyp is expressed as .

[0056] Based on the concept of the above method steps, the present invention also provides a colon polyp size detection system under colonoscopy, comprising:

[0057] In the first training module, a YOLOv5 target detection model is trained using a colon polyp dataset to obtain a YOLOv5 colon polyp detection model.

[0058] In the second training module, the DepthAnything depth estimation model was trained using the colonoscopy depth estimation dataset, and semi-supervised training was performed using labeled virtual images and unlabeled real images to obtain the Student model.

[0059] The detection module uses the trained YOLOv5 colon polyp detection model and the trained Student model to detect the size of polyps.

[0060] The beneficial effects of the present invention are as follows: The present invention combines the YOLOv5 colon polyp detection model and the DepthAnything depth estimation model to achieve accurate positioning and size measurement of polyps. The method can automatically detect polyps and estimate their actual physical size, improve the accuracy of image recognition, and avoid measurement errors caused by the lack of reference objects in traditional methods. In addition, the technical solution is lightweight and can be deployed instantly without the need for additional equipment support, which simplifies the operation process, reduces costs, and helps to detect and treat diseased tissues that may cause cancer at an early stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific implementation or the prior art description. The drawings constitute part of the specification and are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation of the present invention. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0062] Figure 1 It is the overall flow chart of the method of the present invention;

[0063] Figure 2 is an application flow chart of the method of the present invention;

[0064] Figure 3 Schematic diagram of lens calibration and distortion removal in an embodiment of the present invention;

[0065] Figure 4 Schematic diagram of a clinical true image and a depth image in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application.

[0067] like Figure 1 and Figure 2 As shown, the embodiment of the present application provides a method for detecting the size of colon polyps under colonoscopy, and the specific steps of the method are as follows:

[0068] Step 1: Train the YOLOv5 colon polyp detection model

[0069] Step 1.1: Preparation and preprocessing of colon polyp detection dataset

[0070] This embodiment uses the hyper-kvasir gastrointestinal endoscopy public dataset, which contains 110,079 images and 374 video diagnosis processes, including bounding boxes of 1,000 images of polyps.

[0071] Set the pixel brightness threshold to initially filter out the reflective and dark video frames, and retain the relatively clear parts. Convert the bounding box annotation format to .

[0072] Finally, fill the image with a grayscale value of 128 to fill the square, and then use the bilinear interpolation method to scale it to 640×640 size.

[0073] Randomize the data set The dataset is divided into three parts: training set, validation set and test set. The training set is used for model training, the validation set is used for hyperparameter adjustment, and the test set is used to evaluate model performance. At the same time, image enhancement techniques such as brightness adjustment, rotation, and mapping are applied to the training set to enhance the robustness of the model.

[0074] Step 1.2: YOLOv5 colon polyp detection model training process

[0075] First, load the pre-trained weights provided by YOLOv5 as the initial state of the model and train it using the training set described in step 1.1. The model uses a chessboard grid to represent the feature map. The block size is divided into blocks, each of which is associated with the output of the model vectors, each of which represents a prediction box. Respectively represent the offset of the prediction box relative to the upper left corner of the block in the x and y directions, Represents the length and width of the prediction box, Indicates the probability that the predicted box contains the target, Represents the unique hot encoding of the predicted value of the prediction box category, with a total of There are two categories, polyps and background. Loss function:

[0076]

[0077] in Indicates the output feature map, that is, The feature map is divided into different block sizes. Indicates The number of blocks in a feature map; Used to balance the weights of each output feature map, for For feature maps of size, They are ; Indicates The feature map Block Whether the predicted box is a positive sample (matching a true value), if it is a positive sample, it is 1, otherwise it is 0; is the weight of the corresponding loss function, which is .

[0078] In one embodiment, The expression is as follows:

[0079]

[0080] in Represent the predicted box and the true box respectively; Represents the distance between the center point of the predicted box and the real box, is the diagonal distance between the predicted box and the minimum enclosing rectangle of the real box; w G ,h G Represents the width and height of the prediction box, w P ,h P Represent the width and height of the real frame respectively.

[0081] In one embodiment, The expression is as follows:

[0082]

[0083] in represents the binary cross entropy loss function, is the probability of prediction box prediction, is the predicted box and the corresponding real box value, Represents the weight of the positive sample.

[0084] In one embodiment, The expression is as follows:

[0085]

[0086] in is the cross entropy loss function, is the category predicted by the prediction box, For the real category, Represents the weight of the true category, which is .

[0087] In each epoch, the model predicts the images in the training set, calculates the loss value according to the loss function, and then back-propagates to update the weights. After the training is completed, the best model weights are saved, and the model performance is evaluated on the test set, including indicators such as accuracy, recall, and mAP scores. The final results are shown in Table 1.

[0088] Table 1 Test results of YOLOv5 colon polyp detection model

[0089]

[0090] Step 2: Train a DepthAnything-based depth estimation model

[0091] Step 2.1: Preparation and preprocessing of colonoscopy depth estimation dataset

[0092] The colonoscopy depth estimation dataset includes a labeled virtual image dataset and an unlabeled real image dataset.

[0093] The virtual image dataset uses C3VD (Colonoscopy 3D Video Dataset). The C3VD dataset is a three-dimensional video dataset designed specifically for colonoscopy, which aims to provide high-precision registration data for the application of computer vision technology in the medical field. The dataset integrates 22 colonoscopy videos processed by multimodal 2D-3D registration technology. The videos were shot in an in vitro model, including 2,363 sigmoid colon image frames, 2,736 transverse colon image frames, 148 descending colon image frames, and 4,768 ascending colon image frames, a total of 10,015 frames of images, each with paired real depth, surface normals, optical flow, occlusion, six degrees of freedom (HDoF) camera pose information and coverage. The dataset also provides a 3D model file of the colon and open source code for registration and rendering. In addition, the dataset also includes 4 colonoscopy videos and corresponding camera pose annotations. The C3VD dataset also provides the lens intrinsic parameter matrix and lens distortion coefficient .

[0094] The real image dataset is a series of 30 unlabeled colonoscopy videos obtained from a hospital. For the real clinical dataset, manual calibration is required to obtain the lens intrinsic parameter matrix of the endoscope. and lens distortion coefficient The calibration process is as follows: first, prepare a black and white chessboard with known size and number of corner points, then place the calibration pattern at different positions and angles, and use the endoscope to be calibrated to shoot the image containing the calibration pattern from multiple perspectives. Finally, the OpenCV library is used to solve the endoscope lens internal parameter matrix And the lens distortion coefficient , the chessboard and the lens before and after distortion removal are as follows Figure 3 shown.

[0095] The data preprocessing part includes: filtering out the blurred frames of the video, retaining the relatively high-quality parts, splitting them into pictures, and using the lens intrinsic parameter matrix and lens distortion coefficient to perform fisheye lens correction on the pictures. To zoom to The depth truth value of the virtual image is distributed in mm, divided by To zoom to Finally, the true depth values ​​of the dedistorted input image and the virtual image are uniformly scaled to Pixel.

[0096] Step 2.2: Train the DepthAnything depth estimation model on a dataset of labeled virtual images

[0097] The 15 videos in the C3VD dataset are classified as training sets, and the 7 videos are classified as validation sets. The DepthAnything depth estimation model is trained on the training set and is called the Teacher model. Model input image The dimension is , model output The dimension is , loss function:

[0098]

[0099] The average of the final Teacher model on the C3VD validation set The loss is Millimeters.

[0100] Step 2.3: Train the DepthAnything lighting adjustment model

[0101] This embodiment uses a traditional pinhole camera model. The camera is located at the origin of the world coordinate system and looks in the negative direction of the Z axis, the X axis is horizontal to the left, and the Y axis is vertically pointing to the ground. The camera intrinsic parameter matrix is ​​defined as , define the point in the world coordinate system as and through Mapping to pixel coordinates .

[0102] First, the parameters obtained by lens calibration are used to remove the distortion of the input frame, and the processed image is scaled to size, and then use the trained Teacher model to predict its depth map , and then use the following formula to convert The pixels representing the depth values ​​are restored to the point cloud in the world coordinate system:

[0103]

[0104] Among them, the pixel coordinates are Pixels, represents its world coordinates, Indicates its distance from the camera (depth value).

[0105] After obtaining the point cloud, the per-pixel shading (PPS) of the image is calculated according to the following formula:

[0106]

[0107] in The coordinates of the co-location light of the endoscope are , The lighting direction is the negative direction of the Z axis. The angular attenuation coefficient of the illumination is .

[0108] Use all real images to train the DepthAnything illumination adjustment model, grayscale image of the real image With its depth map The splicing result in the channel dimension is ,Will The side length of bilinear interpolation is downsampled by four times to As the model input, the model predicts a correction to the grayscale image of the real image , and then upsample four times by bilinear interpolation to get , loss function:

[0109]

[0110] After training is completed, the trained DepthAnything illumination adjustment model is used to predict the grayscale image of the real image. , get and save the result For subsequent model training.

[0111] The final average on the training set The brightness loss is .

[0112] Step 2.4: Train the DepthAnything depth estimation model using both virtual and real image datasets

[0113] The DepthAnything depth estimation model is trained using both the training set of the C3VD dataset and the real image dataset. This model is called the Student model. Model input image The dimension is , model output The dimension is , when the input is a virtual image, the loss function is:

[0114]

[0115] When the input is a real image, the loss function is:

[0116]

[0117] in yes enter The formula is obtained.

[0118] The average of the final Student model on the C3VD validation set The loss is Millimeters.

[0119] Step 3: Use the trained YOLOv5 colon polyp detection model and Student model to detect the size of polyps, see Figure 4 .

[0120] The present invention jointly uses the YOLOv5 colon polyp detection model trained in step 1 and the Student model trained in step 2 to detect the size of polyps.

[0121] For the input image frame, first use the endoscope lens intrinsic parameter matrix obtained by step 2.1 lens calibration And the lens distortion coefficient Perform lens correction and divide by To zoom to In the range, bilinear interpolation is used to scale to Pixel, denoted as .

[0122] Will Input the YOLOv5 colon polyp detection model to get the colon polyp bounding box matrix in the image , which includes Bounding box vector of a colon polyp , these four values ​​are in Among them is the offset of the center of the bounding box relative to the upper left corner of the image, are the ratios of the bounding box width and height relative to the image width and height, respectively.

[0123] Will Input the Student model to get the depth map of the image , using the formula Will The pixels representing the depth value in the image are restored to the point cloud in the world coordinate system. ,in express Three coordinate axes, Indicates a total of points.

[0124] for For each colon polyp in the bounding box vector, calculate the midpoint coordinates of the four sides of the bounding box , then remove exist Coordinates in .calculate of ,in and Respectively exist The maximum and minimum values ​​in the direction, and Respectively exist The maximum and minimum values ​​in the direction, and finally the size of the polyp is expressed as .

[0125] The present application also provides a colon polyp size detection system under colonoscopy, comprising:

[0126] In the first training module, a YOLOv5 target detection model is trained using a colon polyp dataset to obtain a YOLOv5 colon polyp detection model.

[0127] In the second training module, the DepthAnything depth estimation model was trained using the colonoscopy depth estimation dataset, and semi-supervised training was performed using labeled virtual images and unlabeled real images to obtain the Student model.

[0128] The detection module uses the trained YOLOv5 colon polyp detection model and the trained Student model to detect the size of polyps.

[0129] Finally, it should be noted that the above is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto, and any modifications, equivalent substitutions, improvements, etc. made within the ideas and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for detecting the size of colon polyps under colonoscopy, characterized in that The method comprises the following steps: Step 1: Use the colon polyp dataset to train the YOLOv5 target detection model to obtain the YOLOv5 colon polyp detection model; Step 2: Use the colonoscopy depth estimation dataset to train the DepthAnything depth estimation model, and use both labeled virtual images and unlabeled real images for semi-supervised training to obtain the Student model; Step 3: Combine the YOLOv5 colon polyp detection model trained in step 1 and the Student model trained in step 2 to detect the size of polyps.

2. The method for detecting the size of colon polyps under colonoscopy according to claim 1, characterized in that: The step 1 specifically includes: Step 1.1: Prepare and preprocess the colon polyp detection dataset; The colon polyp detection dataset consists of a large number of colonoscopy video frames, which contain a large number of various types of colon polyps and are professionally annotated to mark the presence and location of polyps; Normalize the size of the original video frames to meet the requirements of the YOLOv5 target detection model; The data set is randomly divided into three parts: training set, validation set and test set; Step 1.2: YOLOv5 colon polyp detection model training; Load the pre-trained weights provided by YOLOv5 as the initial state of the model, and train using the training set described in step 1.

1. In each epoch, the model predicts the images in the training set, calculates the loss value based on the loss function, and then back-propagates to update the weights. In addition, the performance on the validation set is used to adjust the learning rate and other hyperparameters. After training is complete, the best model weights are saved and the model performance, including precision, recall, and F1 score, is evaluated on the test set.

3. A method for detecting the size of colon polyps under colonoscopy according to claim 2, characterized in that: The annotation format is ,in is the offset of the center of the bounding box relative to the upper left corner of the image, are the ratios of the bounding box width and height relative to the image width and height, respectively.

4. A method for detecting the size of colon polyps under colonoscopy according to claim 2 or 3, characterized in that: It also includes image augmentation by applying brightness adjustment, rotation, and mapping to the training set to enhance the robustness of the model.

5. The method for detecting the size of colon polyps under colonoscopy according to claim 2, characterized in that: During the training process of the YOLOv5 colon polyp detection model, the model divides the feature map into three blocks using a chessboard grid. Each block is associated with three vectors in the model output, and each vector represents a prediction box.

6. The method for detecting the size of colon polyps under colonoscopy according to claim 1, characterized in that: The step 2 specifically includes: Step 2.1: Preparation and preprocessing of colonoscopy depth estimation dataset The colonoscopy depth estimation dataset includes a labeled virtual image dataset and an unlabeled real image dataset; Preprocessing includes: if the data is a video, filter out the blurred frames of the video, split it into pictures, and use the lens intrinsic parameter matrix and lens distortion coefficient to correct the pictures with fisheye lens; Step 2.2: Train the first DepthAnything depth estimation model on the labeled virtual image dataset to obtain the Teacher model; Step 2.3: Train the DepthAnything illumination adjustment model to predict the grayscale image of the real image; Step 2.4: Use the virtual image and real image datasets to train the second DepthAnything depth estimation model to obtain the Student model.

7. A method for detecting the size of colon polyps under colonoscopy according to claim 6, characterized in that: For the real image dataset, the endoscope lens intrinsic parameter matrix and lens distortion coefficient are obtained through calibration.

8. The method for detecting the size of colon polyps under colonoscopy according to claim 1, characterized in that: The step 3 is specifically: The preprocessed image is input into the YOLOv5 colon polyp detection model to obtain the colon polyp bounding box matrix in the image ; Input the preprocessed image into the Student model to obtain the depth map of the image ; The depth map The pixels representing the depth values ​​are restored to the point cloud in the world coordinate system; For the colon polyp bounding box matrix The bounding box vector of each colon polyp in the point cloud is calculated, and the midpoint coordinates of the four sides of the bounding box are taken. ; Calculate coordinates of ,in and Respectively represent coordinates exist The maximum and minimum values ​​in the direction, and Respectively exist Maximum and minimum values ​​in direction; Express the size of the polyp as .

9. A colon polyp size detection system under colonoscopy, characterized in that: include: In the first training module, a YOLOv5 target detection model is trained using a colon polyp dataset to obtain a YOLOv5 colon polyp detection model. In the second training module, the DepthAnything depth estimation model was trained using the colonoscopy depth estimation dataset, and semi-supervised training was performed using labeled virtual images and unlabeled real images to obtain the Student model. The detection module uses the trained YOLOv5 colon polyp detection model and the trained Student model to detect the size of polyps.

Images