Composite treatment method and treatment system for colon polyp

Abstract

The invention discloses a colon polyp composite processing method and processing system, and the method can carry out the classification prediction of an output image based on an image classification model, and extracts a class label with the highest prediction probability and a confidence score of the class label. A label and a confidence score are superposed on an original frame, polyps with different properties are converted into different colors, polyp time points are marked to form a colored time progress bar, and the progress bar and an original colonoscope video are combined together to form a brand new video as an output result. A quick review tool is provided for subsequent colonoscope minimally invasive surgery; the device can help an endoscopic physician to accurately master the position and form of the polyp and quickly position the polyp found previously, so that the operation time is shortened, the risk of postoperative complications is reduced, and the diagnosis and treatment efficiency is improved.

Description

Technical Field

[0001] The invention relates to the field of image processing and computer technology, and in particular to a composite processing method and system for colon polyps. Background Art

[0002] In endoscopic minimally invasive surgery, the successful detection of polyps is the key to the success of endoscopic minimally invasive surgery. With the development of endoscopic technology, minimally invasive surgeries such as endoscopic mucosal resection (EMR), endoscopic submucosal dissection (ESD), and cold resection have been widely used in medical centers around the world. Early removal of polyps with the risk of canceration is crucial to the prevention of colorectal cancer. However, during endoscopic minimally invasive surgery, rapid positioning of polyps found preoperatively remains a common problem that troubles endoscopists. In some cases, the inability to accurately locate polyps may even lead to surgical failure. In addition, it is not enough to remove the same number of polyps as reported in the preoperative report during surgery, because multiple polyps are common and it is not uncommon to miss the diagnosis during preoperative examination. Therefore, endoscopists should carefully check the preoperative report and thoroughly explore the intestine to ensure that all high-risk polyps have been removed.

[0003] At present, in order to solve the problem of difficulty in finding polyps during minimally invasive endoscopic surgery, the following methods are currently used clinically: India ink and indocyanine green are commonly used for endoscopic tattooing, but they have certain limitations. India ink may cause local ulcers, inflammation, and even localized peritonitis. Indocyanine green requires infrared equipment for detection and is only suitable for open surgery. In addition, these dye marks usually disappear within 2 days, which poses a timeliness problem.

[0004] Carbon nanoparticles, as an innovative labeling method, have shown good labeling effects for up to 12 months and reduced potential adverse reactions, but the material has not yet been widely used or included in medical insurance, and its use is still limited; Endoscopic placement of metal clips can mark the location of polyps and overcome drug-related adverse reactions, but it significantly increases medical expenses, and there is uncertainty about the time when the metal clips fall off, and premature shedding may lead to ineffective marking. In addition, metal clips may cause serious adverse events in rare cases, such as clip embedding in tissues, bleeding, and perforation; Therefore, there is an urgent need for a composite treatment method and system for colon polyps to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a composite treatment method and system for colon polyps, which is used to solve the problem that the previously discovered target polyps cannot be quickly located after endoscopic minimally invasive surgery; thereby shortening the minimally invasive surgery time and reducing the risk of postoperative complications.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solution: a composite treatment method for colon polyps, comprising: Based on the image classification model, the output image is classified and predicted, and the category label with the highest prediction probability and its confidence score are extracted; Overlay the labels and confidence scores onto the original frame, and Polyps of different properties are converted into different colors, and the time points of the polyps are marked to form a colorful time progress bar, which is combined with the original colonoscopy video to form a brand new video as the output result.

[0007] Preferably, a colonoscopy image dataset is constructed, the database includes normal intestinal images, adenomatous polyp images and serrated lesion images, and the dataset is divided into a training set, a test set and a validation set; based on the dataset, an image classification model capable of identifying polyps of different properties is trained and screened.

[0008] Preferably, the images in the training set are processed as follows: Perform random image resizing and cropping to preset pixels; Perform random horizontal flipping and color jittering by randomly changing the brightness, contrast, saturation, and hue of the image; and Add noise randomly to the image; The images in the training set are converted to PyTorch Tensor format and normalized to the range [0, 1].

[0009] Preferably, the images in the verification set are processed as follows: Adjust the short side of the image and then crop the center to the preset pixel size; The images in the validation set are converted to PyTorch Tensor format and normalized to the range [0, 1].

[0010] Preferably, based on transfer learning, several CNN pre-trained models with different architectures are trained, and a comprehensive performance comparison is performed on the validation set and the test set to select the model with the best performance as the image classification model.

[0011] Preferably, two fully connected layers with ReLU activation function are added to each pre-trained model, and an output layer with Softmax activation function is added, and the number of nodes in the output layer is set to N, where N represents the number of polyp types.

[0012] Preferably, the training uses the cross entropy loss function and the Adam optimizer, and a total of 30 training cycles are set; and an early stopping strategy is adopted, that is, if the performance of the validation set does not improve in 8 consecutive training cycles, the training is automatically terminated; and the learning rate is halved every 5 cycles.

[0013] The present invention also discloses a composite treatment system for colon polyps, comprising: Image classification models to detect and identify polyps and distinguish their properties; The positioning model is used to mark the appearance time of polyps of different natures in different colors in the complete video; a colored time progress bar is formed, and the marked progress bar is combined with the original colonoscopy video as the final output result.

[0014] Preferably, the image classification model is an EfficientNetV2 model.

[0015] Beneficial effects: The processing method and system of the present invention can identify polyps of different natures in the video during the first colonoscopy, including adenomatous polyps, serrated lesions and normal intestines, and mark the appearance time of polyps of different natures in the complete video with different colors; combine the marked progress bar with the original colonoscopy video and use it as the final output result; in the second colonoscopy (endoscopic minimally invasive surgery for polypectomy), the physician can review the colonoscopy video with time stamps to quickly understand the specific location and morphology of the polyps in the entire intestine. On the one hand, it can help physicians locate the designated polyps found during the first colonoscopy more quickly during minimally invasive surgery, thereby shortening the operation time and reducing the risk of postoperative complications, reducing the adverse reactions and additional costs caused by the use of dye markers and titanium clip markers, and on the other hand, it can provide more reference information for postoperative review and reduce the risk of poor prognosis caused by local recurrence. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0017] In the attached picture: Figure 1 It is a diagram of the processing method of the present invention; Figure 2 is a flow chart of the processing method of the present invention; Figure 3 is a colonoscopy image in the data set of the present invention; Figure 4 It is a trend diagram of the performance indicators of different models of the present invention changing with the number of training steps; Figure 5This is a graph showing the prediction results of EfficientNetV2 for different types of images in the test set; Figure 6 It is a prediction performance graph of EfficientNetV2 of the present invention on an external test set; Figure 7 is a Grad-CAM visualization of the decision-making process of the present invention; Figure 8 It is a SHAP interpretability analysis diagram of polyp images with different properties of the present invention; Fig. 9 It is a visualization diagram of the output results of Case 1 and Case 2 of the present invention. DETAILED DESCRIPTION

[0018] The embodiments of the present invention are described below in conjunction with the drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention. The embodiments of the present application are described below in conjunction with the drawings.

[0019] Example 1: A composite treatment method for colon polyps, which can detect polyps in the first colonoscopy, determine their nature, and mark the time of appearance of polyps in the complete colonoscopy in different colors in the progress bar, providing a quick review tool for subsequent minimally invasive colonoscopy surgery. This function can help endoscopists accurately grasp the location and morphology of polyps, quickly locate previously discovered polyps, thereby shortening the operation time, reducing the risk of postoperative complications, and improving the efficiency of diagnosis and treatment. The structure of the treatment method is as follows Figure 1 The generated polyp localization results were compared with the annotations provided by two senior endoscopists with more than 10 years of endoscopic experience to evaluate the performance; a detailed overview of the study process is shown in Figure 2 As shown; the processing method specifically includes: Collect a large number of colonoscopic images, including normal intestine, adenomatous polyps, and serrated lesions, to construct a colonoscopic image dataset, and divide the dataset into training set, test set, and validation set; In a specific case, four datasets were used, covering the period from January 2018 to August 2024. Dataset 1 and Dataset 2 are from Changshu Hospital Affiliated to Soochow University and Changshu Traditional Chinese Medicine Hospital, respectively, containing 3041 intestinal polyp images for model training and verification; Dataset 3 is provided by Changshu Xinzhuang People's Hospital and contains 567 images, which is used as the image test set of the model. Dataset 4 also comes from Changshu Xinzhuang People's Hospital and includes 10 colonoscopy videos, which are used as the video test set of the model; the test set is only used to evaluate the performance of the model and is not involved in training or parameter adjustment to ensure its independence; colonoscopy uses endoscope systems of multiple brands, with a total of 13 electronic endoscopes, including SonoScape HD-550, OLYMPUS CV-V1, PentaxEPK-i7000, ELUXEO 7000, etc.; images include adenomatous polyps, serrated lesion images, and normal intestines (reference Figure 3 ); In order to enhance the generalization ability and robustness of the model, the selected endoscopic images cover intestinal conditions of different cleanliness; Preprocess the images in the dataset: To ensure that the model has strong generalization capabilities, the image data was comprehensively preprocessed and enhanced. The enhancement operation was performed in real time during the training process, and no new image files were generated, ensuring that the model could see slightly different versions of the images each time it was trained. For the training set, random image resizing and cropping to 224x224 pixels were first performed. To increase the diversity of the dataset, random horizontal flipping and color jittering were further performed to make the model better adaptable to different lighting conditions by randomly changing the brightness, contrast, saturation and hue of the image. To improve the robustness of the model to noise, Gaussian noise was also added to randomly add noise to the image to simulate interference in actual applications. The images were then converted from PILImage or numpy.ndarray format to PyTorch Tensor and normalized to the range [0, 1]. The RGB channels were normalized using the mean [0.485, 0.456, 0.406] and the standard deviation [0.229, 0.224, 0.225]; the validation set processing strategy is slightly different. First, the short side of the image is resized to 256 pixels, and then the center is cropped to 224x224 pixels. The subsequent transformation and normalization steps are the same as the training set, and the corresponding red, green and blue (RGB) channel normalization parameters are also used. All preprocessing and enhancement steps are completed using the torchvision library of PyTorch.

[0020] Based on the processed data set, an image classification model capable of identifying polyps of different properties is trained and selected, specifically, based on transfer learning, several CNN pre-trained models with different architectures are trained, and a comprehensive performance comparison is performed on the validation set and the test set to select the model with the best performance as the image classification model; Two fully connected layers with ReLU activation function were added to each pre-trained model, and an output layer with Softmax activation function was added. The number of nodes in the output layer was set to N, where N represents the number of polyp types, for example, N is 3, to meet the requirements of the classification task. The model training used the cross entropy loss function and Adam optimizer, and a total of 30 training cycles (epochs) were set. To prevent overfitting, an early stopping strategy was adopted, that is, if the performance of the validation set did not improve in 8 consecutive epochs, the training was automatically terminated. In addition, the learning rate was halved every 5 cycles. All operations were completed under the PyTorch framework.

[0021] Among them, the CNN pre-trained models to be trained include EfficientNetV2, MobileNetV3-Large, DenseNet121, VGG19 and ResNet50.

[0022] A specific case is screened based on the above CNN pre-trained model to be trained, including: Use a computer equipped with an RTX A4000 graphics card (16GB of video memory), 5×E5-2680 v4 CPUs, and 350GB of hard disk space. Use Keras to build and train deep learning models, and use OpenCV to process image data. Use Pandas, NumPy, Matplotlib, and Plotly for data organization, analysis, and visualization. Use PyTorch for model optimization, and rely on H5py for model saving and loading. Use the Weights & Biases (wandb) tool to track the complete training process of the model; Multiple indicators are used to comprehensively evaluate the performance of the AI ​​model, including sensitivity, specificity, precision, accuracy, F1 score, average precision (AP), area under the receiver operating characteristic curve (AUC) and weighted average. The calculation formula is as follows: Sensitivity: ; Specificity: ; Accuracy: ; Accuracy ; F1 score ; Weighted Average: ; Average Precision (AP): ; Area under the receiver operating characteristic curve (AUC): ; Among them, TP represents the number of samples correctly predicted as positive, TN represents the number of samples correctly predicted as negative, FP represents the number of samples incorrectly predicted as positive, and FN represents the number of samples incorrectly predicted as negative. i It is The performance index values ​​of each category are It is The weight of each category; In this case, a total of 3608 images were included, covering three categories: serrated lesion, adenomatous polyp, and normal. Among them, 3041 images were used for model development and verification, and 567 images were independently collected for testing, as shown in Table 1:

[0023] Table 1 Distribution of images in the training set, validation set, and test set class Train set Validation Set Test Set Serrated lesion 497 124 123 adenomatous polyp 852 212 219 normal 1085 271 225 The five different neural network models mentioned above were trained using the same dataset, and the complete training process of the model was tracked using wandb. As the number of training steps increased, the model loss gradually decreased and tended to be stable, indicating that the model was converging towards the optimization direction ( Figure 4 A in (a); Figure 4 B in Figure 4 C in Figure 4 D in the figure shows the trends of the accuracy, precision and F1 scores of different models during training as the number of training steps increases; as the number of training steps increases, these performance indicators initially rise slowly and fluctuate greatly, then gradually stabilize, and remain stable after reaching a high level; compared with the other four models, EfficientNetV2 achieved the best accuracy (93.25%), precision (91.73%), sensitivity (91.60%) and f1-score (91.66%) on the validation set, so it was selected as the best model, see Table 2 for details: Table 2 Performance comparison of different DL models on the validation set (%) Model accuracy precision sensitivity f1-score VGG19 87.31 84.28 84.14 84.20 DenseNet121 90.28 88.06 87.91 87.96 EfficientNetV2 93.25* 91.73* 91.60* 91.66* MobilenetV3 90.94 88.97 88.30 88.61 Resnet50 80.40 76.83 75.11 75.63 Note: * represents the best; Table 3 evaluates the performance of the best performing EfficientNetV2 model on 567 test images. The table shows the recognition accuracy, sensitivity, specificity, F1 score, accuracy, average precision (AP), and AUC of the model for the three categories. In addition, macro avg and weighted avg are used as summary statistics. Figure 5 As shown in the figure, the t-SNE method is used to reduce the dimension and visualize the semantic features extracted by the model. The results show that the features of adenomatouspolyp and serrated lesion are partially overlapped, which intuitively shows the reason why the model has a small amount of misclassification.

[0024] Figure 6 Two key evaluation curves of the prediction performance of the EfficientNetV2 model on the test set are shown: the receiver operating characteristic (ROC) curve and the precision-recall (PR) curve. Figure 6 In A, the curves for all categories are close to the upper left corner of the graph, indicating that the model performs well on these categories. Figure 6 The PR curve in B shows that the closer the category curve is to the upper right corner, the better the performance of the model. Table 3 Performance evaluation results of the EfficientNetV2 model on the test set category precision sensitivity specificity f1-score accuracy AP AUC adenomatous polyp 0.892 0.904 0.931 0.898 0.904 0.975 0.983 normal 0.918 0.893 0.947 0.905 0.893 0.975 0.983 Serrated lesion 0.897 0.919 0.971 0.908 0.919 0.968 0.986 macro avg 0.902 0.905 0.95 0.904 0.905 0.973 0.984 weighted avg 0.903 0.903 0.946 0.903 0.903 0.974 0.983 Note: The weighted average indicator takes into account the sample size of each category, giving higher weights to categories with larger sample sizes.

[0025] Figure 7 The visualization of Grad-CAM technology in the AI ​​model decision-making process is shown; Column A is the original image; Column B shows the pixel activation heat map generated based on the EfficientNetV2 model, highlighting the key areas of model decision; Column C overlays the activation heat map with the original image, and warm colors (such as red and yellow) indicate the key lesion areas identified by the model; In order to deeply analyze the decision-making mechanism of the EfficientNetV2 model in the diagnosis of polyp properties, the SHAP library was used to explain the model's diagnosis; each feature was assigned an influence score, where red represents a positive contribution to the diagnosis result and blue represents a negative contribution; when the red area is more than the blue area, the image is more likely to be diagnosed as this category; based on this principle, Figure 8 The A in the genotype is diagnosed as adenomatous polyp. Figure 8 B in the case was diagnosed as serrated lesion.

[0026] Through the above operations, the best performing model is selected as the final classifier, which is then deployed on the GPU device. The following operations are performed for each frame of the input video: Convert the image from BGR format to RGB format, After preprocessing, the pretrained model is input for classification prediction; the category label with the highest prediction probability and its confidence score are extracted; Using the PIL library, labels and confidence scores are superimposed on the original frames. In order to intuitively present the classification results of each frame, a dynamically updated progress bar is created, with different colors representing different categories and the accumulated video time displayed to help observers track the prediction trend. All processed frames, including those with classification results and progress bars superimposed, are temporarily stored; After all frames are processed, the mmcv library is used to combine these frames into a new video file, which contains polyp position markers to facilitate endoscopists to quickly locate them during the operation. Example

[0027] A composite treatment system for colon polyps, comprising: An image classification model is used to detect and identify polyps and distinguish the properties of polyps. The image classification model is selected by the above method. For example, the image classification model is an EfficientNetV2 model. The positioning model is used to mark the appearance time of polyps of different natures in different colors in the complete video; a colored time progress bar is formed, and the marked progress bar is combined with the original colonoscopy video as the final output result.

[0028] Based on the above system, the practical process is as follows: During the first colonoscopy, the system's polyp recognition module is used to identify polyps of different natures in the video, including adenomatous polyps, serrated lesions and normal intestines; the positioning model marks the appearance of polyps of different natures in different colors in the complete video; the annotated progress bar is combined with the original colonoscopy video and used as the final output result.

[0029] During a second colonoscopy (a minimally invasive endoscopic procedure for polypectomy), physicians can quickly review time-stamped colonoscopy videos to understand the specific location and morphology of polyps throughout the intestine. This feature helps physicians more quickly locate specific polyps found during the first colonoscopy during minimally invasive surgery, thereby shortening the operation time and reducing the risk of postoperative complications.

[0030] The practical applications based on the above processing method or processing system are as follows: Fig. 9The output results based on the above processing methods or processing systems are shown; in these videos, the prediction is performed at a rate of 5 frames per second, that is, 5 predictions are made per second; the prediction progress bar with category visualization is displayed synchronously in real time below the colonoscopy video, and the cumulative time (in minutes) is displayed in real time on the right side of the progress bar; in order to show the prediction accuracy of the model, a visual progress bar is created for these two cases ( Fig. 9 ), where the top row represents the ground truth and the bottom row shows a visualization progress bar with polyp time markers.

[0031] Based on the above, a new video reference was obtained in a specific case. Fig. 9 As shown, users can scan Fig. 9 The QR code in the video is as follows: Case 1: A serrated lesion with a diameter of about 5 mm was found in the descending colon, and the endoscopist directly removed it with a biopsy forceps; the total colonoscopy time was 8 minutes and 14 seconds. The patient's bowel preparation was poor, with a Boston score of 4 points; the location of the polyp was accurately recorded through the processing system; the physician reviewed the video with polyp location marks and located the polyp through a visual progress bar in just 5 seconds; compared with traditional paper reports, dynamic videos with contextual relationships allow endoscopists to have a more comprehensive understanding of the location and morphology of polyps, which helps to observe the local wound surface during the next colonoscopy review and prevent the residual and recurrence of local lesions.

[0032] Case 2: Three adenomatous polyps were found in the colon. The endoscopist used biopsy forceps to biopsy them separately and sent them for pathological examination. The total colonoscopy time was 9 minutes and 47 seconds. The patient had a good intestinal preparation and a Boston score of 7 points. The patient was admitted to the gastroenterology department for EMR polypectomy one week later. The location of the three polyps was accurately recorded through the processing system. The doctor quickly understood the location and morphology of the polyps by reviewing the video with polyp positioning marks, which helped the surgeon to find the polyps in the previous report more quickly during the operation, thereby shortening the operation time.

[0033] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. For ordinary technicians in this technical field, after knowing the contents recorded in the present invention, they can make several equivalent changes and substitutions without departing from the principle of the present invention. These equivalent changes and substitutions should also be regarded as belonging to the protection scope of the present invention.

Claims

1. A composite treatment method for colon polyps, characterized in that: include: Based on the image classification model, the output image is classified and predicted, and the category label with the highest prediction probability and its confidence score are extracted; Overlay the labels and confidence scores onto the original frame, and Polyps of different properties are converted into different colors, and the time points of the polyps are marked to form a colorful time progress bar, which is combined with the original colonoscopy video to form a brand new video as the output result.

2. A composite treatment method for colon polyps according to claim 1, characterized in that: A colonoscopy image dataset was constructed, which included normal intestinal images, adenomatous polyp images, and serrated lesion images. The dataset was divided into training set, test set, and validation set. An image classification model that could identify polyps of different natures was trained and screened based on the dataset.

3. A composite treatment method for colon polyps according to claim 2, characterized in that: The images in the training set are processed as follows: Perform random image resizing and cropping to preset pixels; Perform random horizontal flipping and color jittering by randomly changing the brightness, contrast, saturation, and hue of the image; as well as Add noise randomly to the image; The images in the training set are converted to PyTorch Tensor format and normalized to the range [0, 1].

4. A composite treatment method for colon polyps according to claim 2, characterized in that: The images in the validation set are processed as follows: Adjust the short side of the image and then crop the center to the preset pixel size; The images in the validation set are converted to PyTorch Tensor format and normalized to the range [0, 1].

5. The combined treatment method for colon polyps according to claim 2, characterized in that: Based on transfer learning, several CNN pre-trained models with different architectures are trained, and through comprehensive performance comparison on the validation set and test set, the model with the best performance is selected as the image classification model.

6. A composite treatment method for colon polyps according to claim 5, characterized in that: Two fully connected layers with ReLU activation function were added to each pre-trained model, and an output layer with Softmax activation function was added. The number of nodes in the output layer was set to N, where N represents the number of polyp types.

7. A composite treatment method for colon polyps according to claim 5 or 6, characterized in that: The training uses the cross entropy loss function and Adam optimizer, with a total of 30 training cycles. The early stopping strategy is adopted, that is, if the performance of the validation set does not improve in 8 consecutive training cycles, the training is automatically terminated. The learning rate is halved every 5 cycles.

8. A composite treatment system for colon polyps, characterized in that: include: Image classification models to detect and identify polyps and distinguish their properties; A positioning model is used to mark the appearance of polyps of different natures in different colors in the complete video; A colored time progress bar is formed, and the annotated progress bar is combined with the original colonoscopy video as the final output result.

9. A complex treatment system for colon polyps according to claim 8, characterized in that: The image classification model is the EfficientNetV2 model.

Images