Polyp and surgical instrument segmentation and real-time early warning method and system based on digestive endoscopy video

Abstract

The invention discloses a polyp and surgical instrument segmentation and real-time early warning method and system based on a digestive endoscope video, and belongs to the technical field of computer-aided endoscope image analysis. The problems that an existing method is poor in segmentation performance and the safety of interactive operation of the surgical instrument and the polyp is difficult to accurately evaluate are solved. According to the method, space and time context information is integrated through the built collaborative learning network model, and collaborative precise segmentation of the polyp and the surgical instrument in the endoscopic surgery video is achieved. The polyp area, the polyp area change rate, the surgical instrument speed, the distance between the polyp and the surgical instrument and the overlapping area and the overlapping percentage of the polyp and the surgical instrument are quantitatively analyzed on the basis of the segmentation results, the surgical risk level is evaluated in real time in combination with a preset risk threshold value, the safety of interaction between the surgical instrument and the polyp is guaranteed, and the surgical efficiency is improved. Therefore, the safety and the intelligent level of the digestive endoscopy minimally invasive surgery are improved.

Description

Technical Field

[0001] This invention relates to the field of computer-aided endoscopic image analysis technology, specifically to a method and system for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video. Background Technology

[0002] Colorectal cancer (CRC), the third most common malignant tumor worldwide, poses a serious threat to human health. Colonoscopic polypectomy is a key intervention for preventing CRC and reducing its mortality. However, the complexity of this procedure and the stringent requirements for the high skill level of endoscopists make accurate, real-time analysis of surgical instruments and abnormal tissues within complex and dynamic endoscopic video sequences a significant technical challenge. Although existing computer-aided systems have been applied to improve surgical safety and accuracy, providing support such as instrument positioning, tissue identification, and safety alerts through video analysis, many difficulties remain in practical operation. Specifically, colonoscopic polypectomy involves a dynamic, narrow, and physiologically variable visual environment, including peristalsis, mucosal folds, and variable intraluminal contents. Artifacts such as specular reflection, occlusion, and suboptimal illumination further exacerbate the difficulty of distinguishing polyps from surgical instruments during the procedure. Existing methods struggle to effectively handle these variations and accurately model the dynamic interactions between abnormal tissues and surgical instruments, resulting in insufficient accuracy and temporal consistency in the simultaneous real-time segmentation of polyps and instruments, and computational efficiency often fails to meet real-time requirements. Furthermore, the presence of abnormal tissues and surgical instruments across video frames exhibits heterogeneity, manifesting as image distortion, occlusion, and rapid changes in appearance. This can distract the model and affect segmentation performance. Current research on digestive endoscopy robots largely focuses on basic technologies, with relatively little exploration of higher-level surgical situational awareness. This results in a lack of operational feedback and tactile information for both the surgeon and the endoscopy robot during surgery, making it difficult to accurately assess the safety of operations such as the interaction between surgical instruments and polyps. This can easily lead to complications such as soft tissue tears and massive bleeding, significantly increasing the cognitive burden on surgeons and reducing the overall safety of digestive endoscopy.

[0003] In summary, to address the issues of poor segmentation performance and difficulty in accurately assessing the safety of surgical instruments interacting with polyps in existing methods, a new segmentation and assessment method is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to address the problems of poor segmentation performance and difficulty in accurately assessing the safety of the interaction between surgical instruments and polyps in existing methods. Therefore, this invention proposes a method and system for segmenting polyps and surgical instruments and providing real-time early warning based on digestive endoscopy video.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] According to one aspect of the present invention, a method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video is provided, the method specifically including the following steps:

[0007] Step S1: Construct a digestive endoscopy surgery video dataset through actual collection and simulation. After the constructed video dataset is processed by frame segmentation, each image obtained by frame segmentation is manually labeled. Then, the labeled images are divided into three parts: training set, validation set and test set.

[0008] Step S2: Construct a collaborative learning network model. In the spatial domain, the collaborative learning network model includes a wavelet transform feature decomposer, a two-branch wavelet feature extraction module, and a diffuse block-based decoder. In the temporal domain, the collaborative learning network model includes a dynamic attention fusion module and a convolutional exponential time integrator.

[0009] The output of the convolutional exponential time integrator is then passed through the Softmax activation function, and the segmentation result is obtained based on the output of the Softmax activation function.

[0010] The constructed collaborative learning network model is trained, validated, and tested using the training set, validation set, and test set to obtain a well-trained collaborative learning network model.

[0011] Step S3: Use the digestive endoscopy video image to be processed as the input of the trained collaborative learning network model, and output the polyp and instrument segmentation results in the digestive endoscopy video in real time through the trained collaborative learning network model;

[0012] Step S4: Conduct risk assessment and real-time early warning based on the segmentation results of step S3.

[0013] Furthermore, the specific process of step S1 is as follows:

[0014] Step S11: Collect real digestive endoscopy surgery video data. After processing the collected digestive endoscopy surgery video data into frames, manually annotate each frame image. Through manual annotation, spatial segmentation and semantic classification labels are added to the instruments, polyps and background in each frame image.

[0015] Step S12: Create a 3D colon model and surgical instrument model using Maya 3D modeling software, and generate virtual data covering complex interaction scenarios between various surgical instruments and polyps by designing a digestive endoscopy surgery simulation.

[0016] The generated virtual digestive endoscopy video data is processed by frame segmentation, and spatial segmentation and semantic classification labels of instruments, polyps and background in each frame image are obtained synchronously according to the generation environment of the virtual data during frame segmentation.

[0017] Step S13: Divide the dataset consisting of labeled images from real and virtual data into three parts: training set, validation set, and test set.

[0018] Furthermore, the operation process of the wavelet transform feature decomposer is as follows:

[0019] Step C1: Perform wavelet transform on the image to separate the low-frequency and high-frequency information in the image. The specific process is as follows:

[0020] Step C11: Convert the input RGB image to a grayscale image; if the input is already a grayscale image, skip step C11.

[0021] Step C12: Perform a two-dimensional discrete wavelet transform on the obtained grayscale image to decompose the grayscale image into four sub-bands: LL, LH, HL and HH.

[0022] Step C13: Normalize the images of the LL, LH, HL and HH subbands respectively, and scale the pixel values ​​of each subband image to the range [0, 1].

[0023] Step C14: The images of the three sub-bands LH, HL and HH after normalization are added and fused pixel by pixel to obtain a comprehensive high-frequency feature map;

[0024] The image of the normalized LL subband is used as the low-frequency feature map, and the label mask of the high-frequency feature map and the low-frequency feature map is obtained according to the image label.

[0025] Step C2: Adjust the size of the integrated high-frequency feature map, low-frequency feature map, and label mask;

[0026] Step C21: Adjust the dimensions of the integrated high-frequency feature map and low-frequency feature map to the preset dimensions;

[0027] Step C22: Adjust the label mask of the integrated high-frequency feature map and low-frequency feature map to the preset size;

[0028] Step C23: Convert the high-frequency feature map and low-frequency feature map obtained in step C2 into float32 type, and normalize the standard deviation of the converted high-frequency feature map and normalize the mean of the converted low-frequency feature map.

[0029] Convert the label mask of the low-frequency feature map obtained in step C2 to a PyTorch FloatTensor type, and convert the label mask of the high-frequency feature map obtained in step C2 to a PyTorch LongTensor type.

[0030] Furthermore, the dual-branch wavelet feature extraction module includes two parallel coding branches, which are used respectively for the low-frequency feature map and the high-frequency feature map obtained by the wavelet transform feature decomposer; wherein:

[0031] In step C23, the converted low-frequency feature map passes sequentially through the first Double Basic block, the first upsampling layer, the second Double Basic block, the second upsampling layer, the third Double Basic block, the third upsampling layer, the fourth Double Basic block, the fourth upsampling layer, and the fifth Double Basic block within the first branch.

[0032] The high-frequency feature map transformed in step C23 is sequentially passed through the sixth Double Basic block, the fifth upsampling layer, the seventh Double Basic block, the sixth upsampling layer, the eighth Double Basic block, the seventh upsampling layer, the ninth Double Basic block, the eighth upsampling layer, and the tenth Double Basic block in the second branch.

[0033] Furthermore, the working process of the first Double Basic block is as follows:

[0034] Within the first Double Basic block, the input of the first Double Basic block first passes through the first convolutional layer, the first ReLU activation function layer, and the second convolutional layer. The output of the second convolutional layer is then concatenated with the input of the first Double Basic block to obtain the concatenated result a.

[0035] Then, the concatenated result a is passed through the second ReLU activation function layer, the third convolutional layer, the third ReLU activation function layer, and the fourth convolutional layer. The output of the fourth convolutional layer is concatenated with the output of the third ReLU activation function layer to obtain the concatenated result b.

[0036] The concatenated result b is used as the input to the fourth ReLU activation function layer, and the output of the fourth ReLU activation function layer is used as the output of the first Double Basic block.

[0037] Furthermore, the diffusion block-based decoder includes a first diffusion decoding unit, a second diffusion decoding unit, a third diffusion decoding unit, a fourth diffusion decoding unit, and a fifth diffusion decoding unit. The working process of the diffusion block-based decoder is as follows:

[0038] After concatenating the output of the first Double Basic block with the output of the sixth Double Basic block, the concatenation result is used as the input of the first diffusion decoding unit;

[0039] The output of the first diffusion decoding unit, the output of the second Double Basic block, and the output of the seventh Double Basic block are concatenated, and the concatenation result is used as the input of the second diffusion decoding unit.

[0040] The output of the second diffusion decoding unit, the output of the third Double Basic block, and the output of the eighth Double Basic block are concatenated, and the concatenation result is used as the input of the third diffusion decoding unit.

[0041] The output of the third diffusion decoding unit, the output of the fourth Double Basic block, and the output of the ninth Double Basic block are concatenated, and the concatenation result is used as the input of the fifth diffusion decoding unit.

[0042] The output of the fifth diffusion decoding unit, the output of the fifth Double Basic block, and the output of the tenth Double Basic block are concatenated, and the concatenation result is used as the output of the diffusion block-based decoder.

[0043] Furthermore, the first diffusion decoding unit includes a first diffusion block to an eighth diffusion block in sequence. The input of the first diffusion decoding unit is used as the input of the first diffusion block, the output of the first diffusion block is used as the input of the second diffusion block, and so on. The output of the j-th diffusion block is used as the input of the (j+1)-th diffusion block, and the output of the eighth diffusion block is used as the output of the first diffusion decoding unit.

[0044] Each diffusion block includes a Gaussian noise layer, a convolutional layer, a BN layer, and a ReLU activation function layer. Within the diffusion block, the input of the diffusion block passes through the Gaussian noise layer, the convolutional layer, the BN layer, and the ReLU activation function layer in sequence, and the output of the ReLU activation function layer is used as the output of the diffusion block.

[0045] Furthermore, the second diffusion decoding unit sequentially includes an upsampling layer, a first diffusion block to an eighth diffusion block. The input of the second diffusion decoding unit is used as the input of the upsampling layer, the output of the upsampling layer is used as the input of the first diffusion block, the output of the first diffusion block is used as the input of the second diffusion block, and so on. The output of the j-th diffusion block is used as the input of the (j+1)-th diffusion block, and the output of the eighth diffusion block is used as the output of the second diffusion decoding unit.

[0046] Furthermore, the working processes of the third, fourth, and fifth diffusion decoding units are all the same as those of the second diffusion decoding unit.

[0047] Furthermore, the working process of the Gaussian noise addition layer is as follows:

[0048]

[0049] in, Input feature map; It is the first Feature map after adding noise; This indicates the total number of steps to add noise; It is a nonlinear transformation; For hyperparameters; For added noise, obey distributed, This indicates that the mean is 0 and the variance is 0. Gaussian distribution, It is the identity matrix;

[0050]

[0051] in, It is a convolution kernel. This represents the convolution operation. It is a batch of normalization, It is an activation function;

[0052]

[0053] in, Indicates the first Feature map after adding noise; Represents the projection operation of the feature map; This represents the output of the Gaussian noise addition layer;

[0054]

[0055] in, Indicates weight, Indicates bias.

[0056] Furthermore, the working process of the dynamic attention fusion module is as follows:

[0057] The feature map output by the current frame after passing through the diffuse block-based decoder is denoted as... The list of feature maps output by the diffusion block-based decoder for historical frames is denoted as... ;

[0058] The current frame feature map Convert to query vectors: Convert the feature map of each historical frame in the historical frame feature list into a key vector, and convert the feature map of each historical frame in the historical frame feature list into a value vector.

[0059]

[0060] in, Represents the feature map of the current frame The corresponding query vector, Indicates the first The key vector corresponding to each historical frame Indicates the first The value vector corresponding to each historical frame;

[0061]

[0062] in, express The Middle Data from each channel, express The Middle Data from each channel, This represents element-wise multiplication. Indicates the current frame and the first frame. Similarity scores of historical frames;

[0063]

[0064] in, Represents the attention weight matrix;

[0065] The value vectors of historical frames are weighted and summed based on the calculated attention weights to obtain the fused historical features:

[0066]

[0067] in, Indicates the first Attention weights for each historical frame at each spatial location, This indicates that attention weights will be broadcast to On all the passages, This indicates the historical characteristics after the integration;

[0068] Combine current frame features with fused historical features Perform weighted fusion:

[0069]

[0070] in, This represents the output of the dynamic attention fusion module. This is a hyperparameter.

[0071] The operation of the convolutional exponential time integrator module is as follows:

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] in, and These are the previous hidden state and the cellular state, respectively. , This represents a two-dimensional convolutional layer with a kernel size of 3 and padding of 1. , , and These represent the outputs of the input gate, forget gate, output gate, and cell state update gate, respectively. It is the Sigmoid activation function. This indicates element-wise multiplication.

[0079] Furthermore, the specific process of step S4 is as follows:

[0080] Step S41: Based on the segmentation results, obtain the polyp area, instrument area, polyp centroid, and instrument centroid. Then calculate the distance between the polyp centroid and the instrument centroid, the overlap area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the polyp area change rate, and the instrument speed; where:

[0081] Step S42: Take the distance between the centroid of the polyp and the centroid of the instrument, the overlapping area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the rate of change of the polyp area, and the speed of the instrument as feature values. Set thresholds for the distance between the centroid of the polyp and the centroid of the instrument, the overlapping area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the rate of change of the polyp area, and the speed of the instrument, respectively. Compare each feature value of each frame of the image with the corresponding threshold. Whenever a feature value does not meet the threshold requirement, the risk score increases by 1.

[0082] The cumulative value of the statistical risk score is used to classify the risk level based on the current cumulative risk score.

[0083] According to another aspect of the present invention, a system for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video is provided. The system includes an image data preprocessing module, a collaborative segmentation network module, and a real-time risk early warning module, wherein:

[0084] The image data preprocessing module is used to acquire real digestive endoscopy surgical videos and generate virtual digestive endoscopy surgical videos. After the videos are processed by frame segmentation, they are manually annotated to finally construct a comprehensive instrument and polyp segmentation database containing real and virtual data.

[0085] The collaborative segmentation network module is trained using the constructed database. The trained collaborative segmentation network module is used to achieve pixel-level collaborative segmentation of polyps and surgical instruments in the digestive endoscopy video to be processed, and outputs segmented images of polyps and instruments.

[0086] The real-time risk warning module is used to perform risk assessment and real-time warning based on the segmentation results.

[0087] The beneficial effects of this invention are:

[0088] 1. This invention achieves accurate and real-time segmentation in complex dynamic environments. By constructing a unique collaborative learning architecture, IAST-Net, it achieves joint high-precision segmentation of polyps and surgical instruments, effectively handling dynamic and complex surgical environments. This architecture introduces a wavelet transform feature decomposer and a diffusion block-based decoder in the spatial domain, enabling image decomposition into low-frequency and high-frequency components. Iterative optimization enhances boundary delineation and suppresses background noise. In the temporal domain, the network employs a dynamic attention fusion module and a convolutional exponential temporal integrator. It selectively fuses temporal information by calculating the similarity between the current frame and historical frames, and dynamically scales new information through an exponential gating mechanism, thereby effectively capturing long-distance spatiotemporal dependencies and maintaining temporal consistency. On the EPD (Endo Polyp-Instrument Dataset), the method achieves 0.979 mIoU, 0.982 F1-score, and 0.989 Accuracy. Furthermore, the method achieves 59 FPS on the EPD dataset and 63 FPS on the CVC-VideoClinicDB dataset. These metrics fully demonstrate the invention's ability to provide real-time and accurate segmentation results in complex dynamic scenarios, laying a solid foundation for subsequent quantitative analysis.

[0089] 2. This invention provides objective safety feedback and real-time high-risk warnings for the interaction between surgical instruments and polyps, significantly reducing the incidence of complications. Based on the high-precision joint segmentation results of the segmentation network, the method of this invention can quantitatively analyze key features such as polyp area, polyp area change rate, surgical instrument speed, distance between the polyp and surgical instrument, and the overlapping area and percentage of the polyp and surgical instrument. Combined with preset risk thresholds, the system can assess the surgical risk level (low, medium, high) in real time and clearly identify the key contributing factors that lead to the risk. This real-time quantitative risk assessment capability effectively solves the pain point of lack of operational feedback for doctors and surgical robots, and can provide early warnings of potentially high-risk operations (such as excessive force or instrument distance being too close), thereby effectively preventing complications such as soft tissue tears and massive bleeding. By providing objective and quantifiable real-time feedback, it significantly reduces the cognitive burden on doctors and improves the overall safety and intelligence level of minimally invasive digestive endoscopic surgery.

[0090] 3. Improved standardization, trainability, and research potential of endoscopists' procedures. The high-precision segmentation results provided by the segmentation network ensure the reliability of subsequent quantitative indicators. Video-level quantitative indicators such as polyp instrument interaction time, high-risk duration, and video risk score generated based on this precise data provide objective evidence and personalized quantitative feedback for evaluating physicians' operational skills. This not only helps new endoscopists master operational skills more quickly and in a more standardized manner but also helps experienced physicians further optimize their operational procedures and improve the overall standardization of operations. In addition, the detailed quantitative features per frame and comprehensive video-level indicators generated by this invention form structured and analyzable large-scale data results, which can be used to conduct in-depth research on the relationship between different operational modes and clinical prognosis, laying the foundation for developing more intelligent endoscopic-assisted diagnostic and treatment systems. Achieving an mIoU of 0.915 and an F1-score of 0.955 on synthetic datasets further validates the model's application potential in future complex scenarios and robot-assisted surgeries.

[0091] 4. Spatial Domain Refined Segmentation Capability. The segmentation network architecture decomposes the image into low-frequency and high-frequency components through a wavelet transform feature decomposer, effectively separating global structural information from fine texture edges. Combined with a diffusion feature refinement module, through iterative optimization and noise injection, it significantly enhances the accuracy of boundary depiction and suppresses background clutter, especially excelling in handling challenging problems such as blurred edges, specular reflections, and small polyps.

[0092] 5. Temporal Consistency Modeling Capability. This invention employs a dynamic attention fusion module, which selectively fuses key temporal context information by calculating the similarity between the current frame and historical frames, effectively addressing dynamic changes such as illumination fluctuations, instrument occlusion, and polyp deformation. Simultaneously, the convolutional exponential temporal integrator utilizes a convolutional LSTM structure with exponential gating to capture long-distance spatiotemporal dependencies, ensuring high consistency and robustness in continuous video streams and overcoming the segmentation instability problem caused by single-frame processing in traditional methods.

[0093] 6. Multi-dimensional Quantification and Intelligent Risk Assessment. Based on high-precision segmentation results, the method of this invention can extract core quantitative features such as polyp area, surgical instrument speed, distance between polyp and surgical instrument, overlap area, overlap percentage, and area change rate, comprehensively depicting the dynamic interaction process between the polyp and surgical instruments. Furthermore, a multi-level risk assessment mechanism is designed, providing real-time frame-level risk scoring and aggregated video-level risk indicators, enabling macro-level assessment and retrospective analysis of the entire surgical process.

[0094] 7. Excellent real-time performance and data traceability. The segmentation network achieves processing speeds of over 50-60 FPS on multiple datasets (EPID, CVC-VideoClinicDB), ensuring that the method of this invention can provide near real-time processing and risk warnings, and all quantified data can be traced back to specific video frames.

[0095] This invention, through the constructed collaborative learning architecture IAST-Net, can simultaneously perform high-precision segmentation of polyps and instruments in colonoscopy videos. Moreover, it provides early warnings based on the segmentation results, solving the problems of difficulty in accurate and real-time analysis, lack of objective feedback during operation, and delayed or missing early warnings of high-risk events in current colonoscopy surgery, thereby significantly improving the safety and intelligence level of the surgery. Attached Figure Description

[0096] Figure 1 These are segmentation results from different image segmentation models;

[0097] In the diagram, GT is the actual bounding box;

[0098] Figure 2 These are the segmentation results of different video segmentation models in scenarios where there are significant differences between consecutive frames;

[0099] Figure 3 It is virtual data video generation;

[0100] a is image 1 corresponding to the virtual video, and b is image 2 corresponding to the virtual video;

[0101] Figure 4This is a flowchart of a method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to the present invention; in the figure, the dual-branch wavelet feature encoder includes a wavelet transform feature decomposer and a dual-branch wavelet feature extraction module;

[0102] Figure 5 This is a diagram combining wavelet transform and diffusion feature optimization modules;

[0103] Wavelet Transform represents the wavelet transform feature decomposer; upsample represents the upsampling layer; Diffusion Decoder represents the diffusion decoding unit; Diffusion Block represents the diffusion block, and each yellow cuboid in the figure represents 4 diffusion blocks.

[0104] Figure 6 This is a schematic diagram of the dynamic attention fusion module;

[0105] Figure 7 This is a schematic diagram of the quantitative real-time early warning system module;

[0106] Figure 8 This is a schematic diagram illustrating the segmentation result of the method of the present invention in a dynamic interaction scenario between surgical instruments and polyps. Detailed Implementation

[0107] Specific implementation method one: Combining Figure 4 This embodiment describes a method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video. The method specifically includes the following steps:

[0108] Step S1: Construct a digestive endoscopy surgery video dataset through actual collection and simulation: The training data of this invention comes from real-collected digestive endoscopy surgery video data and virtual data generated through a digestive endoscopy surgery simulation platform, aiming to provide a large-scale, diverse and finely labeled dataset for training and validating models.

[0109] Step S11: Collect real digestive endoscopy video data. This video data includes the complete process of endoscopic operation, movement of surgical instruments, and lesions such as polyps. After the collected digestive endoscopy video data is processed by frame segmentation, each frame image is manually annotated by three experienced professionals. Through manual annotation, spatial segmentation and semantic classification labels are assigned to instruments, polyps, and background in each frame image. That is, the outlines of instruments and polyps are accurately delineated in each frame image and distinguished from the background. At the same time, corresponding semantic labels are assigned to these segmented areas, such as "biopsy forceps", "electrocoagulation hook", "polyp", etc.

[0110] To overcome the limitations and scarcity of real digestive endoscopic surgical video data collection, and to effectively enhance the model's generalization ability and robustness, this invention constructs a comprehensive virtual digestive endoscopic surgical video dataset. The specific construction method is as follows:

[0111] Step S12: Create a detailed 3D colon model and a precisely scaled surgical instrument model using Maya 3D modeling software. Then, generate virtual data covering complex interaction scenarios between various surgical instruments and polyps by designing a digestive endoscopy surgery simulation, filling the gaps in rare or hard-to-obtain scenarios in real-world data. After frame-by-frame processing, we can automatically and accurately obtain spatial segmentation and semantic classification labels of instruments, polyps, and background in each frame image by utilizing the inherent ground truth information of the virtual environment.

[0112] It should be noted that the 3D colon model created in this step was obtained by processing real CT scan data and performing precise model cutout and 3D reconstruction in Mimics software, ensuring that the model can accurately simulate the anatomical structure and physiological morphology of the human digestive tract. To enhance realism, intestinal wall texture, blood vessel distribution, and physiological folds, texture technology extracted from real endoscopic image data was meticulously mapped onto the surface of the 3D colon model, ensuring its visual effect is highly consistent with the real intestine. The surgical instrument model was modeled strictly according to the size, shape, mechanical characteristics, and multiple degrees of freedom of real instruments, ensuring its behavior in the virtual environment is highly consistent with real instruments. Based on the constructed 3D colon model and surgical instrument model, a surgical simulation was carefully designed to simulate digestive endoscopic surgical scenarios and model and generate complex and diverse interaction scenarios between digestive endoscopic surgical instruments and polyps. Through simulation, complex interactions that are difficult to obtain in large quantities in real surgery can be systematically generated, such as the various manipulation methods of instruments on polyps of different shapes, sizes, and locations. The images corresponding to the generated virtual video data are as follows: Figure 3 As shown, in the simulation environment, image data from various views can be generated at high resolution, and pixel-level precise annotations can be automatically obtained, including the precise location, shape, and category information of instruments and polyps, which is more efficient and accurate than manual annotation methods.

[0113] Step S13: The collected real-world digestive endoscopic surgery video annotation data is combined with the virtual dataset to establish a comprehensive segmentation database of instruments and polyps. This ensures that the dataset encompasses both the complexity of the real world and the diversity and controllability of the virtual data. After preprocessing each frame of the annotated image, the dataset, composed of preprocessed images from both real and virtual data, is divided into three parts: a training set, a validation set, and a test set, with a ratio of 7:2:1. Wherein:

[0114] The training set is used for model learning and parameter optimization;

[0115] The validation set is used to evaluate model performance during training, adjust hyperparameters, and effectively prevent overfitting.

[0116] The test set, as independent data without training and validation processes, is used to ultimately and objectively evaluate the model's true performance and generalization ability.

[0117] Step S2: Construct a collaborative learning network model (IAST-Net) to integrate spatial and temporal contextual information to achieve collaborative segmentation of polyps and surgical instruments in colonoscopy videos. The collaborative learning network model includes modules in the spatial domain, modules in the temporal domain, and a Softmax activation function. For example... Figure 5 As shown, in the spatial domain, the collaborative learning network model includes a wavelet transform feature decomposer, a two-branch wavelet feature extraction module, and a diffuse block-based decoder; in the temporal domain, the collaborative learning network model includes a dynamic attention fusion module and a convolutional exponential time integrator.

[0118] The output of the convolutional exponential time integrator is then passed through the Softmax activation function, and the segmentation result is obtained based on the output of the Softmax activation function.

[0119] The collaborative learning network model can capture and integrate spatiotemporal dependencies. The collaborative learning network model is trained using the comprehensive dataset constructed in step S1 to optimize the network parameters, and finally the optimal parameter set is determined, thereby forming an efficient and robust segmentation model to achieve high-precision temporal segmentation of polyps and surgical instruments in digestive endoscopy video sequences.

[0120] (1) To address the common problems of low contrast, uneven lighting, and noise in endoscopic videos, this invention introduces a wavelet transform feature decomposer. The wavelet transform feature decomposer can efficiently decompose input surgical images into low-frequency and high-frequency components, thereby simultaneously extracting global structural information and local fine-grained details, which is crucial for colonoscopy video analysis. Specifically, each input image is decomposed into a low-frequency subband (LL) and three high-frequency subbands (LH, HL, HH) using discrete wavelet transform (DWT). The LL subband robustly preserves the global structural information of polyps and surgical instruments, enabling the network to recognize the overall shape and spatial location. The LH, HL, and HH subbands capture fine edge and texture details, which is crucial for accurately delineating boundaries and fine structures under noisy conditions. This decomposition process also strategically reduces channel dimensions to eliminate redundant information, thereby enhancing the isolation of high-frequency features. This module integrates and normalizes the three high-frequency subbands, ensuring synergistic enhancement with low-frequency data. By utilizing wavelet transform for efficient decomposition of image signals, this method introduces frequency domain information into the network, generating richer and more comprehensive feature representations. Spatial domain features are decomposed into high-frequency components containing local details and low-frequency components encoding global features, providing the segmentation network with a holistic feature set that fully leverages frequency domain insights. This hierarchical frequency domain fusion framework jointly models low-frequency anatomical structures and high-frequency edge features, thereby achieving precise decoupling and collaborative optimization of complex surgical scenarios. The calculation method for the fused features is as follows:

[0121]

[0122] in, Indicates the input image. It is a transformation function. Indicates the use of wavelet basis Two-dimensional discrete wavelet transform, generating low-frequency subbands. (Global structure) and high-frequency subband , , (Edge details). It is a normalization function that scales pixel values ​​to the range [0, 1]. Extracting high-frequency subbands , This indicates feature splicing.

[0123] Taking any frame of image as an example, the working process of the wavelet transform feature decomposer is as follows:

[0124] Step C1: Perform wavelet transform on the image to separate the low-frequency (structure, contour) and high-frequency (detail, texture) information. The specific process is as follows:

[0125] Step C11: Convert the input RGB image to a grayscale image; if the input is already a grayscale image, skip step C11.

[0126] Step C12: Perform a two-dimensional discrete wavelet transform on the obtained grayscale image (preferably using the "db2" wavelet basis function, with a decomposition level of 1). The wavelet transform decomposes the grayscale image into four sub-bands: LL, LH, HL, and HH; where:

[0127] The LL component is the low-frequency part of an image, representing the main contours and overall structure of the image.

[0128] The LH component is the high-frequency part of an image, representing detailed information in the horizontal direction.

[0129] The HL component is the high-frequency part of an image, representing the detailed information in the vertical direction of the image.

[0130] The HH component is the high-frequency part of an image, representing detailed information along the diagonal direction.

[0131] Step C13: Normalize the images of the LL, LH, HL and HH subbands respectively, and scale the pixel values ​​of each subband image to the range [0, 1].

[0132] Step C14: The images of the three sub-bands LH, HL and HH after normalization are added and fused pixel by pixel (i.e., energy adaptive addition) to obtain a comprehensive high-frequency feature map, which captures the fine texture and edge information of the image.

[0133] The image of the normalized LL subband is used as the low-frequency feature map, and the label mask of the high-frequency feature map and the low-frequency feature map is obtained according to the image label.

[0134] Step C2: Adjust the size of the integrated high-frequency feature map, low-frequency feature map, and label mask;

[0135] Step C21: Adjust the size of the integrated high-frequency feature map and low-frequency feature map to the preset size. The adjustment method is bilinear interpolation to maintain a smooth transition of image content.

[0136] Step C22: Adjust the label mask of the integrated high-frequency feature map and low-frequency feature map to a preset size. The adjustment method is nearest neighbor interpolation to avoid introducing new pixel values ​​and maintain the discreteness of the labels.

[0137] Step C23: Convert the high-frequency feature map and low-frequency feature map obtained in step C2 into float32 type, and normalize the standard deviation of the converted high-frequency feature map and normalize the mean of the converted low-frequency feature map.

[0138] Convert the label mask of the low-frequency feature map obtained in step C2 to a PyTorch FloatTensor type, and convert the label mask of the high-frequency feature map obtained in step C2 to a PyTorch LongTensor type.

[0139] The purpose of step C23 is to adapt to the input requirements of the neural network and accelerate the training process.

[0140] (2) Dual-branch wavelet feature extraction module and diffuse block-based decoder

[0141] To address challenges such as blurred polyp boundaries, complex specular reflections from medical devices, and difficulties in integrating multi-scale features, this invention introduces a dual-branch wavelet feature extraction module and a diffusion feature refinement unit (DFR) based on a diffusion block decoder. DFR injects scaled randomness into the feature space through controlled noise perturbation, enhancing the model's robustness to illumination fluctuations and viewing angle changes, and progressively refines features, suppressing artifacts and amplifying true boundary details.

[0142] The working process of the dual-branch wavelet feature extraction module is as follows:

[0143] In step C23, the converted low-frequency feature map passes sequentially through the first Double Basic block, the first upsampling layer, the second Double Basic block, the second upsampling layer, the third Double Basic block, the third upsampling layer, the fourth Double Basic block, the fourth upsampling layer, and the fifth Double Basic block within the first branch.

[0144] That is, the low-frequency feature map is processed through the first Double Basic block to obtain the output of the first Double Basic block; then the output of the first Double Basic block is used as the input of the first upsampling layer; the output of the first upsampling layer is used as the input of the second Double Basic block; the output of the second Double Basic block is used as the input of the second upsampling layer; the output of the second upsampling layer is used as the input of the third Double Basic block; the output of the third Double Basic block is used as the input of the fourth upsampling layer; and the output of the fourth upsampling layer is used as the input of the fifth Double Basic block.

[0145] The high-frequency feature map transformed in step C23 passes sequentially through the sixth Double Basic block, the fifth upsampling layer, the seventh Double Basic block, the sixth upsampling layer, the eighth Double Basic block, the seventh upsampling layer, the ninth Double Basic block, the eighth upsampling layer, and the tenth Double Basic block in the second branch.

[0146] Therefore, two parallel coding branches are used to process the low-frequency and high-frequency feature maps obtained from wavelet transform, respectively. Each coding branch consists of multiple Double Basic blocks, each used to extract features at different scales layer by layer. Within each coding branch, upsampling layers are used to gradually increase the receptive field of the feature maps and reduce the spatial dimension. Convolutional layers with a kernel size of 1×1 and BN layers are used to adjust the number of channels and promote information flow. Deep within the two coding branches, the feature maps of the same level from the two coding branches are concatenated to form a fused multi-scale feature representation, which is then used as the input to the decoder.

[0147] It should be noted that the working process of the first Double Basic block is as follows:

[0148] Within the first Double Basic block, the input of the first Double Basic block first passes through the first convolutional layer, the first ReLU activation function layer, and the second convolutional layer. The output of the second convolutional layer is then concatenated with the input of the first Double Basic block to obtain the concatenated result a.

[0149] Then, the concatenated result a is passed through the second ReLU activation function layer, the third convolutional layer, the third ReLU activation function layer, and the fourth convolutional layer. The output of the fourth convolutional layer is concatenated with the output of the third ReLU activation function layer to obtain the concatenated result b.

[0150] The concatenated result b is used as the input to the fourth ReLU activation function layer, and the output of the fourth ReLU activation function layer is used as the output of the first Double Basic block.

[0151] Furthermore, the working process of the second to tenth Double Basic blocks is the same as that of the first Double Basic block.

[0152] The diffusion block-based decoder includes a first diffusion decoding unit, a second diffusion decoding unit, a third diffusion decoding unit, a fourth diffusion decoding unit, and a fifth diffusion decoding unit. The working process of the diffusion block-based decoder is as follows:

[0153] After concatenating the output of the first Double Basic block with the output of the sixth Double Basic block, the concatenation result is used as the input of the first diffusion decoding unit;

[0154] The output of the first diffusion decoding unit, the output of the second Double Basic block, and the output of the seventh Double Basic block are concatenated, and the concatenation result is used as the input of the second diffusion decoding unit.

[0155] The output of the second diffusion decoding unit, the output of the third Double Basic block, and the output of the eighth Double Basic block are concatenated, and the concatenation result is used as the input of the third diffusion decoding unit.

[0156] The output of the third diffusion decoding unit, the output of the fourth Double Basic block, and the output of the ninth Double Basic block are concatenated, and the concatenation result is used as the input of the fifth diffusion decoding unit.

[0157] The output of the fifth diffusion decoding unit, the output of the fifth Double Basic block, and the output of the tenth Double Basic block are concatenated, and the concatenation result is used as the output of the diffusion block-based decoder.

[0158] The first diffusion decoding unit includes a first diffusion block to an eighth diffusion block in sequence. The input of the first diffusion decoding unit is used as the input of the first diffusion block, the output of the first diffusion block is used as the input of the second diffusion block, and so on. The output of the j-th diffusion block is used as the input of the (j+1)-th diffusion block, and the output of the eighth diffusion block is used as the output of the first diffusion decoding unit.

[0159] Each diffusion block includes a Gaussian noise layer, a convolutional layer, a BN layer, and a ReLU activation function layer. Within the diffusion block, the input of the diffusion block passes through the Gaussian noise layer, the convolutional layer, the BN layer, and the ReLU activation function layer in sequence, and the output of the ReLU activation function layer is used as the output of the diffusion block.

[0160] The second diffusion decoding unit includes an upsampling layer, a first diffusion block to an eighth diffusion block in sequence. The input of the second diffusion decoding unit is used as the input of the upsampling layer, and the output of the upsampling layer is used as the input of the first diffusion block. The output of the first diffusion block is used as the input of the second diffusion block, and so on. The output of the j-th diffusion block is used as the input of the (j+1)-th diffusion block, and the output of the eighth diffusion block is used as the output of the second diffusion decoding unit.

[0161] Furthermore, the working processes of the third, fourth, and fifth diffusion decoding units are all the same as those of the second diffusion decoding unit.

[0162] The core of the decoder is the diffusion block, in which random noise is explicitly added to the feature map within the internal loop of each diffusion block. This noise injection, together with subsequent convolutional layers, simulates the "diffusion" process of information in the feature space, helping the model learn more robust feature representations and potentially enhancing the model's adaptability to noise and detail variations. The decoder upsamples the feature map of the current layer to increase its resolution and concatenates it with the corresponding scale feature map from the encoder. By using skip connections, it effectively fuses low-level detail information and high-level semantic information, and gradually upsamples to restore spatial resolution.

[0163] Among them: Gaussian noise addition layer in In each diffusion step, the input feature map is processed... Refine and generate output The specific working process of the Gaussian noise addition layer is as follows:

[0164]

[0165] in, Input feature map; It is the first Feature map after adding noise; This indicates the total number of steps to add noise; It is a nonlinear transformation; These are hyperparameters used to control noise intensity; For added noise, obey distributed, This indicates that the mean is 0 and the variance is 0. Gaussian distribution, It is the identity matrix;

[0166]

[0167] in, It is a 3×3 convolution kernel. This represents the convolution operation. It is a batch of normalization, It is an activation function;

[0168]

[0169] in, Indicates the first Feature map after adding noise; This represents the projection operation of the feature map, used to adjust the channel dimensions; This represents the output of the Gaussian noise addition layer;

[0170]

[0171] in, Indicates weight, Indicates bias.

[0172] noise It improved the visibility of abnormal polyp areas and instrumentation areas, while Iterative noise reduction enhances feature reliability and produces clear distinctions between polyps and tissues, and between instruments and reflective background. Controlled feature space diversification is achieved through noise injection, while subsequent nonlinear transformations enable implicit variational filtering, effectively decoupling semantically meaningful patterns from typical imaging artifacts and illumination variations in the complex environment of endoscopy. Embedding diffusion blocks in hierarchical diffusion decoding units, acting on multiple resolution levels after skip connections, effectively combines high-resolution spatial cues and low-resolution semantic context to optimize multi-scale integration, overcoming the limitations of traditional fusion methods.

[0173] (3) Dynamic attention fusion module

[0174] To address the finite temporal consistency problem in video sequences caused by dynamic scene factors such as camera movement, polyp deformation, and lighting changes, this invention proposes a Dynamic Attention Fusion (DAF) module to effectively fuse feature information from the current frame and historical frames. This mechanism selectively fuses features from historical frames with the current frame, efficiently utilizing temporal context and improving segmentation robustness and accuracy. The DAF module solves the problem of maintaining segmentation consistency between video frames by dynamically focusing on relevant historical features, mitigating the impact of transient interference on single-frame analysis. This is crucial in colonoscopy videos, as reliable clues may be occluded in a single frame, thus requiring a strategy for effectively propagating temporal context. The DAF module integrates a fixed number of preceding frame information based on an attention mechanism, balancing computational efficiency and performance.

[0175] like Figure 6 As shown, the operation process of the DAF module can be formally described as follows:

[0176] The feature map output by the current frame after passing through the diffuse block-based decoder is denoted as... , The list of feature maps output by the diffusion block-based decoder for historical frames is denoted as... This means taking the k frames before the current frame as historical frames;

[0177] In this invention, The DAF module projects these features into query, key, and value representations. Specifically, it converts the current frame features into a query vector, converts each historical frame in the historical frame feature list into a key vector, and converts each historical frame in the historical frame feature list into a value vector to calculate spatial attention weights. To optimize efficiency and reduce computational complexity, this invention uses a reduction factor to appropriately reduce the number of channels for queries and keys, setting... :

[0178]

[0179] in, , and All are convolutional layers with a kernel size of 1×1. and Used to reduce the number of channels to ,and The original number of channels is retained. Therefore, and All dimensions are , The dimension is ;

[0180] Next, the query for the current frame is multiplied element-wise with the key of each historical frame, and then summed along the channel dimension to obtain a spatial similarity map, which is used to evaluate the relevance of historical frames at each pixel location:

[0181]

[0182] in, express The Middle Data from each channel, express The Middle Data from each channel, This represents element-wise multiplication. Indicates the current frame and the first frame. Similarity scores of historical frames ;

[0183] The spatial similarity maps of all historical frames are stacked in the channel dimension, and the Softmax function is applied to the stacking result to normalize it in the historical frame dimension, generating an attention weight map that indicates the degree of contribution of each historical frame to the current frame.

[0184]

[0185] in, , , Represents the attention weight matrix. Indicates the first Attention weights for each historical frame at each spatial location, Represents tensor The first dimension performs a Softmax operation, which means that for each spatial location (h,w), the attention weight of the k-th historical frame at that location is calculated, so that the sum of the attention weights of all historical frames at that location is 1, ensuring that for each pixel location in the image, the attention weights of all historical frames are a probability distribution.

[0186] The value vectors of historical frames are weighted and summed based on the calculated attention weights to obtain the fused historical features:

[0187]

[0188] in, This indicates that attention weights will be broadcast to On all the passages, Indicates the historical characteristics after the integration. ;

[0189] Combine current frame features with fused historical features Perform weighted fusion:

[0190]

[0191] in, This represents the output of the dynamic attention fusion module. For hyperparameters, It is a learnable parameter used to adaptively adjust the contribution strength of historical information to the current frame.

[0192] The DAF module selectively weights past information by utilizing a fixed history window and spatial attention, avoiding the computational intensity of exhaustive iterative processes while enhancing the resilience of features to dynamic changes. This approach is crucial for maintaining segmentation stability when single-frame data is unreliable. When integrated into a temporal refinement framework, the DAF module provides robust, context-aware features and multi-scale consistency.

[0193] (4) Convolutional exponential time integrator module

[0194] In colonoscopy video segmentation, maintaining inter-frame temporal consistency is a key challenge due to the dynamic nature of the scene (including camera movement, polyp deformation, and lighting changes). To maintain inter-frame temporal consistency, a Convolutional Exponential Temporal Integrator (CETI) is employed. CETI enhances the integration of temporal features and captures spatiotemporal dependencies by combining convolutional operations with an exponential gating mechanism. CETI receives temporally aligned features from the Dynamic Attention Fusion (DAF) module and further refines these features through its exponential gating mechanism, implicitly enforcing smooth transitions between frames.

[0195] To model temporal dependencies in video sequences, CETI combines the standard LSTM framework, convolutional spatial feature maps, and exponential gating. All gating computations are performed through convolutional operations, enabling the network to capture local spatial and temporal correlations. CETI's operations are formally defined by the following equations, initializing the hidden states and cell states to zero tensors:

[0196] The input gate in this invention uses an exponential function activation. This exponential gating mechanism can more flexibly control the inflow of new information, especially when processing video data with dynamic characteristics, it can enhance the model's ability to capture subtle changes.

[0197]

[0198]

[0199]

[0200]

[0201] Cell state controls the retention of old information through the forget gate and controls the addition of new information through the exponential input gate and the current input.

[0202]

[0203] The hidden state is determined by both the output gate and the updated cell state:

[0204]

[0205] in, It is time The input feature map (fused features from DAF). and These are the previous hidden state and the cellular state, respectively. , This represents a 2D convolutional layer with a kernel size of 3 and padding of 1. It should be noted that the input gate, forget gate, output gate, and cell state update gate each have their own convolutional layers; however, the representation of convolutional layers is not differentiated here. , , and These represent the outputs of the input gate, forget gate, output gate, and cell state update gate, respectively. , It is the Sigmoid activation function. This indicates element-wise multiplication.

[0206] The exponential input gate is the cornerstone of CETI's innovation. Unlike traditional LSTMs which use the sigmoid function to... The constraints differ between 0 and 1; the exponential function allows... Take unbounded positive values. This allows the model to dynamically scale new information based on the input and previous state. This amplifies key features and suppresses noise, which is particularly advantageous in the complex dynamics of colonoscopy videos.

[0207] Step S3, Model Application:

[0208] The digestive endoscopy video images to be processed are used as input to the trained collaborative learning network model (i.e., the hidden state of the last layer output). The trained collaborative learning network model outputs high-precision segmentation results of polyps and instruments in the digestive endoscopy video in real time.

[0209] Step S4: The frame-by-frame polyp segmentation mask and surgical instrument segmentation mask generated after processing by the segmentation network model accurately identify the pixel-level contours and spatial positions of polyps and instruments. Risk assessment and real-time early warning are performed based on the segmentation results of step S3.

[0210] like Figure 7 As shown, the specific process of step S4 is as follows:

[0211] Step S41: Based on the segmentation results, obtain the polyp area, instrument area, polyp centroid, and instrument centroid. Then calculate the distance between the polyp centroid and the instrument centroid, the overlap area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the polyp area change rate, and the instrument speed; where:

[0212] The areas of the polyp region and the instrument region are calculated by counting the number of non-zero pixels in the mask, representing the size of the target in the field of view.

[0213] Polyp centroid and instrument centroid: The coordinates of the geometric center are calculated using the zero-order and first-order moment methods of the image, serving as representative positions for subsequent tracking and distance calculation.

[0214] Distance between the centroid of the polyp and the instrument: Spatial proximity is quantified using Euclidean distance.

[0215] The overlap area between the polyp and the instrument: the number of non-zero pixels in the logic and operation of the polyp mask and the instrument mask, quantifying the physical contact range.

[0216] The percentage overlap between the polyp area and the instrument area can more robustly reflect the intensity of the interaction.

[0217] Polyp area change rate: The difference in polyp area between the current frame and the previous frame divided by the time interval. Abnormal changes may indicate unexpected operations.

[0218] Surgical instrument speed: The Euclidean distance between the instrument centroids of the current frame and the previous frame divided by the time interval. Excessive speed indicates recklessness and risk in operation.

[0219] Step S42: Take the distance between the centroid of the polyp and the centroid of the instrument, the overlapping area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the rate of change of the polyp area, and the speed of the instrument as feature values. Set thresholds for the distance between the centroid of the polyp and the centroid of the instrument, the overlapping area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the rate of change of the polyp area, and the speed of the instrument, respectively. Compare each feature value of each frame of the image with the corresponding threshold. Whenever a feature value does not meet the threshold requirement, the risk score increases by 1.

[0220] The cumulative risk score is calculated. In this invention, the risk score is accumulated starting from the first frame. If there is a feature value in the first frame that does not meet the requirements, the risk value is increased by 1. This process is repeated, and the risk value is accumulated starting from the first frame. The risk level is divided according to the current cumulative risk score. Therefore, this invention can obtain the cumulative risk value corresponding to each moment and obtain the risk level up to each moment.

[0221] To capture dynamic risk trends, the system accumulates risk scores across consecutive frames, resulting in a total cumulative risk score ranging from 0 to 25. Based on this, risk levels are clearly defined, providing real-time, quantitative safety warnings for surgical procedures.

[0222] A cumulative risk score between 0 and 10 is considered low risk.

[0223] A score of 11-20 points indicates a medium risk level.

[0224] A score exceeding 20 points is considered high-risk.

[0225] This invention utilizes precise image segmentation for real-time early warning: Deep analysis is performed on the frame-by-frame segmentation masks of polyps and surgical instruments in a continuous video stream to precisely extract and quantify key geometric features (including pixel area of ​​polyps and surgical instruments, centroid coordinates, Euclidean distance between polyps and instruments, pixel area of ​​overlapping regions and their percentage), and key dynamic interaction indicators (polyp area change rate and surgical instrument speed). An intelligent risk assessment model based on these multi-dimensional indicators is constructed, comprehensively quantifying various indicators. Through preset adjustable thresholds, the surgical risk level (low, medium, high) is calculated and classified in real time, while clearly identifying the key contributing factors leading to the risk.

[0226] The training, validation, testing, and prediction processes of the model are explained in further detail below:

[0227] This invention uses the PyTorch Lightning framework for model training, validation, and testing, ensuring the standardization and repeatability of the training process.

[0228] 1. Training Phase

[0229] Data input: Training data is input in batches, each batch containing multiple frames of images and their labels. The shapes of the images and labels are (B, T, C, H, W) and (B, T, H, W) respectively, where B is the batch size and T is the sequence length; the input data is propagated forward within the model.

[0230] Loss function: Cross-entropy loss is used to calculate the loss between the model output and the true label.

[0231] Optimizer and Learning Rate Scheduler: The AdamW optimizer is used to optimize the model parameters; the WarmupCosineAnnealingLR learning rate scheduler is adopted. This scheduler first warms up the learning rate, and then the learning rate decreases in a cosine curve, which helps the model to converge stably in the early stage of training and avoid getting trapped in local optima.

[0232] 2. Verification Phase

[0233] Performance metrics: For the benchmark dataset, independent performance metrics are maintained for each subset of the dataset. These metrics include mean intersection-over-union ratio (mIoU), F1 score (mF1), accuracy (ACC), and frame rate (FPS).

[0234] Validation steps: Similar to the training steps, perform forward propagation on the validation set data and calculate the cross-entropy loss.

[0235] Metric update: Update the confusion matrix based on the model output (after Softmax activation) and the true labels.

[0236] Results Recording and Logs: At the end of the validation period, the calculated performance index values ​​are written to the log file, the performance indexes on the validation set are monitored, and the best performing model is saved.

[0237] 3. Testing Phase

[0238] Perform forward propagation on the test set data and record the inference time. For the video dataset, count the total number of frames and calculate the total inference time. Also calculate the cross-entropy loss.

[0239] Update the confusion matrix based on the model output (after Softmax activation) and the true labels.

[0240] Output results: At the end of the test period, if it is a video dataset, calculate and print performance metrics such as frames per second (FPS) and average inference time.

[0241] Calculate and print comprehensive segmentation evaluation results, including mean intersection-over-union ratio (mIoU), mean F1 score (mF1), and mean accuracy (mAcc), as well as detailed metrics for each category.

[0242] For the benchmark dataset, detailed evaluation results for each subset will be displayed separately.

[0243] 4. Forecasting Phase

[0244] Input processing: Receives a batch dictionary containing low-frequency and high-frequency image data, labels, and image paths.

[0245] Model inference: The processed low-frequency and high-frequency input sequences are fed into the model for forward propagation to extract the features of the current frame. The features of the current frame are then added to the feature list for fusion of subsequent historical frames.

[0246] For frames other than the first frame, the nearest k frames are obtained from the feature list as historical features. The features of the current frame are then dynamically fused with the historical features to obtain the fused features of the current frame.

[0247] The fused features are input into a convolutional exponential temporal integrator module to update the hidden state and cell state. The updated hidden state is then converted into the segmentation prediction output for the current frame, which involves applying the Softmax activation function to obtain the probability of each pixel belonging to each category, and taking the category corresponding to the highest probability value as the final category prediction result for that pixel. Finally, the prediction results of all frames are concatenated to form the final video sequence segmentation result.

[0248] Post-processing of results: The segmentation results output by the model are reshaped and upsampled to the original label size.

[0249] Output: Returns a dictionary containing the predicted segmentation results, the corresponding image paths, and the original low-frequency and high-frequency inputs, facilitating subsequent visualization and analysis.

[0250] Specific Implementation Method Two: This implementation method describes a polyp and surgical instrument segmentation and real-time early warning system based on digestive endoscopy video. The system includes an image data preprocessing module, a collaborative segmentation network module, and a real-time risk early warning module, wherein:

[0251] The image data preprocessing module is used to acquire real digestive endoscopy surgical videos and generate virtual digestive endoscopy surgical videos. After the videos are processed by frame segmentation, they are labeled to finally construct a comprehensive instrument and polyp segmentation database containing real and virtual data.

[0252] The collaborative segmentation network module is trained using the constructed database. The trained collaborative segmentation network module is used to integrate spatial and temporal context information to achieve pixel-level collaborative segmentation of polyps and surgical instruments in the digestive endoscopy video to be processed, and output high-precision segmented images of polyps and instruments.

[0253] The real-time risk warning module is used to perform risk assessment and real-time warning based on the segmentation results. Specifically, it performs in-depth analysis on the frame-by-frame masks of the segmented polyps and surgical instruments, accurately extracts and quantifies key geometric features (including pixel area, centroid coordinates, Euclidean distance, overlapping area and percentage) and key dynamic interaction indicators (polyp area change rate, surgical instrument speed), and constructs an intelligent risk assessment model based on these multi-dimensional indicators. It calculates and classifies the surgical risk level (low, medium, high) in real time through preset thresholds, and identifies key risk contributing factors.

[0254] Results and Analysis

[0255] 1. Comparative Experiment

[0256] To provide a comprehensive evaluation, the comparative experiments covered 12 state-of-the-art segmentation methods across two main categories.

[0257] 1.1 Experimental Setup

[0258] (1) Image-based segmentation methods: These methods are widely used in image segmentation tasks, including medical imaging, and include well-known models such as UNet, MSNet, TransUNet, SegFormer, DCRNet, and CFANet. These methods utilize diverse architectures to achieve accurate segmentation and are often used as basic frameworks adaptable to specific applications.

[0259] (2) Video-based segmentation methods: This category includes advanced techniques such as Mask R-CNN, Mask2Former, YOLACT, Cascade R-CNN, SOLOv2, and Grid R-CNN. These methods excel at dynamic scene analysis and achieve robust segmentation by integrating spatiotemporal information. All experiments were conducted under the same training configuration to ensure fairness and consistency in the comparison between the two types of methods.

[0260] 1.2 Evaluation on synthetic datasets

[0261] The model constructed in this invention, trained on the EPD dataset, was evaluated on synthetic datasets. The model achieved a mIoU of 0.915, an F1 score of 0.955, and an accuracy of 0.934 on synthetic data. These results demonstrate the model's robustness and generalization ability, showcasing its ability to effectively segment surgical instruments and polyps in synthetic scenarios involving simulated instrument-polyp interactions.

[0262] 1.3 Comparison results on the EPDID dataset

[0263] Table 1 shows the comparative results on the EPDID dataset. The model of this invention achieves an mIoU of 0.979, an F1 score of 0.982, an accuracy of 0.989, and a real-time processing speed of 59 FPS on this dataset. Specifically, the method of this invention improves mIoU by 10.5% compared to the second-best video-based method Mask2former (0.886) and by 14.4% compared to YOLACT (0.856). Similarly, HAST-Net achieves an F1 score 5.9% higher than Mask2former and 9.2% higher than YOLACT. Our processing speed (59 FPS) is more than 5 times faster than Mask2former (11 FPS) and also faster than YOLACT (42 FPS). Compared to image-based methods, the model of this invention demonstrates a leading advantage in mIoU, exceeding Segformer (0.848) by 15.4% and CFANet (0.854) by 14.6%. These results highlight the accuracy and efficiency of the model of this invention in collaborative segmentation of endoscopic videos.

[0264] Table 1

[0265]

[0266] 2. Qualitative Results Analysis

[0267] The lack of robust modeling for organ-instrument interactions, combined with the temporal characteristics of video data and the inherent complexity of surgical procedures, presents significant challenges to the accurate segmentation of critical structures in endoscopic surgery. To evaluate our method, we selected three challenging scenarios for qualitative demonstration, and the results are presented in… Figure 1 These results demonstrate that our predictions align more closely with ground truth compared to existing methods, especially in complex settings. Our method exhibits robustness in cases involving small polyps or partial occlusion by surgical instruments and enhances the segmentation of instrument interaction points. In the first scenario ( Figure 1 In the first row, we examined small polyps during surgical procedures. Existing methods often have limitations in achieving precise segmentation due to the variability in light and limited field of view commonly found in surgical environments. However, our method consistently and clearly delineates polyp boundaries, thus mitigating these challenges and contributing to improved segmentation accuracy required for successful intervention. In the second scenario ( Figure 1 In the second row, we focus on polypectomy using a snare. While other methods show inconsistency in segmenting the snare tip (a key element in assessing the resection process), our method consistently identifies the distal end of the instrument. This capability supports a more accurate assessment of the spatial relationship between the polyp and the instrument, which is highly valuable for robot-assisted surgery. The third scenario ( Figure 1 The third line addresses the problem of polyp segmentation during continuous instrument movement and occlusion, a common occurrence in dynamic surgical videos. Under such conditions, the segmentation accuracy of other methods typically declines, often failing to maintain consistent target depiction amidst occlusion. In contrast, our method maintains consistent polyp segmentation even when the polyp is partially occluded by a moving instrument. While DCRNet achieves partial recognition in this case, it includes redundant colonic folds in the segmentation, indicating a deficiency in accurate depiction. Our method avoids such errors, contributing to more accurate target segmentation. These findings collectively underscore the applicability of our method in endoscopic robotic surgery, where accurate segmentation of polyp and instrument interaction points is crucial. By addressing the limitations observed in existing techniques, particularly in scenarios involving small polyps, occlusion, and dynamic interactions, our method contributes to improved accuracy and reliability in minimally invasive surgery.

[0268] The heterogeneous distribution of polyps and surgical instruments in video sequences introduces complexity to endoscopic surgery. This variability primarily stems from the dynamic nature of colonoscopy, reflected in significant differences between consecutive frames. These differences arise from large movements during the procedure, changes in polyp appearance due to instrument interaction, and alterations in camera perspective. Figure 2 As shown, despite significant temporal variations, our method consistently segments polyps and instruments. Specifically, Figure 2 The second and third rows show adjacent frames with significant variations, where our method exhibits higher segmentation quality than other video-based methods. While existing techniques typically experience a decline in polyp segmentation accuracy after rapid temporal changes, our method maintains consistent segmentation, a capability beneficial for continuous segmentation in robotic surgery. Figure 2 In the last line, the inter-frame temporal variation is minimal, and our method, along with other methods, effectively segments polyps and instruments. However, the second and third lines capture moments of significant change during instrument manipulation of polyps, illustrating the limitations observed in competing methods. These instances are affected by the flexibility of the colon and endoscope, leading to significant deformation and positional adjustments. In these cases, other methods exhibit lower segmentation accuracy, while our method maintains consistent performance. This ability to manage dynamic changes supports its practicality for real-time, accurate segmentation in endoscopic robotic surgery, especially in demanding surgical phases.

[0269] We present qualitative results on synthetic datasets, demonstrating the effectiveness of our segmentation method. The results show high accuracy, particularly in depicting the tips of surgical instruments. Our method also excels at capturing the dynamic interactions between surgical instruments and polyps, such as… Figure 8 As shown, these interactions in the segmented images are clearly illustrated. For qualitative demonstration, we selected three distinct scenes from a synthetic dataset, each represented by three frames from its video sequence. The leftmost scene depicts surgical instruments sampling a small polyp. The middle scene simulates the examination and sampling process for cancerous abnormalities. The rightmost scene illustrates the annotation process for abnormalities. In all these cases, our method consistently provides accurate segmentation, capturing instruments and targets with high fidelity. This performance highlights the ability of our method to accurately depict structures and adapt to different interactive dynamics within a simulated surgical environment.

[0270] In summary, the method of this invention maintains high accuracy and consistent segmentation performance even in challenging scenarios such as small polyps, instrument occlusion, dynamic fluid injection, and significant inter-frame variations. It excels particularly in recognizing instrument tips and interaction points, and effectively avoids including irrelevant areas such as colonic folds in the segmentation range. Furthermore, it maintains stable and reliable segmentation capabilities under complex dynamic conditions (such as large movements, instrument interactions, and tissue deformation), ensuring reliable application of the system in complex and ever-changing clinical environments.

[0271] The key performance indicators of this invention have been fully verified in rigorous experiments:

[0272] 1. Segmentation accuracy:

[0273] (1) On the Endo Polyp-Instrument Dataset (EPID), the present invention achieves an mIoU of 0.979, an F1-score of 0.982 and an accuracy of 0.989.

[0274] (2) On the CVC-Video ClinicDB dataset, the present invention achieved an mIoU of 0.897 and an F1-score of 0.940.

[0275] (3) On synthetic datasets, the model showed good generalization ability, achieving an mIoU of 0.915, an F1-score of 0.955 and an accuracy of 0.934, proving its effectiveness in simulating complex interactive scenarios.

[0276] 2. Processing speed (FPS):

[0277] (1) On the EPDID dataset, the present invention achieves a real-time processing speed of 59 FPS.

[0278] (2) On the CVC-Video ClinicDB dataset, the processing speed reached 63 FPS.

[0279] This invention enables precise segmentation of polyps and surgical instruments in endoscopic videos and provides real-time alerts based on these segmentation results. By quantitatively analyzing key features such as polyp area, polyp area change rate, surgical instrument speed, distance between the polyp and surgical instruments, and the overlapping area and percentage of overlap, combined with preset risk thresholds, this invention assesses the surgical risk level (low, medium, high) in real time. These functions aim to reduce the cognitive burden on surgeons in complex and dynamic surgical environments, ensure the safety of surgical instrument-polyp interaction, effectively prevent surgical complications, and thus improve the safety and intelligence of minimally invasive digestive endoscopic surgery.

[0280] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video, characterized in that, The method specifically includes the following steps: Step S1: Construct a digestive endoscopy surgery video dataset through actual collection and simulation. After the constructed video dataset is processed by frame segmentation, each image obtained by frame segmentation is manually labeled. Then, the labeled images are divided into three parts: training set, validation set and test set. Step S2: Construct a collaborative learning network model. In the spatial domain, the collaborative learning network model includes a wavelet transform feature decomposer, a two-branch wavelet feature extraction module, and a diffuse block-based decoder. In the temporal domain, the collaborative learning network model includes a dynamic attention fusion module and a convolutional exponential time integrator. The output of the convolutional exponential time integrator is then passed through the Softmax activation function, and the segmentation result is obtained based on the output of the Softmax activation function. The constructed collaborative learning network model is trained, validated, and tested using the training set, validation set, and test set to obtain a well-trained collaborative learning network model. Step S3: Use the digestive endoscopy video image to be processed as the input of the trained collaborative learning network model, and output the polyp and instrument segmentation results in the digestive endoscopy video in real time through the trained collaborative learning network model; Step S4: Conduct risk assessment and real-time early warning based on the segmentation results of step S3.

2. The method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to claim 1, characterized in that, The specific process of step S1 is as follows: Step S11: Collect real digestive endoscopy surgery video data. After processing the collected digestive endoscopy surgery video data into frames, manually annotate each frame image. Through manual annotation, spatial segmentation and semantic classification labels are added to the instruments, polyps and background in each frame image. Step S12: Create a 3D colon model and surgical instrument model using Maya 3D modeling software, and generate virtual data covering complex interaction scenarios between various surgical instruments and polyps by designing a digestive endoscopy surgery simulation. The generated virtual digestive endoscopy video data is processed by frame segmentation, and spatial segmentation and semantic classification labels of instruments, polyps and background in each frame image are obtained synchronously according to the generation environment of the virtual data during frame segmentation. Step S13: Divide the dataset consisting of labeled images from real and virtual data into three parts: training set, validation set, and test set.

3. The method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to claim 1, characterized in that, The working process of the wavelet transform feature decomposer is as follows: Step C1: Perform wavelet transform on the image to separate the low-frequency and high-frequency information in the image. The specific process is as follows: Step C11: Convert the input RGB image to a grayscale image; if the input is already a grayscale image, skip step C11. Step C12: Perform a two-dimensional discrete wavelet transform on the obtained grayscale image to decompose the grayscale image into four sub-bands: LL, LH, HL and HH. Step C13: Normalize the images of the LL, LH, HL and HH subbands respectively, and scale the pixel values ​​of each subband image to the range [0, 1]. Step C14: The images of the three sub-bands LH, HL and HH after normalization are added and fused pixel by pixel to obtain a comprehensive high-frequency feature map. The image of the normalized LL subband is used as the low-frequency feature map, and the label mask of the high-frequency feature map and the low-frequency feature map is obtained according to the image label. Step C2: Adjust the size of the integrated high-frequency feature map, low-frequency feature map, and label mask; Step C21: Adjust the dimensions of the integrated high-frequency feature map and low-frequency feature map to the preset dimensions; Step C22: Adjust the label mask of the integrated high-frequency feature map and low-frequency feature map to the preset size; Step C23: Convert the high-frequency feature map and low-frequency feature map obtained in step C2 into float32 type, and normalize the standard deviation of the converted high-frequency feature map and normalize the mean of the converted low-frequency feature map. Convert the label mask of the low-frequency feature map obtained in step C2 to a PyTorch FloatTensor type, and convert the label mask of the high-frequency feature map obtained in step C2 to a PyTorch LongTensor type. The dual-branch wavelet feature extraction module includes two parallel coding branches, which are used to extract the low-frequency and high-frequency feature maps obtained by the wavelet transform feature decomposer, respectively; wherein: In step C23, the converted low-frequency feature map passes sequentially through the first Double Basic block, the first upsampling layer, the second Double Basic block, the second upsampling layer, the third Double Basic block, the third upsampling layer, the fourth Double Basic block, the fourth upsampling layer, and the fifth Double Basic block within the first branch. The high-frequency feature map transformed in step C23 passes sequentially through the sixth Double Basic block, the fifth upsampling layer, the seventh Double Basic block, the sixth upsampling layer, the eighth Double Basic block, the seventh upsampling layer, the ninth Double Basic block, the eighth upsampling layer, and the tenth Double Basic block in the second branch. The working process of the first Double Basic block is as follows: Within the first Double Basic block, the input of the first Double Basic block first passes through the first convolutional layer, the first ReLU activation function layer, and the second convolutional layer. The output of the second convolutional layer is then concatenated with the input of the first Double Basic block to obtain the concatenated result a. Then, the concatenated result a is passed through the second ReLU activation function layer, the third convolutional layer, the third ReLU activation function layer, and the fourth convolutional layer. The output of the fourth convolutional layer is concatenated with the output of the third ReLU activation function layer to obtain the concatenated result b. The concatenated result b is used as the input to the fourth ReLU activation function layer, and the output of the fourth ReLU activation function layer is used as the output of the first Double Basic block.

4. The method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to claim 3, characterized in that, The diffusion block-based decoder includes a first diffusion decoding unit, a second diffusion decoding unit, a third diffusion decoding unit, a fourth diffusion decoding unit, and a fifth diffusion decoding unit. The working process of the diffusion block-based decoder is as follows: After concatenating the output of the first Double Basic block with the output of the sixth Double Basic block, the concatenation result is used as the input of the first diffusion decoding unit; The output of the first diffusion decoding unit, the output of the second Double Basic block, and the output of the seventh Double Basic block are concatenated, and the concatenation result is used as the input of the second diffusion decoding unit. The output of the second diffusion decoding unit, the output of the third Double Basic block, and the output of the eighth Double Basic block are concatenated, and the concatenation result is used as the input of the third diffusion decoding unit. The output of the third diffusion decoding unit, the output of the fourth Double Basic block, and the output of the ninth Double Basic block are concatenated, and the concatenation result is used as the input of the fifth diffusion decoding unit. The output of the fifth diffusion decoding unit, the output of the fifth Double Basic block, and the output of the tenth Double Basic block are concatenated, and the concatenation result is used as the output of the diffusion block-based decoder.

5. The method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to claim 4, characterized in that, The first diffusion decoding unit includes a first diffusion block to an eighth diffusion block in sequence. The input of the first diffusion decoding unit is used as the input of the first diffusion block, the output of the first diffusion block is used as the input of the second diffusion block, and so on. The output of the j-th diffusion block is used as the input of the (j+1)-th diffusion block, and the output of the eighth diffusion block is used as the output of the first diffusion decoding unit. Each diffusion block includes a Gaussian noise layer, a convolutional layer, a BN layer, and a ReLU activation function layer. Within the diffusion block, the input of the diffusion block passes through the Gaussian noise layer, the convolutional layer, the BN layer, and the ReLU activation function layer in sequence, and the output of the ReLU activation function layer is used as the output of the diffusion block.

6. The method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to claim 4, characterized in that, The second diffusion decoding unit includes an upsampling layer, a first diffusion block to an eighth diffusion block in sequence. The input of the second diffusion decoding unit is used as the input of the upsampling layer, and the output of the upsampling layer is used as the input of the first diffusion block. The output of the first diffusion block is used as the input of the second diffusion block, and so on. The output of the j-th diffusion block is used as the input of the (j+1)-th diffusion block, and the output of the eighth diffusion block is used as the output of the second diffusion decoding unit. Furthermore, the working processes of the third, fourth, and fifth diffusion decoding units are all the same as those of the second diffusion decoding unit.

7. The method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to claim 5, characterized in that, The working process of the Gaussian noise addition layer is as follows: in, Input feature map; It is the first Feature map after adding noise; This indicates the total number of steps to add noise; It is a nonlinear transformation; For hyperparameters; For added noise, obey distributed, This indicates that the mean is 0 and the variance is 0. Gaussian distribution, It is the identity matrix; in, It is a convolution kernel. This represents the convolution operation. It is a batch of normalization, It is an activation function; in, Indicates the first Feature map after adding noise; Represents the projection operation of the feature map; This represents the output of the Gaussian noise addition layer; in, Indicates weight, Indicates bias.

8. The method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to claim 1, characterized in that, The working process of the dynamic attention fusion module is as follows: The feature map output by the current frame after passing through the diffuse block-based decoder is denoted as... The list of feature maps output by the diffusion block-based decoder for historical frames is denoted as... ; The current frame feature map Convert to query vectors: Convert the feature map of each historical frame in the historical frame feature list into a key vector, and convert the feature map of each historical frame in the historical frame feature list into a value vector. in, Represents the feature map of the current frame The corresponding query vector, Indicates the first The key vector corresponding to each historical frame Indicates the first The value vector corresponding to each historical frame; in, express The Middle Data from each channel, express The Middle Data from each channel, This represents element-wise multiplication. Indicates the current frame and the first frame. Similarity scores of historical frames; in, Represents the attention weight matrix; The value vectors of historical frames are weighted and summed based on the calculated attention weights to obtain the fused historical features: in, Indicates the first Attention weights for each historical frame at each spatial location, This indicates that attention weights will be broadcast to On all the passages, This indicates the historical characteristics after the integration; Combine current frame features with fused historical features Perform weighted fusion: in, This represents the output of the dynamic attention fusion module. For hyperparameters; The operation of the convolutional exponential time integrator module is as follows: in, and These are the previous hidden state and the cellular state, respectively. , This represents a two-dimensional convolutional layer with a kernel size of 3 and padding of 1. , , and These represent the outputs of the input gate, forget gate, output gate, and cell state update gate, respectively. It is the Sigmoid activation function. This indicates element-wise multiplication.

9. The method for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video according to claim 1, characterized in that, The specific process of step S4 is as follows: Step S41: Based on the segmentation results, obtain the polyp area, instrument area, polyp centroid, and instrument centroid. Then calculate the distance between the polyp centroid and the instrument centroid, the overlap area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the polyp area change rate, and the instrument speed; where: Step S42: Take the distance between the centroid of the polyp and the centroid of the instrument, the overlapping area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the rate of change of the polyp area, and the speed of the instrument as feature values. Set thresholds for the distance between the centroid of the polyp and the centroid of the instrument, the overlapping area between the polyp and the instrument, the percentage overlap between the polyp area and the instrument area, the rate of change of the polyp area, and the speed of the instrument, respectively. Compare each feature value of each frame of the image with the corresponding threshold. Whenever a feature value does not meet the threshold requirement, the risk score increases by 1. The cumulative value of the statistical risk score is used to classify the risk level based on the current cumulative risk score.

10. A system for polyp and surgical instrument segmentation and real-time early warning based on digestive endoscopy video, characterized in that, The system includes an image data preprocessing module, a collaborative segmentation network module, and a real-time risk warning module, wherein: The image data preprocessing module is used to acquire real digestive endoscopy surgical videos and generate virtual digestive endoscopy surgical videos. After the videos are processed by frame segmentation, they are manually annotated to finally construct a comprehensive instrument and polyp segmentation database containing real and virtual data. The collaborative segmentation network module is trained using the constructed database. The trained collaborative segmentation network module is used to achieve pixel-level collaborative segmentation of polyps and surgical instruments in the digestive endoscopy video to be processed, and outputs segmented images of polyps and instruments. The real-time risk warning module is used to perform risk assessment and real-time warning based on the segmentation results.

Images