A light-weight endoscopic polyp real-time segmentation method based on feature sparse sampling

Abstract

The present application relates to the technical field of medical image segmentation, and particularly relates to a light-weight endoscopic polyp real-time segmentation method based on feature sparse sampling, comprising: acquiring a data set, the data set comprising endoscopic polyp images; constructing a polyp segmentation model, the polyp segmentation model comprising an encoder and a decoder connected in sequence, the encoder comprising a feature extraction module and a feature post-processing module connected in sequence, the feature extraction module comprising a light-weight convolution backbone module and an efficient hybrid visual Transformer module, the light-weight convolution backbone module comprising a convolution module and a convolution layer, the efficient hybrid visual Transformer module comprising a convolution module and a sliding window dilated attention mechanism, and the decoder comprising an up-sampling; training the polyp segmentation model by using a training set and a validation set to obtain a trained polyp segmentation model; and inputting a test set into the trained polyp segmentation model to obtain a polyp detection result. The polyp segmentation model has high segmentation accuracy and greatly reduces the model complexity.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation technology, specifically to a lightweight real-time endoscopic polyp segmentation method based on feature sparse sampling. Background Technology

[0002] Colorectal cancer (CRC) is a prevalent malignant tumor worldwide, ranking among the top in both incidence and mortality. Accurate early detection and segmentation of colorectal polyps are crucial for preventing CRC progression. Endoscopic examination, as the gold standard for clinical diagnosis, is increasingly being incorporated into computer-aided diagnostic systems to improve detection efficiency and consistency. In recent years, deep learning-based medical image segmentation techniques, especially U-Net and its improved architectures, have become the mainstream method in polyp segmentation due to their superior performance. These models significantly improve the localization accuracy of polyp regions through encoder-decoder structures combined with multi-scale feature fusion strategies.

[0003] Despite significant progress in polyp segmentation methods based on Convolutional Neural Networks (CNNs) and Transformers, several limitations remain. First, traditional manual feature extraction methods struggle to handle the high diversity of polyps in morphology, color, and texture, resulting in insufficient generalization ability. Second, the scarcity of high-quality labeled data leads to overfitting under small sample conditions and decreased cross-device generalization performance. Furthermore, existing high-precision models generally rely on complex network structures (such as deep convolutions and multi-scale attention modules), resulting in large parameter counts and high computational costs, making real-time inference difficult on resource-constrained clinical equipment (such as portable endoscopy workstations). While some studies have attempted to reduce computational burden with lightweight backbone networks (such as MobileNet), this often comes at the cost of segmentation accuracy, particularly in small polyps or low-contrast regions. Finally, existing methods lack targeted optimization for endoscopic image characteristics (such as illumination variations and tissue deformation), and most studies neglect practical deployment requirements (such as memory usage and platform compatibility), making it difficult to integrate high-performance models into clinical workflows. Therefore, there is an urgent need for a polyp segmentation method that combines lightweight design, real-time performance, and high precision to meet the needs of real-world diagnostic and treatment scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems of high model complexity, poor real-time performance, difficulty in balancing accuracy and efficiency, and lack of clinical deployment support, this invention provides a lightweight real-time endoscopic polyp segmentation method based on sparse feature sampling. This invention primarily utilizes the fusion of a lightweight convolutional backbone module and a high-efficiency hybrid vision Transformer module, along with the construction of an endoscopic examination assistance platform, to achieve high-precision, low-latency, and deployable intelligent endoscopic assisted diagnosis.

[0005] The technical means employed in this invention are as follows: A lightweight real-time endoscopic polyp segmentation method based on feature sparse sampling includes the following steps: Obtain a dataset that includes endoscopic polyp images; The dataset is divided into a training set, a validation set, and a test set; A polyp segmentation model is constructed, comprising an encoder and a decoder connected in sequence. The encoder comprises a feature extraction module and a feature post-processing module connected in sequence. The feature extraction module comprises a lightweight convolutional backbone module and an efficient hybrid visual Transformer module. The lightweight convolutional backbone module comprises a convolutional module and a convolutional layer. The efficient hybrid visual Transformer module comprises a convolutional module and a sliding window dilation attention mechanism. The decoder comprises an upsampling module. The polyp segmentation model is trained using the training set and validation set to obtain a trained polyp segmentation model. The test set is input into the trained polyp segmentation model to obtain polyp detection results.

[0006] Furthermore, the network architecture of the polyp segmentation model includes an image preprocessing module, a feature extraction module, a feature post-processing module, an upsampling module, and a feature fusion module connected in sequence. The workflow of the polyp segmentation model includes: The endoscopic image is input into the image preprocessing module to obtain the preprocessed endoscopic image; The preprocessed endoscopic image is input into the feature extraction module to obtain multi-scale features and depth features; The depth features are input into the feature post-processing module to obtain post-processed features; The multi-scale features and post-processed features are passed through the upsampling module to obtain upsampled multi-scale features and upsampled post-processed features. The preprocessed endoscopic image, the upsampled multi-scale features, and the upsampled post-processed features are fused to obtain the polyp segmentation result.

[0007] Furthermore, the feature extraction module includes a first lightweight convolutional backbone module, a second lightweight convolutional backbone module, a third lightweight convolutional backbone module, and several lightweight hybrid Transformer modules connected in sequence. The lightweight hybrid Transformer module includes a fourth lightweight convolutional backbone module and an efficient hybrid vision Transformer module connected in sequence. The first lightweight convolutional backbone module, the second lightweight convolutional backbone module, the third lightweight convolutional backbone module, and the fourth lightweight convolutional backbone module have the same structure. The network architecture of the first lightweight convolutional backbone module includes a first convolutional module, a second convolutional module, and a first convolutional layer connected in sequence. The network architecture of the first convolutional module includes a second convolutional layer, a first batch normalization layer, and a SiLU activation layer connected in sequence. The second convolutional module includes a third convolutional layer, a second batch normalization layer, and a SiLU activation layer connected in sequence. The kernel size of the second convolutional layer is 1×1, the kernel size of the third convolutional layer is 3×3, and the kernel size of the first convolutional layer is 1×1. The efficient hybrid vision Transformer module includes a third convolutional module, a fourth convolutional module, a sliding window dilation attention mechanism, a fifth convolutional module, and a sixth convolutional module connected in sequence. The third and fifth convolutional modules have the same network architecture. The third convolutional module includes a fourth convolutional layer, a third batch normalization layer, and a SiLU activation layer connected in sequence. The fourth and sixth convolutional modules have the same network architecture. The fourth convolutional module includes a fifth convolutional layer, a fourth batch normalization layer, and a SiLU activation layer connected in sequence. The kernel size of the fourth convolutional layer is 3×3, and the kernel size of the fifth convolutional layer is 1×1.

[0008] Furthermore, the sliding window dilation attention mechanism is an improvement on the multi-head self-attention mechanism in the visual Transformer. The improvement is that, within the local sliding window, the spatial response of the input features is sparsely sampled based on linear dilation convolution, and the contextual information of key positions is retained to participate in the attention calculation.

[0009] Furthermore, the feature post-processing module includes a sixth convolutional layer, a fifth batch normalization layer, and a SiLU activation layer connected in sequence, with the kernel size of the sixth convolutional layer being 3×3.

[0010] Furthermore, the upsampling module is a step-by-step upsampling structure, which uses transposed convolution to amplify the post-processed features, and then concatenates the amplified features with the outputs of the lightweight convolution backbone module and the efficient hybrid vision Transformer module to achieve skip connections. Residual connections are introduced in the skip connections.

[0011] Furthermore, the polyp segmentation model employs a fusion loss function, the calculation formula of which is:

[0012] in, L To describe the fusion loss function, For Dice's loss, Let be the cross-entropy loss, y be the true label, and p be the model's predicted probability output. This is a smoothing factor.

[0013] Furthermore, the polyp segmentation model is deployed in an endoscopy-assisted examination platform, which includes: The user interface module provides real-time endoscopic examination, endoscopic video retrieval, and local video analysis functions, and is implemented through a web page. The data preprocessing module is used to perform quality control and standardization on raw frames from cameras or local videos, including a fast blurred frame detection algorithm based on frequency domain analysis, and bilinear interpolation size normalization processing for valid frames. The segmentation module is used to load and run the polyp segmentation model, supporting real-time segmentation and batch video segmentation, and outputting a binary segmentation mask; The record management module is used to realize the structured and persistent storage and efficient retrieval of examination data, including records of patient unique identifiers, examination timestamps, original video file paths, and segmentation mask sequences, as well as support for quick querying and visual playback by fields such as patient ID and examination date range.

[0014] Furthermore, the user interface module includes a real-time inspection module, a video retrieval module, and a local analysis module. The real-time examination module is responsible for interacting with the doctor, displaying a real-time video stream via a USB endoscope camera, starting the video stream by clicking the start button, and controlling the examination process by clicking the save and stop buttons. The video retrieval module is used to retrieve historical examination records from the platform database by patient ID, examination date, or time period. The local analysis module is used to process the video frame by frame, generate segmentation results, and display them visually.

[0015] Furthermore, the workflow of the endoscopic examination assistance platform includes: Receives video streams or local video files from an endoscope camera or system as input; The system employs a multi-threaded approach to acquire each frame of the input video stream, converts RGB images to grayscale images, performs a Fast Fourier Transform (FFT) to map the image to the frequency domain, centers the spectrum to suppress low-frequency components and highlight high-frequency details, and then reconstructs the image through an Inverse Fast Fourier Transform (IFT). The average value of the high-frequency amplitude in the reconstructed image is calculated and compared with a preset threshold. Based on the comparison between the average value of the high-frequency amplitude and the preset threshold, it determines whether the frame is a blurry frame and selects to skip the frame or reduce its weight according to the system configuration. All valid frames are adjusted to a size of 256×256 using bilinear interpolation. The pre-processed valid video frames are input into the lightweight polyp segmentation model for inference. The lightweight polyp segmentation model outputs a binary segmentation mask, where 0 represents the background and 1 represents the polyp region. The binary segmentation mask is superimposed on the original video frame in real time for display. The system writes the patient's unique identifier, examination timestamp, original video file path, and corresponding segmentation mask sequence into a local SQLite database. It also supports quick queries by fields such as patient ID and examination date range, and allows for visual playback in the user interface by overlaying the original video with the segmentation mask.

[0016] Compared with the prior art, the present invention has the following advantages: 1. This invention, within the U-shaped encoder-decoder (U-Net) framework, employs a lightweight convolutional backbone module and an efficient hybrid vision Transformer module, stacked alternately, to construct the encoder. The efficient hybrid vision Transformer module introduces a sliding window dilation attention mechanism, calculating self-attention within a local sliding window and expanding the receptive field through dilation operations. This significantly enhances the model's ability to model the global context of polyps with only a slight increase in computational overhead. This design keeps the model parameter count at 5.798 M and FLOPs at 3.609 G, significantly lower than U-Net (34.53 M model parameters) and U-Net++ (36.63 M model parameters, 1380 G FLOPs).

[0017] 2. This invention addresses the common motion blur problem in endoscopic videos by employing Fast Fourier Transform (FFT) to convert the image to the frequency domain. It then suppresses low-frequency components, preserves high-frequency components, and reconstructs the image, calculating the mean amplitude of the high-frequency components as a blur criterion. This method has low computational cost and can be used as a preprocessing step to filter low-quality frames in real time, improving the stability of subsequent segmentation.

[0018] 3. The platform of this invention employs four threads for parallel processing: each thread is responsible for video frame acquisition, preprocessing, model inference, and result rendering, with each stage connected via a queue buffer. This architecture effectively alleviates I / O bottlenecks, achieving an inference speed of 94.64 FPS on an NVIDIA RTX 3090 GPU, meeting the basic clinical requirements for real-time performance (≥25 FPS).

[0019] 4. This invention is based on a web platform built on Streamlit, supporting three main functions: real-time camera segmentation, local video batch analysis, and historical record retrieval. All examination data (including raw videos and segmentation masks) are structured and stored in an SQLite database, supporting retrieval by patient ID or time and visual playback, facilitating clinical review and follow-up.

[0020] 5. The model of this invention achieves Dice similarity coefficients of 92.56% and 92.45% on the Kvasir-SEG and CVC-ClinicDB datasets, respectively, which is better than the lightweight methods compared. At the same time, it can still output reasonable segmentation results in real-world scenarios with staining agents and instrument occlusion, indicating that it has a certain degree of robustness.

[0021] Based on the above reasons, this invention can be widely applied in fields such as medical image segmentation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to the present invention.

[0024] Figure 2 This is an architectural diagram of the endoscopic examination auxiliary platform of the present invention.

[0025] Figure 3 This is a schematic diagram of the workflow of the endoscopic examination auxiliary platform of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] The purpose of this invention is to address the key shortcomings of existing endoscopic polyp segmentation methods in clinical applications, such as high model complexity, poor real-time performance, difficulty in balancing accuracy and efficiency, insufficient adaptability to endoscopic image characteristics, weak generalization ability with small samples, and lack of systematic functional support for real-world diagnostic and treatment scenarios. The invention aims to provide a lightweight, efficient, robust, and fully clinically supportive real-time endoscopic polyp segmentation and auxiliary diagnosis platform.

[0029] Specifically, the present invention aims to: (1) Design a lightweight segmentation network optimized for endoscopic images, which significantly reduces the number of model parameters and computational complexity while maintaining high segmentation accuracy for small polyps, blurred boundaries and complex backgrounds.

[0030] (2) Achieve the clinically required real-time inference speed (≥25 FPS) on low-power devices and support smooth processing of continuous endoscopic video streams.

[0031] (3) Enhance the model’s generalization ability on small-scale labeled data through structural or training strategies, and improve its robustness on images collected from different devices and centers.

[0032] (4) Construct a modular and scalable interactive auxiliary diagnostic platform that integrates functions such as real-time polyp localization and segmentation, result visualization, medical record storage and retrospection, and local video upload and analysis, so as to effectively reduce the workload of doctors and improve the efficiency and consistency of endoscopic examinations.

[0033] Through the aforementioned technical means, this invention aims to bridge the gap between high-precision algorithms and clinically usable systems, and promote the true implementation and popularization of artificial intelligence technology in endoscopic diagnosis and treatment.

[0034] The technical solution of this invention includes the following core components: (1) Lightweight polyp segmentation model: Based on the improved U-shaped encoder-decoder structure, a lightweight convolutional backbone module and the proposed efficient hybrid vision Transformer module are used to hybrid encoder, combined with the sliding window dilation attention mechanism, which significantly reduces the number of parameters and computation while taking into account the local details and global semantic modeling capabilities. (2) Real-time endoscopic examination auxiliary platform: a web application system built on the Streamlit framework, integrating user interface, data preprocessing module, segmentation and inference module and record management module, supporting real-time video stream processing, local video analysis, result backtracking and storage; (3) Data preprocessing mechanism: A fuzzy frame detection algorithm based on Fast Fourier Transform (FFT) is introduced to screen the quality of the input video frames, thereby improving the consistency of the model input and the robustness of the segmentation; (4) Multi-threaded parallel processing architecture: OpenCV and multi-threading technology are used to realize the pipeline parallelism of video frame acquisition, preprocessing, inference and rendering, ensuring high frame rate real-time response.

[0035] like Figure 1 As shown, this invention provides a lightweight real-time endoscopic polyp segmentation method based on feature sparse sampling, comprising the following steps: S1. Obtain the dataset, which includes endoscopic polyp images.

[0036] S2. Divide the dataset into training set, validation set and test set.

[0037] S21. Dataset Acquisition and Partitioning: This invention utilizes two widely recognized and representative publicly available endoscopic polyp segmentation datasets: Kvasir-SEG and CVC-ClinicDB, to ensure the reliability and generalization of model training and evaluation. The Kvasir-SEG dataset contains 1000 pairs of high-quality endoscopic images and their corresponding expert-level pixel-level segmentation masks (ground truth). The images are derived from real clinical colonoscopy examinations, with resolutions ranging from 332×487 to 1920×1072, covering the diverse appearance of polyps under different sizes, shapes, colors, lighting conditions, and background interference, and possessing strong clinical representativeness. The CVC-ClinicDB dataset consists of 612 pairs of professionally annotated polyp images, all with a uniform resolution of 384×288, acquired and annotated by clinicians using standard endoscopic equipment, primarily used to evaluate the model's segmentation performance in high-contrast, clear-boundary scenes.

[0038] To scientifically and robustly evaluate model performance, this invention employs a standard 5-fold cross-validation strategy for both datasets. Specifically, all samples in each dataset are randomly divided into five mutually exclusive subsets ("folds"). Each time, four folds (approximately 80% in total) are used as the training and validation sets, and the remaining fold (approximately 20%) is used as the test set. Further, during the training process of each fold, 80% of the samples are divided into a training set (60%) and a validation set (20%) in a 6:2 ratio for model training and hyperparameter tuning. After five rounds of independent training and testing, the final report shows the model's average performance metrics (Dice similarity coefficient, IoU, etc.) across all test samples.

[0039] This partitioning strategy effectively avoids evaluation bias caused by a single random partition, significantly improving the stability and repeatability of model performance evaluation, and is especially suitable for scenarios with limited medical image samples.

[0040] S22. Data Preprocessing and Augmentation: To improve the generalization ability of the model and adapt to the input requirements of lightweight networks, this invention designs targeted data processing procedures for the training and testing phases respectively.

[0041] (1) Data processing during the training phase. All input images undergo the following preprocessing and enhancement operations before being fed into the network: Pixel value normalization: The pixel values ​​of the RGB channels of the image are linearly scaled from [0, 255] to the range of [0, 1] to accelerate model convergence.

[0042] Random space enhancement: includes random rotation (probability 0.25, angle range 90° to 270°) and random cropping (cropped to a fixed size of 256×256).

[0043] The above enhancement strategies significantly expand the diversity of training samples without introducing additional annotation costs, effectively alleviating the overfitting problem caused by the limited scale of labeled data.

[0044] (2) Testing / Inference Phase Processing. To balance inference efficiency and segmentation accuracy, a sliding window inference strategy is adopted in the testing phase. For input images of any size, they are divided into multiple overlapping windows of 256×256 to avoid loss of edge information. Each window is independently fed into the model for forward inference. The inference results are weighted and fused (averaging of overlapping regions) to generate a full-size segmentation mask. The final output is a binary segmentation map with the same resolution as the original image, which can be used for platform visualization or subsequent analysis. This strategy effectively supports high-precision segmentation of images of any resolution while maintaining the model's lightweight nature (fixed input size), improving the model's adaptability in real clinical scenarios.

[0045] S3. Construct a polyp segmentation model. The polyp segmentation model includes an encoder and a decoder connected in sequence. The encoder includes a feature extraction module and a feature post-processing module connected in sequence. The feature extraction module includes a lightweight convolutional backbone module and an efficient hybrid visual Transformer module. The lightweight convolutional backbone module includes a convolutional module and convolutional layers. The efficient hybrid visual Transformer module includes a convolutional module and a sliding window dilation attention mechanism. The decoder includes an upsampling module.

[0046] The polyp segmentation model proposed in this invention adopts a U-shaped encoder-decoder structure, and its core innovation lies in the lightweight design of the encoder and the feature fusion mechanism of the decoder.

[0047] Specifically, the network architecture of the polyp segmentation model includes an image preprocessing module, a feature extraction module, a feature postprocessing module, an upsampling module, and a feature fusion module connected in sequence.

[0048] The workflow of the polyp segmentation model includes: The first step is to input the endoscopic image into the image preprocessing module to obtain the preprocessed endoscopic image.

[0049] The second step is to input the preprocessed endoscopic image into the feature extraction module to obtain multi-scale features and depth features.

[0050] The third step is to input the depth features into the feature post-processing module to obtain the post-processed features.

[0051] The fourth step involves passing the multi-scale features and post-processed features through an upsampling module to obtain upsampled multi-scale features and upsampled post-processed features.

[0052] The fifth step involves fusing the preprocessed endoscopic image, the upsampled multi-scale features, and the upsampled post-processed features to obtain the polyp segmentation result.

[0053] The encoder of this invention mainly consists of alternating stacked lightweight convolutional backbone modules and efficient hybrid vision Transformer modules. The lightweight convolutional backbone modules are used to efficiently extract low-level spatial features (such as edges and textures), further reducing the computational overhead of the model by using depthwise separable convolutional structures. The efficient hybrid vision Transformer module is used to capture high-level semantic information, and its structure integrates a lightweight Transformer design with a sliding window dilation attention mechanism. Specifically, the sliding window dilation attention mechanism is embedded between the convolutional layers of the efficient hybrid vision Transformer module, calculating self-attention within a local sliding window and expanding the receptive field through dilation operations, thereby enhancing the model's ability to model the global context of polyps without significantly increasing FLOPs.

[0054] The preprocessing module performs preliminary feature extraction on the input image. This module mainly consists of a 3×3 convolutional layer, a batch normalization layer, and a SiLU activation layer. This module reduces the dimensionality of the input image to half of its original value and increases the feature channels from 3 to 16.

[0055] The feature extraction module comprises a first lightweight convolutional backbone module, a second lightweight convolutional backbone module, a third lightweight convolutional backbone module, and several connected lightweight hybrid Transformer modules. The lightweight hybrid Transformer modules include a fourth lightweight convolutional backbone module and a high-efficiency hybrid vision Transformer module, all connected in sequence. The first, second, and third lightweight convolutional backbone modules extract shallow and local information from the image. The processed feature resolution is further reduced to 1 / 4 of the input image, while the feature channels are increased to 64. Subsequently, the high-efficiency hybrid vision Transformer module and the lightweight convolutional backbone modules combine to process the features three times consecutively, obtaining deep features that incorporate global information. The resolution of these features is further reduced to 1 / 32 of the input image, and the feature channels are increased to 160.

[0056] The first, second, third, and fourth lightweight convolutional backbone modules have the same structure. The network architecture of the first lightweight convolutional backbone module includes a first convolutional module, a second convolutional module, and a first convolutional layer connected in sequence. The network architecture of the first convolutional module includes a second convolutional layer, a first batch of normalization layers, and a SiLU activation layer connected in sequence. The second convolutional module includes a third convolutional layer, a second batch of normalization layers, and a SiLU activation layer connected in sequence. The kernel size of the second convolutional layer is 1×1, the kernel size of the third convolutional layer is 3×3, and the kernel size of the first convolutional layer is 1×1.

[0057] The efficient hybrid vision Transformer module includes a third convolutional module, a fourth convolutional module, a sliding window dilation attention mechanism, a fifth convolutional module, and a sixth convolutional module connected in sequence. The third and fifth convolutional modules have the same network architecture. The third convolutional module includes a fourth convolutional layer, a third batch normalization layer, and a SiLU activation layer connected in sequence. The fourth and sixth convolutional modules have the same network architecture. The fourth convolutional module includes a fifth convolutional layer, a fourth batch normalization layer, and a SiLU activation layer connected in sequence. The kernel size of the fourth convolutional layer is 3×3, and the kernel size of the fifth convolutional layer is 1×1.

[0058] The sliding window dilated attention mechanism is a lightweight improvement upon the multi-head self-attention mechanism in the visual Transformer. The improvement lies in the sparse sampling of the spatial response of the input features within a local sliding window using linear dilated convolution, preserving contextual information at key locations for attention computation. This strategy significantly reduces the computational complexity of self-attention operations while expanding the effective receptive field through dilation. This allows the model to capture the long-range dependencies and structural features of irregular polyps more effectively while maintaining low computational overhead, thereby improving segmentation performance.

[0059] Locality refers to a fixed-size neighborhood (3×3) on the input feature map covered by the sliding window, centered at the current computation position. This neighborhood is uniquely determined by the size of the sliding window and is the scope of subsequent sparse sampling and attention computation.

[0060] The feature post-processing module consists of a sixth convolutional layer, a fifth batch normalization layer, and a SiLU activation layer connected in sequence. The kernel size of the sixth convolutional layer is 3×3. After multiple repetitions of processing by combining the efficient hybrid vision Transformer module with the lightweight convolutional backbone module, this module further processes the deep features to supplement additional local information and increases the channel dimension of the features to 640.

[0061] The upsampling module employs a stepwise upsampling structure, using transposed convolutions to amplify post-processed features. These amplified features are then concatenated with the outputs of the lightweight convolutional backbone module and the efficient hybrid vision Transformer module to achieve skip connections. Residual connections are introduced into these skip connections to mitigate gradient vanishing, stabilize the training process, and facilitate the efficient transfer of spatial details from the encoder to the decoder.

[0062] The model of this invention has only 5.798 M parameters and 3.609 G FLOPs with a 256×256 image input, significantly lower than U-Net (34.53 M model parameters, 65.447 G FLOPs) and U-Net++ (36.63 M model parameters, 1380 G FLOPs). It achieves an inference speed of 94.64 FPS on an NVIDIA 3090 GPU, meeting the clinical real-time requirement of ≥25 FPS.

[0063] The polyp segmentation model uses a fusion loss function, which is calculated using the following formula:

[0064] in, L To describe the fusion loss function, This is called Dice Loss. This is the Cross Entropy Loss, where y is the true label and p is the model's predicted probability output. The smoothing factor is set to the default value. This fusion loss function incorporates supervision for both segmentation accuracy and classification, effectively optimizing the model's training parameters.

[0065] S4. Train the polyp segmentation model using the training set and validation set to obtain a well-trained polyp segmentation model.

[0066] Specifically, during the model training phase, the input data undergoes real-time processing and augmentation using the aforementioned pre-defined data preprocessing and augmentation strategies, including pixel normalization, random rotation, and center cropping. This strategy effectively expands sample diversity without increasing annotation costs, significantly improving the model's generalization ability and segmentation performance with limited data scales. The model employs the AdamW optimizer (a weight decay version of the adaptive moment estimation optimizer) for parameter optimization, coupled with a dynamic learning rate adjustment strategy. Initially, 100 warm-up epochs are set to stabilize the initial training process; subsequently, the learning rate decay coefficient is set to 0.25 to suppress overfitting. The total number of training epochs is set to 2000, the batch size is set to 16, and training is performed from scratch without loading any pre-trained weights.

[0067] During the model validation phase, this invention trains the model on two public datasets, Kvasir-SEG and CVC-ClinicDB, and monitors its performance on their respective validation sets. After each training round, the optimal model weights are selected based on the segmentation metrics (such as the Dice coefficient) on the validation set for subsequent testing. This mechanism effectively achieves dynamic supervision of the training process, prevents overfitting, and ensures that the selected weights have the best generalization ability.

[0068] S5. Input the test set into the trained polyp segmentation model to obtain polyp detection results.

[0069] In the model testing phase, inference is performed on the test set based on the optimal weights of the validation set, and the model's performance is evaluated. This invention selects mainstream classification and segmentation evaluation metrics: Dice coefficient and Intersection over Union (IoU). Simultaneously, to evaluate the model's complexity and inference speed, this invention selects model parameters, FLOPs, and FPS as evaluation metrics for this stage.

[0070] like Figure 2As shown, this embodiment of the invention also includes an endoscopy examination assistance platform. The polyp segmentation model is deployed within this platform. The Real-time Endoscopy Examination Assisted Platform (REAP) constructed by this invention adopts a modular web architecture, developed based on the open-source Python web framework Streamlit, supports cross-operating system deployment (Windows / Linux / macOS), and can be accessed through a standard browser without the need for a dedicated client. The platform's modules have clearly defined responsibilities and loosely coupled interfaces, facilitating subsequent functional expansion and system maintenance.

[0071] Endoscopic examination support platforms include: The user interface module provides real-time endoscopic examination, endoscopic video retrieval, and local video analysis capabilities, all implemented via a web page. This user interface module serves as the sole entry point for interaction between the platform and physicians.

[0072] The data preprocessing module is used to perform quality control and standardization on raw frames from cameras or local videos, including a fast blurred frame detection algorithm based on frequency domain analysis, and bilinear interpolation size normalization processing on valid frames.

[0073] The segmentation module is used to load and run the polyp segmentation model, supporting real-time segmentation and batch video segmentation, and outputting a binary segmentation mask.

[0074] The record management module is used to realize the structured and persistent storage and efficient retrieval of examination data, including records of patient unique identifiers, examination timestamps, original video file paths, and segmentation mask sequences, as well as support for quick querying and visual playback by fields such as patient ID and examination date range.

[0075] Specifically, the data preprocessing module is responsible for quality control and standardization of raw frames from cameras or local videos to ensure that the input meets the inference requirements of the lightweight segmentation model. Addressing the common motion blur problem in endoscopic examinations (caused by instrument movement, intestinal peristalsis, or body fluid interference), this invention employs a fast blurred frame detection algorithm based on frequency domain analysis: First, RGB frames are converted to grayscale images, then mapped to the frequency domain using Fast Fourier Transform (FFT); the spectrum is centered to suppress low-frequency components and highlight high-frequency details, and then the image is reconstructed using inverse FFT; finally, the average value of high-frequency amplitudes in the reconstructed image is calculated and compared with a preset threshold. If the value is below the threshold, it is determined to be a blurred frame, and the system configuration can be used to skip the frame or reduce its weight, thus avoiding low-quality input affecting segmentation accuracy. All valid frames are uniformly adjusted to 256×256 pixels using bilinear interpolation before being fed into the model to ensure input consistency. To improve overall throughput efficiency, the module adopts a multi-threaded pipeline architecture: independent threads are responsible for video frame acquisition, preprocessing (blur detection + size normalization), model inference and front-end rendering. Each thread communicates through a queue buffer, effectively avoiding I / O blocking, maximizing the use of CPU / GPU resources, and supporting high frame rate real-time processing.

[0076] The segmentation module, as the core intelligent engine of the platform, loads and runs the lightweight polyp segmentation model proposed in this invention, supporting two inference modes to adapt to different clinical scenarios. In real-time camera segmentation mode, the system performs forward inference frame by frame on the preprocessed valid video frames, outputting a binary segmentation mask of the same size (0 represents the background, 1 represents the polyp region). The inference latency is controlled at the millisecond level, and the measured frame rate on an NVIDIA RTX 3090 GPU reaches 94.64 FPS, meeting the requirements for real-time assistance during surgery. In video batch segmentation mode, the platform decodes and infers frame by frame on the complete endoscopic video uploaded by the user, and generates a complete segmentation sequence aligned with the timestamp of the original frame, forming a structured analysis report. The model achieves Dice similarity coefficients (DSC) of 92.56% and 92.45% on the Kvasir-SEG and CVC-ClinicDB standard datasets, respectively, and can still accurately segment lesion regions in complex clinical scenarios containing stains (such as indigo carmine) or surgical instruments, fully verifying its high robustness in real-world diagnostic and treatment environments.

[0077] The record management module enables structured, persistent storage and efficient retrieval of examination data. After each real-time examination or local video analysis task is completed, the system automatically writes the patient's unique identifier (ID), examination timestamp, original video file path (or video stream segment), and corresponding segmentation mask sequence to a local SQLite database (or can be configured to use a network database such as MySQL). All records support quick querying by fields such as patient ID and examination date range, and can be visualized and replayed in the user interface as an overlay of the original video and segmentation mask. This not only facilitates doctors' review of diagnostic results but also provides reliable and traceable data support for subsequent follow-up, teaching demonstrations, and research analysis.

[0078] Furthermore, the user interface module includes a real-time inspection module, a video retrieval module, and a local analysis module.

[0079] The real-time examination module is responsible for interacting with the doctor, displaying a real-time video stream via a USB endoscope camera, starting the video stream by clicking the start button, and controlling the examination process by clicking the save and stop buttons.

[0080] Specifically, once the real-time endoscopy function is activated, it automatically scans and lists all available video acquisition devices (such as USB endoscope cameras) on the current host. The doctor can select the target camera from the device list and start the real-time video stream by clicking the "Start" button. The platform simultaneously performs polyp segmentation and renders the generated binary mask on the original image in real-time using a semi-transparent overlay. During the examination, the doctor can click the "Save" button at any time to package and store the current video segment and its corresponding segmentation results. After the examination, clicking the "Stop" button terminates video acquisition.

[0081] The video retrieval module is used to retrieve historical examination records from the platform database by patient ID, examination date, or time period.

[0082] Specifically, once the endoscopic video retrieval function is activated, doctors can retrieve historical examination records from the platform's database by entering the patient ID, examination date, or time period. The search results are displayed as a time list, supporting playback of the original endoscopic video with simultaneous overlay of historical segmentation masks, facilitating disease tracking and treatment evaluation.

[0083] The local analysis module is used to process the video frame by frame, generate segmentation results, and visualize them.

[0084] Specifically, after the local video analysis function is activated, the platform supports uploading locally stored endoscopic video files (such as MP4, AVI, and other common formats). Once uploaded, the system automatically calls the segmentation module to process the video frame by frame, generating a complete segmentation sequence, and storing the results along with the original video in the record database. After analysis, the results are directly visualized in the right-hand area of ​​the interface and can be retrieved later through the video search function, achieving unified management of offline analysis and online examination data.

[0085] In the real-time endoscopic examination assistance platform constructed by this invention, each functional module achieves efficient collaboration and orderly flow through clearly defined data interfaces.

[0086] When a doctor initiates a real-time endoscopy or local video analysis function through the user interface module, the raw video frames (from camera capture or local file decoding) are first passed from the user interface module to the data preprocessing module. This module performs blur detection and size normalization processing based on frequency domain analysis on each frame to generate a standardized image, which is then sent to the segmentation module.

[0087] The segmentation module loads a lightweight polyp segmentation model, performs forward inference on the standardized image frame by frame, and outputs a binary segmentation mask of the same size. This mask is then returned to the user interface module and rendered on the original image in real time in a semi-transparent overlay manner for doctors to observe intuitively.

[0088] After the inspection or analysis task is completed, the user interface module packages and submits the original video clips, the corresponding segmentation mask sequence, patient ID, timestamps and other metadata to the record management module; the latter is responsible for persistently storing this structured data in an SQLite database.

[0089] When doctors use the endoscopy video retrieval function later, the record management module extracts the corresponding records from the database according to the query conditions and returns the original video path and segmentation mask sequence to the user interface module to realize the visual playback of historical results.

[0090] The entire system forms a closed-loop data flow of "acquisition → preprocessing → inference → visualization / storage → retrieval and playback". The modules communicate with each other through asynchronous queues and standardized interfaces to ensure high throughput and low latency real-time performance, while ensuring data consistency, traceability and system scalability.

[0091] like Figure 3 As shown, the platform's workflow is as follows: The first step is to receive video streams or local video files from the endoscope camera or system as input.

[0092] The second step involves acquiring each frame of the input video stream using a multi-threaded approach, converting the RGB image to a grayscale image, and performing a Fast Fourier Transform (FFT) to map the image to the frequency domain. The spectrum is then centered to suppress low-frequency components and highlight high-frequency details. The image is then reconstructed using an inverse FFT, and the average high-frequency amplitude in the reconstructed image is calculated and compared with a preset threshold. Based on the comparison between the average high-frequency amplitude and the preset threshold, it is determined whether the frame is a blurry frame. Depending on the system configuration, the frame is either skipped or downweighted. All valid frames are then adjusted to a size of 256×256 using bilinear interpolation.

[0093] The third step involves inputting the preprocessed valid video frames into the lightweight polyp segmentation model for inference. The lightweight polyp segmentation model outputs a binary segmentation mask, where 0 represents the background and 1 represents the polyp region. The binary segmentation mask is then overlaid on the original video frame in real time for display.

[0094] The fourth step involves writing the patient's unique identifier, examination timestamp, original video file path, and corresponding segmentation mask sequence into a local SQLite database. It also supports quick queries by fields such as patient ID and examination date range, and allows for visual playback in the user interface by overlaying the original video with the segmentation mask.

[0095] This invention achieves quantifiable and verifiable technical results in three dimensions: computational efficiency, segmentation accuracy, and system usability, through the synergistic implementation of lightweight model design and a modular real-time platform. The following description focuses on model performance, system efficiency, and clinical applicability, and compares these results with typical existing methods.

[0096] (1) The model is lightweight and the computational efficiency is significantly improved. The polyp segmentation model proposed in this invention significantly reduces model complexity while maintaining high segmentation accuracy. Using the THOP tool (based on PyTorch-OpCounter) to evaluate a 256×256 image as input, the model has 5.798 M parameters and a computational cost (FLOPs) of 3.609 G. Compared with mainstream methods (Table 1), the advantages are obvious: Table 1. Quantitative comparison results of the model of this invention with other comparative networks in terms of model complexity.

[0097] (2) The segmentation accuracy remains at a high level and it has good robustness. The model was evaluated using five-fold cross-validation on two standard datasets. The Dice similarity coefficient (DSC) and intersection-over-union ratio (IoU) of this model are compared with those of mainstream methods as follows (Table 2): Table 2. Quantitative comparison results of the segmentation accuracy between this patented model and other comparative networks.

[0098] (3) Enhanced system usability and clinical workflow support capabilities This invention constructs a complete real-time endoscopy-assisted platform that not only provides segmentation algorithms but also closely aligns with actual clinical needs: the platform's end-to-end processing latency is less than 30ms (>33 FPS), achieving an acceptable real-time level for endoscopic examinations; it supports both intraoperative real-time assistance and postoperative video retrospective analysis modes, improving system efficiency; by automatically saving patient IDs, original videos, segmentation masks, and quantitative indicators (such as Dice and IoU), it reduces the burden of manual recording for physicians and supports one-click retrieval of historical cases by patient or time, facilitating follow-up and comparison; the entire system is developed based on Streamlit, requiring no dedicated client installation, and the model size is less than 25 MB, making it easy to integrate into existing endoscopic equipment or workstations.

[0099] (4) Resource saving and scalability Thanks to the lightweight model and efficient platform design, this invention has significant advantages in resource utilization: under the same hardware conditions, compared with heavy models with large parameters and high computational cost, it can support continuous video processing for longer periods or higher concurrency tasks; data storage adopts a structured approach, retaining only the original video keyframes, segmentation masks, and metadata, effectively avoiding redundant storage of the entire video; the platform adopts a modular architecture, which can easily integrate new functional modules (such as upper gastrointestinal lesion detection, endoscopic image quality scoring, etc.) in the future without reconstructing the overall system, and has good maintainability and expansion potential.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight real-time endoscopic polyp segmentation method based on feature sparse sampling, characterized in that, Includes the following steps: Obtain a dataset that includes endoscopic polyp images; The dataset is divided into a training set, a validation set, and a test set; A polyp segmentation model is constructed, comprising an encoder and a decoder connected in sequence. The encoder comprises a feature extraction module and a feature post-processing module connected in sequence. The feature extraction module comprises a lightweight convolutional backbone module and an efficient hybrid visual Transformer module. The lightweight convolutional backbone module comprises a convolutional module and a convolutional layer. The efficient hybrid visual Transformer module comprises a convolutional module and a sliding window dilation attention mechanism. The decoder comprises an upsampling module. The polyp segmentation model is trained using the training set and validation set to obtain a trained polyp segmentation model. The test set is input into the trained polyp segmentation model to obtain polyp detection results.

2. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 1, characterized in that, The network architecture of the polyp segmentation model includes an image preprocessing module, a feature extraction module, a feature postprocessing module, an upsampling module, and a feature fusion module connected in sequence. The workflow of the polyp segmentation model includes: The endoscopic image is input into the image preprocessing module to obtain the preprocessed endoscopic image; The preprocessed endoscopic image is input into the feature extraction module to obtain multi-scale features and depth features; The depth features are input into the feature post-processing module to obtain post-processed features; The multi-scale features and post-processed features are passed through the upsampling module to obtain upsampled multi-scale features and upsampled post-processed features. The preprocessed endoscopic image, the upsampled multi-scale features, and the upsampled post-processed features are fused to obtain the polyp segmentation result.

3. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 1, characterized in that, The feature extraction module includes a first lightweight convolutional backbone module, a second lightweight convolutional backbone module, a third lightweight convolutional backbone module, and several lightweight hybrid Transformer modules connected in sequence. The lightweight hybrid Transformer module includes a fourth lightweight convolutional backbone module and an efficient hybrid vision Transformer module connected in sequence. The first lightweight convolutional backbone module, the second lightweight convolutional backbone module, the third lightweight convolutional backbone module, and the fourth lightweight convolutional backbone module have the same structure. The network architecture of the first lightweight convolutional backbone module includes a first convolutional module, a second convolutional module, and a first convolutional layer connected in sequence. The network architecture of the first convolutional module includes a second convolutional layer, a first batch normalization layer, and a SiLU activation layer connected in sequence. The second convolutional module includes a third convolutional layer, a second batch normalization layer, and a SiLU activation layer connected in sequence. The kernel size of the second convolutional layer is 1×1, the kernel size of the third convolutional layer is 3×3, and the kernel size of the first convolutional layer is 1×1. The efficient hybrid vision Transformer module includes a third convolutional module, a fourth convolutional module, a sliding window dilation attention mechanism, a fifth convolutional module, and a sixth convolutional module connected in sequence. The third and fifth convolutional modules have the same network architecture. The third convolutional module includes a fourth convolutional layer, a third batch normalization layer, and a SiLU activation layer connected in sequence. The fourth and sixth convolutional modules have the same network architecture. The fourth convolutional module includes a fifth convolutional layer, a fourth batch normalization layer, and a SiLU activation layer connected in sequence. The kernel size of the fourth convolutional layer is 3×3, and the kernel size of the fifth convolutional layer is 1×1.

4. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 3, characterized in that, The sliding window dilation attention mechanism is an improvement on the multi-head self-attention mechanism in the visual Transformer. The improvement is that, within the local sliding window, the spatial response of the input features is sparsely sampled based on linear dilation convolution, and the contextual information of key positions is retained to participate in the attention calculation.

5. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 1, characterized in that, The feature post-processing module includes a sixth convolutional layer, a fifth batch normalization layer, and a SiLU activation layer connected in sequence. The kernel size of the sixth convolutional layer is 3×3.

6. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 1, characterized in that, The upsampling module is a stepwise upsampling structure. It uses transposed convolution to amplify the post-processed features. The amplified features are then concatenated with the outputs of the lightweight convolutional backbone module and the efficient hybrid vision Transformer module to achieve skip connections. Residual connections are introduced in the skip connections.

7. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 1, characterized in that, The polyp segmentation model employs a fusion loss function, the calculation formula of which is as follows: in, L To describe the fusion loss function, For Dice's loss, Let be the cross-entropy loss, y be the true label, and p be the model's predicted probability output. This is a smoothing factor.

8. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 1, characterized in that, The polyp segmentation model is deployed in an endoscopy-assisted platform, which includes: The user interface module provides real-time endoscopic examination, endoscopic video retrieval, and local video analysis functions, and is implemented through a web page. The data preprocessing module is used to perform quality control and standardization on raw frames from cameras or local videos, including a fast blurred frame detection algorithm based on frequency domain analysis, and bilinear interpolation size normalization processing for valid frames. The segmentation module is used to load and run the polyp segmentation model, supporting real-time segmentation and batch video segmentation, and outputting a binary segmentation mask; The record management module is used to realize the structured and persistent storage and efficient retrieval of examination data, including records of patient unique identifiers, examination timestamps, original video file paths, and segmentation mask sequences, as well as support for quick querying and visual playback by fields such as patient ID and examination date range.

9. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 8, characterized in that, The user interface module includes a real-time inspection module, a video retrieval module, and a local analysis module. The real-time examination module is responsible for interacting with the doctor, displaying a real-time video stream via a USB endoscope camera, starting the video stream by clicking the start button, and controlling the examination process by clicking the save and stop buttons. The video retrieval module is used to retrieve historical examination records from the platform database by patient ID, examination date, or time period. The local analysis module is used to process the video frame by frame, generate segmentation results, and display them visually.

10. The lightweight endoscopic polyp real-time segmentation method based on feature sparse sampling according to claim 8, characterized in that, The workflow of the endoscopic examination assistance platform includes: Receives video streams or local video files from an endoscope camera or system as input; The system employs a multi-threaded approach to acquire each frame of the input video stream, converts RGB images to grayscale images, performs a Fast Fourier Transform (FFT) to map the image to the frequency domain, centers the spectrum to suppress low-frequency components and highlight high-frequency details, and then reconstructs the image through an Inverse Fast Fourier Transform (IFT). The average value of the high-frequency amplitude in the reconstructed image is calculated and compared with a preset threshold. Based on the comparison between the average value of the high-frequency amplitude and the preset threshold, it determines whether the frame is a blurry frame and selects to skip the frame or reduce its weight according to the system configuration. All valid frames are adjusted to a size of 256×256 using bilinear interpolation. The pre-processed valid video frames are input into the lightweight polyp segmentation model for inference. The lightweight polyp segmentation model outputs a binary segmentation mask, where 0 represents the background and 1 represents the polyp region. The binary segmentation mask is superimposed on the original video frame in real time for display. The system writes the patient's unique identifier, examination timestamp, original video file path, and corresponding segmentation mask sequence into a local SQLite database. It also supports quick queries by fields such as patient ID and examination date range, and allows for visual playback in the user interface by overlaying the original video with the segmentation mask.

Images