An endoscope video segmentation method and system based on optical flow guidance and target discovery

Abstract

This invention provides a method and system for endoscopic video segmentation based on optical flow guidance and target discovery. The method includes: acquiring a continuous video frame sequence of endoscopic surgery and performing data preprocessing; inputting the preprocessed video frame sequence into an optical flow-appearance dual-branch encoder to extract a unified spatiotemporal feature representation that fuses motion and appearance information; inputting this feature representation into a slot attention module to generate a set of decoupled object-center semantic slots and their corresponding spatial attention weight maps; mapping the semantic slot representations to pixel space through a segmentation decoder to output semantic segmentation masks for surgical instruments, anatomical organs, and the background environment; and post-processing optimization of the obtained segmentation masks to improve the structural integrity and boundary smoothness of the masks. This method can still achieve accurate segmentation in complex surgical scenarios such as instrument occlusion, tissue deformation, and sudden changes in illumination, significantly improving segmentation accuracy and processing efficiency, and providing high-precision real-time visual perception support for intraoperative navigation.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and clinical medical image processing. Specifically, it relates to an endoscopic video segmentation method and system based on optical flow guidance and target discovery, which is particularly suitable for scenarios such as minimally invasive surgery and robot-assisted surgery, to perform high-precision and robust real-time semantic segmentation of surgical instruments, anatomical organs and background environment in endoscopic videos. Background Technology

[0002] Endoscopic video serves as the core visual window for surgical robots to perceive the intraoperative environment, and its analysis quality directly determines the intelligence level and operational safety of the surgical system. In minimally invasive surgery, especially robot-assisted surgery, endoscopic video not only provides real-time visualization of anatomical structures but also carries multi-dimensional dynamic information such as instrument movement, tissue deformation, and surgical procedures. Among these, spatial segmentation (i.e., pixel-level semantic segmentation of surgical instruments and human organs in endoscopic video) is a fundamental task of video intelligent perception, enabling the accurate identification and localization of key targets (such as surgical instruments, blood vessels, and organs) in the surgical scene. Traditional semantic segmentation methods typically rely on encoder-decoder architectures and large-scale pixel-level labeled data, designed for objects with fixed contours and stable layouts in natural scenes, making them difficult to directly transfer to the surgical video domain. The tissue deformation, instrument movement, and scarcity of annotations in endoscopic images severely disrupt the convolutional inductive bias and data assumptions upon which traditional methods rely, leading to a significant performance degradation in practical surgical segmentation tasks. Accurate and robust semantic segmentation results provide indispensable spatial structural information support for surgical navigation, risk warning, process optimization and postoperative analysis, and are a key technology link to promote the evolution of surgical robots from "assisted execution" to "intelligent perception and autonomous decision-making".

[0003] Existing research has explored related fields with different focuses. In the area of ​​surgical image segmentation based on fully supervised learning, the m2caiSeg framework, which uses a convolutional neural network with an encoder-decoder structure as the mainstream, can achieve semantic segmentation of surgical instruments and organs using pixel-level labeled data. (Maqbool S, Riaz A, Sajid H, et al. "m2caiSeg: Semantic Segmentation of Laparoscopic Images using ConvolutionalNeural Networks[J]."). To reduce the dependence on dense annotations, some studies have attempted to introduce self-supervised mechanisms, such as using self-supervised learning to complete a monocular depth estimation task as pre-training, so that the network learns the geometric and structural priors of the surgical scene, and then transfers them to the semantic segmentation task of surgical scenes with limited annotations for fine-tuning. (Yang Z, Pan J, Dai J, et al. Self-supervised endoscopy depth estimation framework with CLIP-guidancesegmentation[J]. Biomedical Signal Processing and Control, 2024, 95:106410.). From the perspective of temporal modeling, some works have proposed a semi-supervised learning framework based on motion flow, which uses optical flow prediction and motion compensation to learn the temporal consistency in the video, thereby improving the stability of the segmentation results in the temporal dimension. (Li H, LuD, Yao X, et al. Endo-SemiS: Towards Robust Semi-Supervised ImageSegmentation for Endoscopic Video[J]. arXiv preprint arXiv:2512.16977,2025.).

[0004] While existing methods have improved the performance of endoscopic video segmentation to some extent, they still face several significant shortcomings: First, limited feature representation capabilities: traditional methods often rely on single appearance or motion features, failing to fully utilize spatiotemporal information, leading to decreased segmentation accuracy in complex surgical scenarios such as instrument occlusion, tissue deformation, and lighting changes. Second, insufficient temporal modeling: most methods focus on segmenting single-frame images, neglecting to model the motion coherence and identity consistency of targets in video sequences, easily resulting in jitter and breakage in the segmentation results over time. Third, strong dependence on labeled data: especially fully supervised methods require a large amount of pixel-level annotation, while the generalization ability of semi-supervised / self-supervised methods in real surgical videos still needs improvement, making it difficult to adapt to diverse working conditions of different hospitals, equipment, and surgical types. Summary of the Invention

[0005] This invention addresses the shortcomings of existing endoscopic video segmentation technologies, such as limited feature representation capabilities, insufficient temporal modeling, and strong dependence on pixel-level annotation. It provides an endoscopic video segmentation method and system based on optical flow guidance and target discovery. By constructing an end-to-end processing framework of "motion perception-entity decoupling-pixel mapping", it fully integrates spatiotemporal information without requiring pixel-level annotation, achieving stable and accurate segmentation of dynamic targets in surgical scenes, and ensuring the temporal consistency and structural integrity of the segmentation results.

[0006] According to one aspect of the present invention, an endoscopic video segmentation method based on optical flow guidance and target detection is provided, comprising:

[0007] Acquire a continuous sequence of video frames to be processed during endoscopic surgery; and preprocess the endoscopic video sequence to obtain a preprocessed video sequence.

[0008] A dual-branch encoder with an optical flow branch and an appearance branch is constructed: the optical flow branch extracts motion features; the appearance branch extracts texture features.

[0009] After concatenating motion features and texture features along the channel dimension, the concatenation is input into the slot attention module to obtain several object-centered semantic slots.

[0010] The semantic feature vector generated by slot attention is fed into the segmentation decoder to obtain a semantic segmentation mask containing surgical instruments, anatomical organs and background environment;

[0011] Post-processing is performed on the segmentation mask to improve its structural integrity and boundary smoothness.

[0012] The extracted motion features are used to guide the slot attention module to focus on and decouple from dynamic targets; the generated object center semantic slot and its spatial attention weight map constitute the direct input for generating the semantic segmentation mask, forming an end-to-end processing link of "motion perception - entity decoupling - pixel mapping".

[0013] Preferably, the acquisition of a continuous video frame sequence of endoscopic surgery and the preprocessing thereof include:

[0014] Decode the stored laparoscopic surgery video files to obtain a sequence of image frames with temporal continuity;

[0015] The dimensions of all image frames are adjusted to a uniform preset resolution using image scaling operations; the RGB channel values ​​of each pixel are numerically transformed and mapped to a preset numerical range or distribution.

[0016] Preferably, the construction of the optical flow branch encoder includes:

[0017] The input consists of consecutive video frame pairs; a feature encoder is used to extract multi-scale feature maps of the input frames; a correlation volume construction module is used to calculate and generate the correlation volume of full pixel pairs between frames; and an iterative optimization module implemented by a convolutional gated recurrent unit is used to progressively regress and update the optical flow field.

[0018] Preferably, the construction of the appearance branch encoder includes:

[0019] Using a single-frame image as input, and a residual network as the backbone, multi-scale spatial representations are obtained through hierarchical downsampling and feature transformation. Furthermore, a feature pyramid structure is combined with parallel multi-branch pooling and upsampling operations to achieve cross-scale contextual information fusion.

[0020] Preferably, the feature is that the slot attention module is used to automatically decouple the fused spatiotemporal features into a representation of a set of object centers, including:

[0021] A set of learnable slot vectors is used as a query (Q) and similarity is calculated with the key (K) obtained by mapping the input features. Competitive normalization is performed on the slot dimension using the Softmax function to generate a spatial attention weight map. The value (V) vector is weighted and summed based on the weight map to generate an update signal. The slot vector state is iteratively updated using GRU units based on the update signal. After multiple rounds of iterative operations, the decoupled central semantic slot and its spatial attention weight map are output.

[0022] Preferably, the segmentation decoder is used to map the feature-decoupled object center representation into a high-resolution semantic segmentation mask for surgical instruments, anatomical organs, and the background environment, including:

[0023] The segmentation decoder takes the object's central semantic slot and its spatial attention weight map as input. First, each slot vector is combined with its attention map through semantic slot reassembly to generate an initial feature representation. Then, the feature map spatial size is progressively expanded through convex combination upsampling operations, and skip connections are made with the shallow features of the encoder during the upsampling process. Finally, high-resolution features are mapped to semantic segmentation masks for surgical instruments, anatomical organs, and the background environment through convolutional layers.

[0024] Preferably, the post-processing of the segmentation mask includes:

[0025] Morphological opening operations are performed on the mask output by the decoder to remove isolated noise pixels; pseudo-regions with an area smaller than a preset threshold are filtered out by connected component analysis; and Gaussian smoothing is applied to the mask edges to eliminate jagged edges.

[0026] According to another aspect of the present invention, an endoscopic video segmentation system based on optical flow guidance and target detection is provided, comprising:

[0027] The image acquisition module is used to capture and preprocess the endoscopic video stream during the operation in real time, and extract key frame sequences from the raw video, and perform necessary format unification, size standardization and basic image enhancement.

[0028] The dual-branch co-coding module is used to simultaneously capture visual appearance and scene motion information. This module contains two parallel and interactive branches: the appearance branch uses a deep convolutional network to extract static features for each frame, while the optical flow branch focuses on analyzing pixel motion vectors between consecutive frames, encoding dynamic information such as instrument movement and tissue deformation.

[0029] The target discovery and decoupling module autonomously discovers and separates multiple potential targets in a video, given only video-level or image-level category labels. This module typically uses a set of learnable "semantic slots" as object-centric representations. Through iterative attention allocation and feature aggregation, it decomposes and categorizes the mixed features extracted by the preceding module into individual semantic slots, forcing each slot to focus on representing a single semantic entity. This process clearly separates and represents high-level semantics from the ambiguous signal.

[0030] The mask generation and decoding module is responsible for converting abstract semantic slots into pixel-level segmentation results. This module receives semantic slots from the object detection module and multi-scale spatial feature maps from the encoding module, and ultimately completes the entire process from video stream to pixel-level understanding.

[0031] According to a third aspect of the present invention, a computer terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform the method described in any one of the above-described inventions, or to run the system described in any one of the above-described inventions.

[0032] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described in any one of the above-described inventions, or to run the system described in any one of the above-described inventions.

[0033] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0034] This invention provides an endoscopic video segmentation method and system based on optical flow guidance and target discovery. It constructs an end-to-end framework of "optical flow-driven—target decoupling—pixel mapping": Optical flow-appearance dual-branch co-coding synchronously captures instrument motion and tissue texture; slot attention unsupervisedly decomposes mixed spatiotemporal features into object-centered semantic slots; and the decoder maps the slot representations to a high-resolution mask. No pixel-level annotation is required throughout the process; the optical flow motion field guides slot attention to focus on dynamic targets, maintaining segmentation continuity even in scenarios with occlusion, deformation, and sudden changes in illumination. Temporal motion information drives spatial feature allocation, enabling dynamic instruments to achieve sustained high response in the feature space. The mask is smooth between frames and has sharp edges, providing a low-annotation-cost real-time perception paradigm for minimally invasive surgery. Attached Figure Description

[0035] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0036] Figure 1 This is an overall architecture diagram of the method provided in the embodiments of the present invention.

[0037] Figure 2 This is a schematic diagram of the optical flow branch encoder in an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of the appearance branch encoder in an embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram of the slot attention module in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of the segmentation decoder in an embodiment of the present invention. Detailed Implementation

[0041] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0042] This invention provides a method and system for endoscopic video segmentation based on optical flow guidance and target discovery, constructing an end-to-end processing framework of "motion perception-entity decoupling-pixel mapping". This framework first extracts complementary spatiotemporal features from dynamic surgical videos using an optical flow-appearance dual-branch co-encoder; then, using a slot attention module, it automatically discovers, separates, and represents independent semantic entities (such as different surgical instruments and organ tissues) in the scene without dense pixel annotation supervision; finally, through a segmentation decoder deeply coupled with the encoding structure, it accurately maps the abstract entity representations back to the pixel space, generating a high-quality segmentation mask. This method significantly improves segmentation robustness, temporal consistency, and model interpretability in complex surgical scenarios such as instrument occlusion, tissue deformation, and sudden changes in illumination.

[0043] like Figure 1 As shown, the overall architecture of this method is as follows: Figure 1 As shown, the main components include five core steps: data preprocessing, dual-branch cooperative coding, slot attention target decoupling, segmentation decoding, and post-processing optimization. Each step is described in detail below with reference to a specific embodiment. The endoscopic video segmentation method based on optical flow guidance and target detection provided in this embodiment includes the following operations:

[0044] S1: Acquire a continuous sequence of video frames to be processed during endoscopic surgery and perform preprocessing.

[0045] S101: Video Frame Sequence Acquisition

[0046] Laparoscopic surgery video files are read from local storage (such as hospital PACS systems or surgical recording equipment) or network storage systems. These video files can be in formats including, but are not limited to, MP4, AVI, and MOV. The video stream is decoded using interfaces such as VideoCapture, and downsampled from 30fps to 15fps based on the original frame rate or computing resources, extracting a sequence of original image frames with temporal continuity. To retain sufficient motion information for optical flow calculation, the time interval between adjacent frames should not be too long; typically, consecutive frames or frames with a one-frame interval are extracted.

[0047] S102. Dimensional Standardization and Numerical Normalization

[0048] To meet the requirements of subsequent convolutional neural networks for consistent input size, bilinear interpolation is used to scale all image frames to a fixed size of 1920×1080 pixels for each extracted image frame.

[0049] Using ImageNet-based statistical parameters, a transformation of (pixel value - mean) / standard deviation is performed on the R, G, and B channels of each pixel, where the mean is [123.675, 116.28, 103.53] and the standard deviation is [58.395, 57.12, 57.375]. Throughout the process, the RGB three-channel format of the image is maintained, and the final output is a normalized tensor sequence with shape (3, 1920, 1080). This operation normalizes the pixel value distribution, which helps to accelerate the convergence of neural network training.

[0050] S103, Data Augmentation (Optional during training phase)

[0051] During the model training phase, online random data augmentation can be further introduced, such as applying the same small-amplitude random rotations, horizontal flips, and brightness adjustments to image frames, to improve the model's robustness to geometric and photometric changes. Crucially, consecutive frames used to calculate optical flow must undergo identical spatial transformations to maintain geometric consistency in motion field estimation. Through this systematic preprocessing, heterogeneous videos from different surgical scenarios are transformed into uniformly formatted and well-distributed tensor sequences, laying a reliable foundation for subsequent feature extraction.

[0052] S2, Construct a two-branch co-encoder for optical flow and appearance branches:

[0053] This step uses two parallel encoders to extract motion and appearance information from video frames, respectively, to achieve feature complementarity.

[0054] The optical flow branch extracts continuous inter-frame motion features through RAFT optical flow estimation, capturing the dynamic changes of instruments and tissues; the appearance branch extracts multi-level spatial appearance features from single-frame images through ResNet-18 and feature pyramids, thereby capturing static information such as texture, color, and shape. Through parallel and collaborative design of the two branches, the model can optimize its ability to model spatial structural information and temporal motion information respectively, thus obtaining a more comprehensive and robust spatiotemporal feature representation.

[0055] S201, the optical flow branch is implemented as follows: First, a weight-shared feature encoder processes the previous and current frames separately. This encoder contains 6 residual blocks, each consisting of two 3×3 convolutional layers, instance normalization, and a ReLU activation function, progressively downsampling the input image to 1 / 8 of its original size. The output feature map has a spatial size of 240×135 and 256 channels. Simultaneously, an independent context encoder, with the same structure as the feature encoder, processes only the previous frame, extracting context features; its size is also 240×135×256. Then, the dot product similarity of all pixel pairs between the two frame feature maps is calculated, forming a four-dimensional correlation volume with dimensions of 240×135×240×135. To capture multi-scale motion, average pooling is performed on the last two dimensions of this correlation volume, constructing a 4-layer correlation pyramid. layer The space dimensions are The optimization of the optical flow field is accomplished by an iterative update module based on a convolutionally gated recurrent unit (GRU), undergoing 12 iterations. In each iteration, motion features are first retrieved from the relevant pyramid using a lookup operator based on the currently estimated optical flow field. Then, two convolutional layers are applied to the current optical flow field to obtain flow features. The motion features, flow features, and previously extracted context features are then concatenated along the channel dimension and input into the GRU. The GRU uses a 3×3 convolutional kernel, and its update process sequentially calculates the update gate, reset gate, and candidate states to finally obtain the hidden state. The hidden state is then passed through two convolutional layers: the first layer compresses 256 channels to 128 channels, and the second layer outputs 2 channels to predict the optical flow residual, which is then used to update the optical flow field. After 12 iterations, a low-resolution optical flow field with dimensions of 240×135×2 is obtained. Next, a convex combination upsampling module is used to restore the image to its original resolution of 1920×1080. For each low-resolution pixel, an 8×8×9 upsampling mask is predicted. After Softmax normalization, the nine flow values ​​in the 3×3 neighborhood are weighted and combined to obtain a high-resolution optical flow field. Finally, this high-resolution optical flow field is fed into a lightweight convolutional network containing two 3×3 convolutional layers with a stride of 2, and output channels of 128 and 256 respectively. Motion features are encoded with a spatial size of 1 / 4 of the original image, i.e., 480×270, and 256 channels. Then, average pooling is used to downsample the image to 1 / 32 resolution, i.e., 60×34, and reshape it into a sequence form, with each spatial location as a feature vector, to obtain the motion feature representation with a dimension of 60×34×256.

[0056] S202, the appearance branch is implemented as follows: The appearance branch is used to extract texture features from a single frame image, taking the current frame as input. First, ResNet-18 is used as the backbone network, followed by an initial 7×7 convolution, stride 2, 3×3 max pooling, stride 2, and four residual blocks. The spatial size of the output feature map is 1 / 32 of the original image, i.e., 60×34, with 512 channels. Then, a pyramid pooling module is introduced, performing adaptive average pooling at four scales: 1×1, 2×2, 3×3, and 6×6 in parallel. Each pooling result is reduced to 64 channels by a 1×1 convolution, and then upsampled back to a spatial size of 60×34 using bilinear interpolation. The original 512-dimensional feature map output by ResNet-18 is concatenated with these four upsampled 64-dimensional feature maps in the channel dimension to obtain a 768-dimensional feature map. Then, a 1×1 convolutional layer is used to fuse the number of channels to 256, resulting in a multi-scale context-enhanced feature map. The feature map is then input into a convolutional gated recurrent unit (CRU) module with a 3×3 kernel and a hidden state dimension of 256 for temporal modeling, outputting temporal features. To enhance spatial awareness, a learnable soft positional encoding is employed to generate a normalized grid coordinate map of size 1×60×34×4, containing four coordinate components. This map is mapped to 256 dimensions through a linear layer to obtain the positional encoding. The positional encoding is then element-wise added to the temporal features output by the CRU. Finally, the summed features are reshaped into a sequence to obtain the appearance feature representation, also with a dimension of 60×34×256. The motion and appearance features are concatenated along the channel dimension to obtain the fused spatiotemporal features, with a dimension of 60×34×512.

[0057] S3 concatenates motion features and appearance features along the channel dimension, then inputs them into the slot attention module to iteratively generate a set of decoupled central semantic slot vectors.

[0058] This module can automatically discover key object regions in videos, achieve adaptive alignment and enhancement of motion and appearance information, and realize semantic-level representation and separation of different surgical targets.

[0059] The number of slots is set to 10, and each slot has a dimension of 256. The initial slot vector is sampled from a learnable normal distribution, with the mean and variance being learnable parameters. Then, three iterations are performed, each iteration performing the following operations: First, the current slot vector and input features are projected onto the query, key, and value respectively through a linear layer, with a total of 60×34 (2040) spatial locations. A scaled dot product attention is calculated, and softmax normalization is applied to the slot dimension to obtain an attention weight matrix with a dimension of 10×2040. The values ​​are weighted and summed using the attention weights to obtain an update signal with a dimension of 10×256. The update signal and the current slot vector are input into a gated recurrent unit with a hidden state dimension of 256 for state update, resulting in a new slot vector. Then, a feedforward network is used, containing two linear layers: the first layer reduces the dimension from 256 to 512, and the second layer reduces it from 512 to 256, using ReLU activation and layer normalization to refine the slot vector in between. After three iterations, the final output consists of 10 semantic slot vectors of 256 dimensions, and the attention weight matrix from the last iteration. This attention weight matrix is ​​then reconstructed into spatial form, resulting in 10 spatial attention heatmaps, each with a size of 60×34.

[0060] S4 reconstructs the semantic slots and corresponding spatial attention maps into a feature map. Through multi-level upsampling and skip connections, spatial details are gradually restored, ultimately outputting pixel-level semantic segmentation results. This method, through feature reconstruction and local weighted fusion, effectively restores target details and boundary structures while maintaining semantic consistency, improving the spatial accuracy and semantic integrity of the segmentation mask.

[0061] The segmentation decoder maps semantic slots and their spatial attention heatmaps to pixel-level segmentation masks. First, feature reassembly is performed, multiplying each semantic slot vector (256 dimensions) by its corresponding 60×34 attention heatmap using a broadcast multiplication method. This multiplies each spatial location of the attention heatmap by the entire slot vector, resulting in 10 feature maps, each 256×60×34 in size. These 10 feature maps are then concatenated along the channel dimension to obtain a 2560×60×34 feature map. The resolution is then gradually restored through four upsampling blocks: The first upsampling block uses a 4×4 transposed convolution with a stride of 2, padding of 1, and an output channel of 1280 to double the feature map resolution to 120×68, followed by ReLU activation; the second upsampling block first uses a 5×5 convolution with a stride of 1, padding of 2, and an output channel of 1280 to refine the features, and then extracts skip connection features from the corresponding level of the appearance branch with a resolution of 120×68, adjusts the channels to 1280 through a 1×1 convolution, and adds them to the current feature map; the third upsampling block uses a 4×4 transposed convolution with a stride of 2 and an output channel of 640 to expand the resolution to 240×135, similarly introducing skip connections from the corresponding level of the appearance branch, and adjusting the channels to 640 through a 1×1 convolution; the fourth upsampling block uses a 4×4 transposed convolution with a stride of 2 and an output channel of 256 to expand the resolution to 480×270. Finally, a convex combination upsampling strategy is employed to predict an 8×8×9 upsampling mask for the 480×270 feature map. After Softmax normalization, the features within the 8×8 neighborhood are weighted and fused, increasing the resolution to the original input size of 1920×1080 while maintaining 256 channels. A 3×3 convolutional layer is then used to map the 256-dimensional features to the number of segmentation categories. In this embodiment, the total number of categories is 20, including 1 background category, 7 surgical instrument categories, and 12 anatomical organs or tissues. The output is a pixel-level semantic segmentation mask with a size of 1920×1080×20.

[0062] S5 sequentially performs morphological denoising, small region filtering, and boundary smoothing on the segmentation results to eliminate scattered noise and unreasonable segmentation regions in the predictions, improving the structural coherence and visual smoothness of the mask. This post-processing workflow is lightweight and efficient, further enhancing the usability and interpretability of the segmentation results in clinical applications.

[0063] S501, perform morphological opening operation on the binarization mask of each category using a 3×3 circular structuring element;

[0064] S502, calculate the pixel area of ​​each connected component in the mask, and filter out regions with an area less than 0.1% of the total number of pixels in the image;

[0065] S503 applies a Gaussian kernel of σ=1.5 to the mask boundary region for smoothing filtering to eliminate the jagged effect caused by pixel-level classification.

[0066] The model training process is as follows. Fifty laparoscopic cholecystectomy surgery videos collected internally were used, named the XinHua50 dataset. All videos had a resolution of 1920×1080, and keyframes were annotated by professional surgeons, totaling 3113 frames. Annotation categories included: background, grasping forceps, hooks, scissors, bipolar electrocautery forceps, irrigator, applicator, gauze, gallbladder, liver, cystic duct, cystic artery, gallbladder plate, gallbladder pedicle, blood vessels, fluid, abdominal wall, adhesions, omentum, peritoneum, intestine, and specimen bag. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio. The loss function used was weighted cross-entropy loss, the mathematical expression of which is:

[0067]

[0068] in For the total number of pixels, For the one-hot encoding of the real label, Here, c represents the predicted probability of the model, c is the class index, and the weights are... Based on the pixel frequency of class c in the training set calculate

[0069] This weight design assigns higher loss weights to less frequently occurring categories (such as gauze and bipolar electrocautery clamps), thus mitigating class imbalance. The Adam optimizer is used, with an initial learning rate of 0.0001 and weight decay of 0.00001. The batch size is set to 2, limited by GPU memory. The learning rate scheduling strategy is to multiply by 0.1 every 30 training epochs. The total number of training epochs is 200. The feature encoder of the optical flow branch is initialized using weights pre-trained on the Sintel dataset, while other modules, including the appearance branch, slot attention module, and segmentation decoder, are initialized using Xavier. In each training iteration, a set of two consecutive frames and their corresponding ground truth segmentation masks are randomly selected from the training set. Forward propagation is performed to obtain the predicted segmentation mask, the loss is calculated, and then backpropagation is used to calculate the gradient. The Adam optimizer updates all trainable parameters. After each training epoch, the average intersection-union ratio (OCR) is calculated on the validation set, and the model parameter with the highest OCR on the validation set is saved. If the mean intersection-union ratio on the validation set no longer improves after 10 consecutive rounds, training is stopped early, and the best-performing model on the validation set is loaded as the final model.

[0070] In inference applications, for each frame of the input video, the aforementioned forward propagation is performed, but without data augmentation, and the training portion in post-processing is disabled, resulting in a segmentation mask with dimensions of 1920×1080×20. For each pixel location, the category with the highest probability among 20 channels is taken as the label for that pixel, generating a single-channel index map, where 0 represents the background, 1 to 7 represent seven types of surgical instruments, and 8 to 19 represent twelve types of anatomical organs or tissues. The output has three forms: First, a pseudo-color visualization, which maps each category to a unique RGB color (e.g., forceps red, hooks green, gallbladder yellow, liver blue), generating a color segmentation map that can be overlaid on the original endoscopic image as a semi-transparent mask for real-time viewing by surgeons; second, an index map, stored as a PNG image or array format, with each pixel value being a category number for easy subsequent calculations; and third, structured instance data, which extracts the category label, bounding box, pixel area, and centroid coordinates for each connected component (i.e., each independent instrument or organ instance) through connected component analysis and outputs it in JSON format. Since the segmentation mask has already assigned a specific category to each pixel, the instrument and organ classification information for all pixels in the entire image can be directly obtained. Furthermore, the list of instrument and organ types appearing in each frame can be statistically analyzed (existence detection), used for subsequent surgical stage auxiliary judgments or instrument usage records.

[0071] This invention also provides an endoscopic video segmentation system based on optical flow guidance and target detection, the system comprising:

[0072] Image acquisition and preprocessing module: Used to capture and preprocess endoscopic video streams during surgery in real time.

[0073] Dual-branch co-coding module: includes appearance branch and optical flow branch, used to extract static appearance features and dynamic motion features.

[0074] Target discovery and decoupling module: It adopts learnable semantic slots and iterative attention mechanism to achieve unsupervised target separation.

[0075] Mask generation and decoding module: maps semantic slot representations to high-resolution semantic segmentation masks.

[0076] Furthermore, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the above-described method. Additionally, the present invention also provides a computer-readable storage medium storing a computer program thereon, wherein the program, when executed by a processor, implements the above-described method.

[0077] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0078] To verify the technical effectiveness of this invention, comparative experiments were conducted on a publicly available dataset of endoscopic surgery videos. Experimental results show that the method of this invention outperforms existing mainstream methods on multiple evaluation metrics. Specifically:

[0079] In the case of instrument occlusion, the average intersection-union ratio (mIoU) of the method of the present invention is improved by about 8.3% compared with the baseline method, demonstrating stronger occlusion robustness;

[0080] In the case of organizational deformation, the segmentation mask of the present invention improves the temporal consistency (measured by average mask similarity) between frames by about 12.6%, effectively reducing inter-frame jitter;

[0081] Under training conditions with only video-level category labels, the segmentation accuracy of the method of this invention reaches or even surpasses that of some fully supervised methods, verifying its advantage of low label dependency.

[0082] The above experimental data fully demonstrate the superiority of this invention in terms of segmentation accuracy, temporal consistency, and annotation efficiency.

[0083] In summary, this invention successfully solves several technical challenges in existing endoscopic video segmentation technologies by constructing an end-to-end framework of "optical flow driving - target decoupling - pixel mapping," providing an efficient and robust solution for real-time visual perception in intelligent minimally invasive surgery.

[0084] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for segmenting endoscopic videos based on optical flow guidance and target detection, characterized in that, Includes the following steps: S1. Obtain a continuous video frame sequence of endoscopic surgery and perform preprocessing to obtain a preprocessed video sequence; S2. Construct a dual-branch encoder, including an optical flow branch and an appearance branch. The optical flow branch is used to extract motion features between consecutive frames, and the appearance branch is used to extract texture features in a single frame image. S3. The motion features and texture features are concatenated in the channel dimension and then input into the slot attention module. The learnable slot vector in the slot attention module is used as the query and the concatenated spatiotemporal feature mapping is used as the key and value. A set of decoupled object-centered semantic slots and their corresponding spatial attention weight maps are generated through an iterative attention mechanism. S4. Input the object center semantic slot and its corresponding spatial attention weight map into the segmentation decoder, and output a semantic segmentation mask containing surgical instruments, anatomical organs and background environment through feature recombination and upsampling operations; S5. Post-process the semantic segmentation mask to improve the structural integrity and boundary smoothness of the mask.

2. The endoscopic video segmentation method based on optical flow guidance and target detection according to claim 1, characterized in that, In step S1, the video frame sequence is preprocessed, including: Decode the temporally continuous sequence of image frames from the stored surgical video files; The image frame sequence is normalized in size, and all image frames are scaled to a uniform preset resolution; The pixel values ​​of the scaled image frame are normalized, and the RGB three-channel pixel values ​​are mapped to a preset value range or distribution.

3. The endoscopic video segmentation method based on optical flow guidance and target detection according to claim 1, characterized in that, The optical flow branch includes: A feature encoder is used to extract multi-scale feature maps from consecutive input frames; The relevant volume construction module is used to calculate and generate the inter-frame full pixel pair relevant volume; The iterative optimization module, implemented by a convolutional gated recurrent unit, is used to progressively regress and update the optical flow field, outputting a dense optical flow field.

4. The endoscopic video segmentation method based on optical flow guidance and target detection according to claim 3, characterized in that, The optical flow branch further includes encoding the dense optical flow field into a motion feature representation consistent with the appearance feature dimension through a lightweight convolutional network.

5. The endoscopic video segmentation method based on optical flow guidance and target detection according to claim 1, characterized in that, The appearance branch uses a residual network as its backbone structure, obtains multi-scale spatial representations through hierarchical downsampling and feature transformation, and combines a feature pyramid structure to achieve cross-scale contextual information fusion using parallel multi-branch pooling and upsampling operations.

6. The endoscopic video segmentation method based on optical flow guidance and target detection according to claim 1, characterized in that, The slot attention module is used to automatically decouple the fused spatiotemporal features into a representation of a set of object centers. The specific steps are as follows: A set of learnable slot vectors is used as the query Q, and similarity is calculated with the key K obtained by the input feature mapping. The slot dimension is competitively normalized using the Softmax function to generate a spatial attention weight map. The value V obtained from the fused feature mapping is weighted and summed according to the spatial attention weight map to generate an update signal; The state of the learnable slot vector is iteratively updated based on the update signal using a gated loop unit. After multiple iterations, each slot vector converges to the feature representation of an independent object in the scene, and finally outputs a set of decoupled central semantic slots and their corresponding attention weight maps, thus completing feature decoupling.

7. The endoscopic video segmentation method based on optical flow guidance and target detection according to claim 1, characterized in that... The segmentation decoder is used to map the feature-decoupled object center representation into a high-resolution semantic segmentation mask for surgical instruments, anatomical organs, and the background environment, including: The segmentation decoder takes the object center semantic slot and its spatial attention weight map as input, and gradually restores the feature map resolution through semantic slot recombination, convex combination upsampling, and skip connection fusion with shallow features of the encoder. Finally, it outputs the semantic segmentation mask of surgical instruments, anatomical organs and background environment through convolutional layers.

8. The endoscopic video segmentation method based on optical flow guidance and target detection according to claim 1, characterized in that... Post-processing of the segmentation mask includes: Morphological opening operations are used to remove isolated noise pixels, connected component analysis is used to filter out pseudo regions with an area smaller than a preset threshold, and Gaussian smoothing is applied to the mask edges output by the decoder to eliminate jagged edges.

9. An endoscopic video segmentation system based on optical flow guidance and target detection, characterized in that, include: The image acquisition and preprocessing module is used to capture and preprocess the endoscopic video stream during the surgical procedure in real time and output a keyframe sequence. The dual-branch co-coding module includes an appearance branch and an optical flow branch. The appearance branch is used to extract static appearance features of a single frame image, and the optical flow branch is used to extract dynamic motion features between consecutive frames. The dual-branch co-coding module concatenates the static appearance features and dynamic motion features in the channel dimension to output fused spatiotemporal features. The target discovery and decoupling module uses a set of learnable semantic slots as the object center representation, and the fused spatiotemporal feature mapping as the key and value. Through an iterative attention allocation and feature aggregation process, the fused spatiotemporal features are decomposed and classified into each semantic slot, and the decoupled semantic slots and their spatial attention weight map are output. The mask generation and decoding module is used to convert the semantic slots and their spatial attention weight maps into semantic segmentation mask results containing surgical instruments, anatomical organs and background environment.

10. The endoscopic video segmentation system based on optical flow guidance and target detection according to claim 9, characterized in that, In the dual-branch cooperative coding module: The optical flow branch includes a feature encoder, a related volume construction module, and an iterative optimization module implemented by a convolutional gated recurrent unit, used to extract dense optical flow fields from consecutive frame pairs; The appearance branch uses a residual network as its backbone structure and combines it with a feature pyramid structure to extract multi-level spatial appearance features from a single frame image.

11. The endoscopic video segmentation system based on optical flow guidance and target detection according to claim 9, characterized in that, The target discovery and decoupling module implements iterative attention allocation in the following way: The learnable slot vector is used as the query Q, and its similarity with the key K of the fused spatiotemporal feature mapping is calculated. The spatial attention weight map is generated by competitive normalization using Softmax. An update signal is generated by weighting and summing the value V vectors based on the spatial attention weight map. The slot vector state is iteratively updated through a gated loop unit until each slot vector converges to the feature representation of an independent object in the scene.

12. The endoscopic video segmentation system based on optical flow guidance and target detection according to claim 9, characterized in that, The mask generation and decoding module includes: The feature reorganization unit is used to combine the semantic slot with the spatial attention weight map to generate a preliminary feature map; Multi-level upsampling units, including skip connection structures, are used to progressively restore feature map resolution; A convex combination upsampling unit is used to perform local weighted fusion in the final upsampling stage; The classification output unit is used to map high-resolution features to semantic segmentation masks.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the program, it implements the method of any one of claims 1-8, or runs the system of claims 9-12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method of any one of claims 1-8, or runs the system of claims 9-12.

Images