Single-hole laparoscopic operation field occlusion segmentation method and system based on deep learning

Abstract

The application provides a single-hole laparoscopic operation field occlusion segmentation method and system based on deep learning, relates to the technical field of medical image segmentation, and comprises the following steps: performing quality evaluation and image enhancement on a single-hole laparoscopic operation video frame to obtain an enhanced feature map; calculating an occlusion confidence score according to the spatial consistency change and the relative area reduction of an adjacent frame visible mask; managing the historical features of instrument instances by using a double-branch memory management mechanism; performing cross-frame context enhancement on the enhanced feature map by using a memory attention mechanism to obtain a memory enhanced feature map; constructing all instrument instances in the video frame into a directed graph, performing neighborhood aggregation reasoning by using a graph convolution network, and obtaining occlusion layer superposition node features; and decoding the memory enhanced feature map by using a parallel double-head mask decoder to obtain a visible region mask, an instrument complete shape mask and an occluded region mask. The application can realize synchronous segmentation output of the visible region and the complete shape of the occluded region of the surgical instrument.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation technology, and in particular to a method and system for segmenting occlusion in single-port laparoscopic surgery field based on deep learning. Background Technology

[0002] Single-port laparoscopic surgery (SPLS) is a minimally invasive surgical technique that involves inserting all surgical instruments and a laparoscope through a single abdominal wall incision. Compared to traditional multi-port laparoscopic surgery, it offers clinical advantages such as smaller incision trauma, faster postoperative recovery, and more concealed incision locations, leading to its expanding application in the surgical field in recent years. However, because all instruments enter the body cavity through the same incision, the manipulable space for each instrument within the abdominal cavity is highly limited, negating the triangular manipulation advantage of multi-port procedures. This results in frequent mutual obstruction of instruments within the surgical field, with instrument heads, joints, and target tissues often covered by other instruments, surgical smoke, blood, or other obstructions. When performing delicate procedures, surgeons find it difficult to directly perceive the complete position and orientation of obstructed instruments from the two-dimensional endoscopic images, posing potential risks to surgical safety and operational precision. In the field of intelligent surgical assistance systems and surgical robots, the complete segmentation and position perception of instruments in the surgical field are the basic prerequisites for navigation, obstacle avoidance and automated operation. The problem of dense occlusion of multiple instruments in single-port scenarios has become one of the core technical bottlenecks restricting the application of the above technologies.

[0003] For the problem of surgical video instrument segmentation, existing methods have evolved from early single-frame instance segmentation methods based on U-Net and Mask R-CNN to video temporal segmentation methods based on the Transformer architecture. Against this backdrop, Chinese patent application CN117789921A discloses a text-assisted online surgical video instrument tracking system. This system uses SAM and CLIP pre-trained models as visual encoders and text encoders, respectively. It performs pixel-level fusion of textual semantic information and visual features through a hierarchical cross-modal fusion module and designs an explicit temporal cue propagation module to pass historical query tags from the mask decoder to subsequent frames, thereby enhancing the spatiotemporal consistency of the segmentation results. This method has made some progress in introducing text prompts and online temporal information, but the propagation of its temporal information only relies on the query markers output by the decoder of the previous frame. It lacks an active perception and response mechanism for instrument occlusion events. When multiple instruments are continuously occluded, the incomplete mask features will be included in the historical information of subsequent frames indiscriminately, which leads to a decrease in the reliability of segmentation under occlusion conditions. Moreover, this method is essentially still in the category of modal segmentation and cannot infer the complete instrument shape of the occluded area. Summary of the Invention

[0004] In view of this, the present invention provides a method and system for single-port laparoscopic surgical field occlusion segmentation based on deep learning. By introducing an occlusion perception dual-branch memory management module, a single-port multi-instrument occlusion relationship graph module, and a non-modal dual-head mask decoder, the present invention achieves active detection of occlusion events and selective preservation of historical features in the temporal segmentation framework of surgical video. At the same time, it explicitly models the spatial stacking relationship between multiple instruments, and finally achieves synchronous segmentation output of the complete shape of the visible area and the occluded area of ​​the surgical instruments.

[0005] The technical solution of this invention is implemented as follows: On the one hand, this invention provides a deep learning-based method for segmenting the surgical field obstruction in single-port laparoscopic surgery, including: S1. Quality assessment and image enhancement processing are performed on the video frames of single-port laparoscopic surgery, and multi-scale feature extraction and fusion are performed using a hierarchical visual encoder to obtain enhanced feature maps. S2. For each device instance in the enhanced feature map, calculate the occlusion confidence score based on the spatial consistency change and relative area reduction of the visible mask in adjacent frames, and use the score exceeding a preset threshold as the trigger condition for the occlusion event. S3. A dual-branch memory management mechanism consisting of a propagation memory bank and an occlusion protection memory bank is adopted to manage the historical features of the device instance. The propagation memory bank continuously stores the device instance features of the unoccluded frames. The occlusion protection memory bank freezes and saves the complete morphological features of several frames before the occlusion occurs when the occlusion event is triggered, and releases them after the occlusion ends. S4. Using a weighted combination of propagation memory features and occlusion protection memory integrity features, the enhanced feature map is enhanced across frames through a memory attention mechanism to obtain a memory-enhanced feature map. The weight of the occlusion protection memory features decreases as the duration of occlusion increases. S5. Construct all instrument instances in the video frame into a directed graph, initialize node features with the appearance and boundary features of each instance, use the occlusion and stacking probability between nodes as the weight of the directed edges, and use a graph convolutional network to perform neighborhood aggregation inference to obtain the occlusion and stacking node features. S6. A parallel dual-head mask decoder is used to decode the memory-enhanced feature map. The visible mask terminal outputs the visible area mask of each instrument instance. The complete mask terminal uses the complete shape feature and the occlusion layer node feature as additional inputs to output the complete shape mask of the instrument, and outputs the difference between the complete shape mask and the visible area mask as the occluded area mask.

[0006] Preferably, in step S2, the occlusion confidence score is calculated as follows: For the i-th device instance in the current frame t, predict the mask using the visible area of ​​the current frame. Predicted mask of visible region from the previous frame The occlusion confidence score is defined by the weighted sum of the decrease in cross-union ratio and the relative reduction in mask pixel area. : ; Where the subscript t-1 represents the previous frame, for pixel area This is the predicted mask for the visible area of ​​the previous frame. for pixel area The cross-joint ratio (CLU) of the two frames' masks. This is the balance coefficient; when Exceeding the occlusion detection threshold When an occlusion event is triggered, record the start time of the occlusion. ; when Persistently below the recovery threshold When the occlusion exceeds a certain number of consecutive frames, the occlusion is determined to have ended and the occlusion protection memory is released. .

[0007] Preferably, in step S3, the occlusion protection memory, when the occlusion event is triggered, uses the occlusion start time as the reference. Based on this, continuous freezing preservation The complete set of morphological features of a frame ,in For the i-th device instance in the th... The complete morphological features of a frame To preserve frame counts, the occlusion protection memory remains frozen during the duration of the occlusion event and is cleared and released after the occlusion ends.

[0008] Preferably, in step S4, the memory attention mechanism is implemented as follows: To enhance feature maps The feature vector of each spatial location is used as the query to propagate the content stored in the memory. With time-decayed weighted complete morphological feature set The concatenated content along the sequence dimension consists of keys and values; multi-head cross-attention computation is then performed. ; in, To enhance memory feature maps, , The duration of the occlusion is given by t, where t is the current time. This indicates concatenation along the sequence dimension. This represents multi-head cross-attention calculation with time-decaying weights. Defined as: ; in This is the decay rate hyperparameter.

[0009] Preferably, in step S5, the calculation method for the occlusion stacking probability between nodes is as follows: For each directed edge (i,j) in the directed graph, extract the node feature vectors of instrument instances i and j respectively. , And the occlusion boundary features obtained after convolutional pooling of the spatially overlapping regions of the two. The three features are concatenated and then mapped through a two-layer fully connected network to scalar edge weights representing the probability that instance i occludes instance j. : ; in and These are the learnable weight matrices and biases of a two-layer fully connected network, respectively. This is the Sigmoid activation function.

[0010] Preferably, in step S5, the node feature update rule for neighborhood aggregation inference in the graph convolutional network is as follows: Each node sums its own features with the features of its directed neighbors according to the corresponding edge weights, and then performs a learnable linear transformation and nonlinear activation to obtain the node features for the next layer; The layer update rules are as follows: ; in, For the first The node feature vector of instance i in layer , Let i be the set of all neighboring nodes of a directed graph that have directed edges pointing to node i. and For the first The learnable weight matrix and bias of the layer; After passing through a two-layer graph convolutional network, the output node features of each instance are the occlusion layer node features.

[0011] Preferably, in step S6, the complete masking terminal constructs an extended query token for each instrument instance. As decoding input: Extract shape prior features from the complete morphological features stored in the occlusion protection memory bank. : ; shape prior features Features of overlapping nodes with occlusion splicing, via linear projection layer After projecting to the query token dimension, it is compared with the object query token. Add them together to form the extended query token: ; To extend query tokens To query and remember enhanced feature maps as keys and values, a complete device morphology mask is obtained through multi-layer Transformer decoding; when the instance is not in an occluded state... Take the zero vector.

[0012] Preferably, in step S1, the quality assessment uses the Laplacian gradient variance as the frame sharpness score for the t-th frame image. Calculate frame sharpness score : ; in for The second-order Laplace gradient plot, For variance calculation; Below the preset resolution threshold The frame is determined to be a fuzzy frame. Fuzzy frames do not participate in the writing of the propagation memory and the occlusion protection memory, but they still participate in the inference process to maintain the temporal continuity. The frames that pass the quality assessment are sequentially processed by contrast-limited adaptive histogram equalization, dehazing based on smoke density estimation, and radial distortion geometric correction to obtain preprocessed images; The preprocessed image is used to extract feature maps at four resolution scales via a hierarchical visual encoder. s=1,2,3,4, and then the feature pyramid network performs top-down cross-scale fusion of feature maps at each scale to output an enhanced feature map with uniform resolution. .

[0013] Preferably, in step S5, the specific method for initializing node features with the appearance and boundary features of each instance is as follows: For the i-th device instance, its enhancement feature map The local feature maps extracted by the region of interest alignment operation are compressed to a fixed dimension using global average pooling to obtain the appearance feature vector. The visible region mask boundary is extracted by the gradient operator, flattened, and then mapped to a fixed dimension through a fully connected layer to obtain the boundary feature vector. The two are then combined to form the initial features of the node. As features of nodes in each layer of a graph convolutional network Initial input.

[0014] In addition, the present invention also provides a deep learning-based single-port laparoscopic surgical field occlusion segmentation system to implement the above-described method, the system comprising: The image preprocessing and feature extraction module is used to perform quality assessment and image enhancement processing on single-port laparoscopic surgery video frames, and uses a hierarchical visual encoder to perform multi-scale feature extraction and fusion to obtain enhanced feature maps. The occlusion event detection module is used to calculate the occlusion confidence score for each device instance in the enhanced feature map based on the spatial consistency change and relative area reduction of the visible mask in adjacent frames. The score exceeding a preset threshold is used as the trigger condition for the occlusion event. The dual-branch memory management module is used to manage the historical features of device instances using a dual-branch memory management mechanism consisting of a propagation memory bank and an occlusion protection memory bank. The propagation memory bank continuously stores the device instance features of unoccluded frames, while the occlusion protection memory bank freezes and saves the complete morphological features of several frames before the occlusion occurs when an occlusion event is triggered, and releases them after the occlusion ends. The memory attention module is used to perform cross-frame context enhancement on the enhanced feature map by weighted combination of propagation memory features and occlusion protection memory integrity features. The weight of occlusion protection memory integrity features decreases as the duration of occlusion increases. The occlusion relationship graph module is used to construct a directed graph of all instrument instances in the video frame. The node features are initialized with the appearance and boundary features of each instance. The occlusion and stacking probability between nodes is used as the weight of the directed edge. A graph convolutional network is used for neighborhood aggregation inference to obtain the occlusion and stacking node features. The parallel dual-head mask decoding module is used to decode the memory-enhanced feature map using a parallel dual-head mask decoder. The visible mask terminal outputs the visible area mask of each instrument instance, while the complete mask terminal outputs the complete shape mask of the instrument with the complete shape feature and the occlusion layer node feature as additional inputs and outputs the occluded area mask. The difference between the complete shape mask and the visible area mask is output as the occluded area mask.

[0015] The present invention has the following advantages over the prior art: (1) This invention employs an occlusion-aware dual-branch memory management module, dividing the memory bank into a propagation memory bank (PMB) and an occlusion-protected memory bank (OGMB) that operate in parallel. The PMB only accepts the writing of features from unoccluded frames that meet the confidence requirements, while the OGMB freezes and saves the complete morphological features of several frames prior to the occlusion when the occlusion event is triggered. This mechanism prevents the pollution of historical memory by incomplete masks during occlusion from the mechanism level, avoiding the problem of segmentation error accumulating and amplifying over time when the effective historical features are gradually covered by the general video segmentation framework when the occlusion duration is long.

[0016] (2) The non-modal dual-head mask decoder designed in this invention outputs two types of results in parallel: visible region mask (Modal Head) and complete shape mask (Amodal Head), and the occluded region mask is directly derived from the difference between the two. AmodalHead achieves reasonable inference of the shape of the occluded part by fusing the shape prior features saved by OGMB and the stacking constraint features of the occlusion relationship graph, which solves the inherent limitation of existing segmentation methods that can only perceive the visible pixel range and cannot predict the occluded region.

[0017] (3) The single-hole multi-instrument occlusion relationship graph module (ORG) constructed in this invention uses each instrument instance as a node, calculates the directed edge weights based on the appearance similarity and overlapping area gradient between instances, aggregates neighborhood stacking relationship information through graph convolutional network, and outputs node features carrying spatial stacking direction constraints, providing geometric direction guidance for shape completion of Amodal Head, effectively avoiding the problem of directional errors in shape completion results when there are no stacking relationship constraints.

[0018] (4) The temporal decay mechanism introduced in this invention simultaneously applies to the memory attention weighting of OGMB and the shape prior fusion of AmodalHead, enabling the system to prioritize the use of historical complete shape features when the occlusion time is short, and automatically reduce the weight of historical priors and switch to prediction based on the visual features of the current frame when the occlusion time is long, thus realizing adaptive processing for different occlusion duration scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the overall model processing of the present invention; Figure 3 This is a schematic diagram of the dual-branch memory management mechanism of the present invention; Figure 4 This is a schematic diagram of the occlusion relationship of the present invention; Figure 5 This is a schematic diagram of the parallel dual-head mask decoder of the present invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, this invention provides a deep learning-based method for segmenting the surgical field obstruction in single-port laparoscopic surgery, comprising the following steps: S1. Quality assessment and image enhancement processing are performed on the video frames of single-port laparoscopic surgery, and multi-scale feature extraction and fusion are performed using a hierarchical visual encoder to obtain enhanced feature maps. S2. For each device instance in the enhanced feature map, calculate the occlusion confidence score based on the spatial consistency change and relative area reduction of the visible mask in adjacent frames, and use the score exceeding a preset threshold as the trigger condition for the occlusion event. S3. A dual-branch memory management mechanism consisting of a propagation memory bank and an occlusion protection memory bank is adopted to manage the historical features of the device instance. The propagation memory bank continuously stores the device instance features of the unoccluded frames. The occlusion protection memory bank freezes and saves the complete morphological features of several frames before the occlusion occurs when the occlusion event is triggered, and releases them after the occlusion ends. S4. Using a weighted combination of propagation memory features and occlusion protection memory integrity features, the enhanced feature map is enhanced across frames through a memory attention mechanism to obtain a memory-enhanced feature map. The weight of the occlusion protection memory features decreases as the duration of occlusion increases. S5. Construct all instrument instances in the video frame into a directed graph, initialize node features with the appearance and boundary features of each instance, use the occlusion and stacking probability between nodes as the weight of the directed edges, and use a graph convolutional network to perform neighborhood aggregation inference to obtain the occlusion and stacking node features. S6. A parallel dual-head mask decoder is used to decode the memory-enhanced feature map. The visible mask terminal outputs the visible area mask of each instrument instance. The complete mask terminal uses the complete shape feature and the occlusion layer node feature as additional inputs to output the complete shape mask of the instrument, and outputs the difference between the complete shape mask and the visible area mask as the occluded area mask.

[0023] like Figure 2As shown, this invention makes targeted modifications to the SAM2 (Segment Anything Model 2) framework to construct an occlusion-aware segmentation model for single-port laparoscopic surgery scenarios. SAM2 is a general video object segmentation framework with a cross-frame temporal memory propagation mechanism. Its memory bank adopts a first-in-first-out rolling write strategy, continuously writing the mask features of historical frames into the memory bank for retrieval in subsequent frames. However, if SAM2 is directly used in amodal segmentation scenarios without any modification to the memory mechanism, incomplete masks generated during occlusion will be continuously written into the memory bank over time, overwriting the complete morphological features saved before occlusion. When the Amodal Head performs shape completion, the historical features it can retrieve are all the incomplete state of the instruments during occlusion. The shape prior is systematically contaminated, and the network will be unable to obtain effective complete morphological references from the memory bank. The completion performance will continuously degrade as the occlusion duration increases. Therefore, combining SAM2 and Amodal segmentation cannot be achieved simply by stacking two existing technologies; an inventive modification to the memory management mechanism is necessary. The core lies in independently isolating the complete morphological features before occlusion from the normal write stream, protecting them in a frozen manner, and injecting them in a controlled manner during the decoding stage to ensure reliable prior input for the shape completion path. This invention addresses this core requirement by adding an occlusion-aware dual-branch memory management module (including a propagation memory bank PMB and an occlusion protection memory bank OGMB), a single-aperture multi-instrument occlusion relationship graph module (ORG), and a non-modal parallel dual-head mask decoder (Modal Head and Amodal Head). These three modules work collaboratively to achieve simultaneous segmentation of the visible and occluded areas of surgical instruments. The overall processing flow is: video frame preprocessing and feature extraction → dual-branch memory management and occlusion detection → occlusion relationship graph construction and graph convolutional inference → memory attention feature enhancement → dual-head decoding and mask output.

[0024] In one embodiment of the present invention, step S1 includes: For each frame of the input surgical video sequence First, frame quality assessment is performed. The Laplacian gradient variance is used as the frame sharpness score for the quality assessment, and the frame sharpness score is calculated for the t-th frame. : ; in for The second-order Laplace gradient plot, This is for variance calculation. Below the preset resolution threshold Frames are classified as ambiguous frames. Ambiguous frames do not participate in the writing of the propagation memory and the occlusion protection memory, but they still participate in the inference process to maintain temporal continuity. The reason for setting this gating mechanism is that severely ambiguous frames caused by rapid instrument movement or smoke contamination during surgery have low mask prediction confidence. If they are written to the memory, they will introduce noise features and interfere with cross-frame context retrieval of subsequent frames.

[0025] Frames that pass quality assessment undergo the following three image enhancement steps: First, Limit Contrast Adaptive Histogram Equalization (CLAHE) is performed to improve local underexposure caused by uneven distribution of endoscopic light sources in surgical scenes, enhancing texture contrast in instrument boundary areas. Second, dehazing based on smoke density estimation is performed to restore clear visual features to frames affected by electrosurgical smoke or irrigation fluid disturbances. Finally, radial distortion geometric correction is performed to eliminate the inherent barrel distortion of the endoscope lens, ensuring that the geometry of the instruments in feature space matches their true physical form. The preprocessed image is obtained after these steps.

[0026] The preprocessed image is fed into a hierarchical visual encoder, which extracts feature maps at four resolution scales. s=1,2,3,4, corresponding to four feature levels from high resolution to low resolution. Shallow features preserve the details of the instrument's edges and contours, while deep features capture semantic category information. Feature maps at each scale are then fed into a Feature Pyramid Network (FPN) for top-down, step-by-step cross-scale fusion: deep semantic features are upsampled and added to shallow detail features level by level, fusing high-level semantics with low-level spatial localization information, ultimately outputting an enhanced feature map of uniform resolution. .

[0027] The design of step S1 decouples the quality control of the original frame from the multi-scale feature extraction. The quality gating ensures that the features entering the memory have sufficient reliability, while FPN fusion provides a feature representation that balances semantic and spatial accuracy for subsequent occlusion detection and mask decoding.

[0028] like Figure 3 As shown, in one embodiment of the present invention, step S2 includes: For the i-th device instance in the current frame t, predict the mask using the visible area of ​​the current frame. Predicted mask of visible region from the previous frame Calculate occlusion confidence score : ; in The cross-union ratio of the masks of adjacent frames measures the spatial consistency of the masks between adjacent frames. When occlusion occurs, the spatial distribution of the visible area of ​​the device will change abruptly, and this term will increase accordingly. and 1. The visible mask area of ​​device instance i in the current frame and the previous frame are respectively. The second term measures the relative reduction of the mask area. When the device is covered by other devices, the visible area will drop sharply, and this term will increase accordingly. This is the weighting coefficient for the two indicators.

[0029] The occlusion event determination adopts a dual threshold mechanism: when Exceeding the occlusion detection threshold When an occlusion event is triggered, record the start time of the occlusion. ;when Persistently below the recovery threshold When the occlusion has exceeded a certain number of consecutive frames, the occlusion is determined to have ended and the occlusion protection memory is released. The reason for using an asymmetric threshold is that there is often a brief period of mask instability in the early stage of the re-exposure of the device during the occlusion recovery process. If a single threshold is used to determine the recovery, it will lead to frequent false switching of the occlusion state. Introducing a recovery threshold lower than the detection threshold and requiring the condition to be met for several consecutive frames can filter out such brief fluctuations and make the determination of the occlusion state more robust.

[0030] In one embodiment of the present invention, step S3 includes: After each frame of inference, the propagation memory (PMB) determines whether the current frame meets the write condition: segmentation confidence. Above the reliability threshold And occlusion confidence Below the occlusion detection threshold In other words, the device instance features of the current frame are written into the PMB only when both conditions are met: the prediction is reliable and the device is not occluded. The PMB stores historical frames in a rolling order to ensure that the memory bank always contains the feature distribution of the device in a normal, visible state, providing a stable cross-frame context reference for the memory attention mechanism.

[0031] The Occlusion Protection Memory (OGMB) activates a protection mechanism when an occlusion event is triggered: based on the occlusion start time. Based on this, continuous freezing preservation The complete set of morphological features of a frame ,in For the i-th device instance in the th... The complete morphological features of a frame To preserve frame counts, the occlusion protection memory remains frozen during the duration of the occlusion event, preventing any new writes and releasing it after the occlusion ends. This freezing mechanism fundamentally blocks the path for incomplete mask features to enter the OGMB during occlusion: once occlusion occurs, the OGMB locks onto a complete shape record of several frames prior to the occlusion, containing the complete outline and shape prior of the instrument under unoccluded conditions, providing a reliable reference for subsequent shape completion of the Amodal Head. This design differs fundamentally from SAM2's native single FIFO memory: SAM2's memory treats occluded and unoccluded frames equally; incomplete features during occlusion continuously replace earlier complete shape records, and the shape prior available for querying in the Amodal Head systematically degrades as occlusion progresses. The OGMB, through its freezing mechanism, closes this degradation path, ensuring that the quality of the shape prior does not change with the duration of occlusion.

[0032] In one embodiment of the present invention, step S4 includes: To enhance feature maps The feature vector of each spatial location is used as the query to propagate the content stored in the memory. With time-decayed weighted complete morphological feature set The concatenated content along the sequence dimension consists of keys and values; multi-head cross-attention computation is then performed. ; in To enhance memory feature maps, This indicates concatenation along the sequence dimension. This represents the multi-head cross-attention calculation. It is a complete set of morphological features weighted by time-decaying. Defined as: ; in The duration of occlusion, where t is the current time, and the temporal decay weight. Defined as: ; in This is the decay rate hyperparameter. The temporal decay design reflects a practical principle: in the early stages of occlusion, the complete morphological features stored in the OGMB are very close to the current state of the instrument in time and can serve as reliable shape priors; as the duration of occlusion increases, the instrument may undergo pose changes during the occlusion period, and the correlation between the early frozen features and the current true state decreases. At this point, their weight in the attention mechanism should be reduced, and more reliance should be placed on the current visual features from the visible phase in the PMB. Decay weights Follow The feature increases monotonically, approaching 1 during short-term occlusion, allowing OGMB features to fully participate in attention calculation; during long-term occlusion, it approaches 0, and the model automatically degenerates into a conservative prediction mode that relies solely on PMB. This adaptive mechanism does not require manual setting of hard switching conditions.

[0033] like Figure 4 As shown, in one embodiment of the present invention, step S5 includes: First, node features are initialized for each instrument instance. For the i-th instrument instance, its features in the enhanced feature map are initialized. The local feature maps extracted by the Region of Interest Alignment (RoI Align) operation are compressed to a fixed dimension using global average pooling to obtain the appearance feature vector. The visible region mask boundary is extracted by the gradient operator, flattened, and then mapped to a fixed dimension through a fully connected layer to obtain the boundary feature vector. The initial node feature vector is obtained by concatenating the two and then projecting them linearly. Appearance features capture the category and appearance information of the device, while boundary features encode the shape information of the currently visible outline of the device. The two complement each other and together describe the perceptual state of the node in the current frame.

[0034] Then, a directed graph is constructed and edge weights are calculated. For each directed edge (i,j) in the directed graph, the node feature vectors of instrument instances i and j are extracted. And the occlusion boundary features obtained after convolutional pooling of the spatially overlapping regions of the two. The three features are concatenated and then mapped through a two-layer fully connected network to scalar edge weights representing the probability that instance i occludes instance j. : ; in and These are the learnable weight matrices and biases of a two-layer fully connected network, respectively. This is the Sigmoid activation function. This represents the probability that instance i is above instance j in the spatial stacking relationship (i.e., i occludes j). The direction of the directed edge encodes the directionality of the stacking relationship, which is key information that undirected graphs cannot express: knowing that two instruments overlap is not enough to infer which one is on the upper layer, while the stacking direction directly determines the direction and range that need to be restored when the shape of the occluded instrument is completed.

[0035] Then, neighborhood aggregation inference is performed using a graph convolutional network. The node feature update rule for the layer is: ; in, For the first The node feature vector of instance i in layer , Let i be the set of all neighboring nodes of a directed graph that have directed edges pointing to node i. and For the first The learnable weight matrix and bias of each layer. After passing through two layers of graph convolutional network, the output node features of each instance are the occlusion layer node features. It incorporates structural information from stacked neighboring nodes (i.e., other devices that are spatially superimposed on the device), which explicitly encodes the approximate location and extent constraints of the occluded area, providing geometric guidance for subsequent shape completion of the Amodal Head.

[0036] The occlusion relationship graph module in step S5 transforms the spatial stacking relationship between multiple instruments from an implicit feature distribution into an explicit structural expression on a directed graph. This solves the problem of complex stacking relationships when multiple instruments are densely intersecting in a single-port laparoscopic scenario, and the difficulty in inferring the direction of the occluded area by relying solely on pixel-level features.

[0037] like Figure 5 As shown, in one embodiment of the present invention, step S6 includes: Parallel dual-head mask decoder for memory-enhanced feature maps During decoding, the two substations share the encoder backbone but have independent decoding paths.

[0038] As can be seen, the Modal Head uses the original lightweight mask decoding structure of SAM2 to query tokens using standard objects. For query, The keys and values ​​are decoded by multiple Transformers and then mapped to a visible mask weight vector by a Multilayer Perceptron (MLP). After inner product and sigmoid activation, the output is the visible region mask of device instance i. .

[0039] The Amodal Head enhances the decoding input by introducing shape prior features and occlusion stacking constraint features, in addition to the standard decoding path. First, shape prior features are extracted from the complete shape features stored in the occlusion protection memory. : ; The meanings of each symbol are consistent with those in step S4. In OGMB The temporally decayed weighted average of the complete morphological features of a frame reduces the impact of single-frame noise by averaging the complete morphological features across multiple frames. The temporal decay weights ensure that the influence of shape priors is moderately reduced as the occlusion duration increases. When the instance is not occluded... Taking the zero vector, the Amodal Head degenerates into the same ordinary segmentation pattern as the Modal Head, activating the shape prior path only after occlusion occurs.

[0040] shape prior features The occlusion layer node features output in step S5 splicing, via linear projection layer After projecting to the query token dimension, it is compared with the object query token. Add them together to form the extended query token: ; To extend query tokens Enhanced feature maps for querying and memorization The keys and values, after being decoded by multiple Transformers, are mapped to a complete mask weight vector through an MLP with the same structure as the MLP in the Modal Head but with completely independent parameters. ,and After inner product and sigmoid activation, the complete shape mask is output. .

[0041] The mask for the occluded region is directly derived from the difference between the mask for the complete shape and the mask for the visible region: ; in This is pixel-by-pixel multiplication. The region with a positive median value represents the spatial distribution of instrument i in the current frame that is occluded by other objects but inferred by this method. It can be superimposed on the video frame in the form of a semi-transparent overlay or a dashed outline in the surgical assistance display interface to indicate the complete location of the occluded instrument to the surgeon.

[0042] The parallel dual-head structure and extended query token mechanism in step S6 enable the Amodal Head to additionally acquire shape priors from the memory bank and geometric constraints from the occlusion relation graph without changing the segmentation accuracy of the Modal Head. Both types of information are superimposed on the standard query token in a residual addition manner, without interfering with the original decoding path of the Modal Head. This achieves decoupling and parallelism between visible segmentation and complete shape segmentation. The output results include three types of masks: visible area mask, complete shape mask, and occluded area mask, with complete information hierarchy, suitable for various auxiliary display needs of surgical navigation.

[0043] The model training process of this invention is as follows.

[0044] The training data used three publicly available surgical video datasets: EndoVis2017, EndoVis2018, and AIS-Lap. EndoVis2017 and EndoVis2018 provide multi-instrument surgical videos and frame-by-frame segmentation and annotation of visible instrument regions. The AIS-Lap dataset additionally provides amodal segmentation annotations for instruments. This contains the complete shape truth value of the occluded region. The truth value label for the occluded region is provided by... Export directly.

[0045] The instrument stacking order labels required for occlusion relationship graph module training Automatically derived from amodal annotations: If the complete shape mask of device i Visible area mask of device j There is overlap, and If there is no corresponding pixel in the overlapping area, then j is determined to occlude i, and denoted as j. , This derivation rule requires no additional manual layering annotations and can be automatically generated directly from the difference relationship between the amodal and modal annotations.

[0046] For samples in EndoVis2017 / 2018 that only have modal annotations and lack amodal annotations, a synthetic occlusion data augmentation method is used to expand the paired amodal ground truth labels: two frames of unoccluded instrument regions within the same video sequence are randomly selected, and the instrument mask of one frame is superimposed onto the other frame in a random stacking order to synthesize the occluded image and the corresponding modal mask, amodal mask, and stacking order label. This generates paired synthetic occlusion training samples in batches, making up for the scarcity of amodal annotations in the public dataset.

[0047] The total training loss is composed of a weighted average of four component losses: ; in The output of the Modal Head is supervised by the sum of the BCE loss and the Dice loss, which is the visible mask segmentation loss. For the complete shape mask loss, the sum of the BCE loss and the Dice loss is used, and the Amodal Head is supervised to annotate the complete shape. Fit: ; To compensate for the consistency loss of the occluded region, two constraints are imposed on the derived occluded region mask: the first is the ground truth of the occluded region. The first term is the standard segmentation supervision; the second term is a hard constraint penalty term, which forces that the predicted area of ​​the complete mask is not less than the predicted area of ​​the visible mask (because the complete shape must include the visible area): ; Where (u,v) are the spatial pixel coordinates on the feature map. For the occlusion graph loss, the supervised ORG module predicts the directed edge weights, labeled in cascading order. For true values, use binary cross-entropy loss: ; in This is the set of all spatially overlapping instrument instance pairs in the current frame. The weights of each loss term are set to... ,in The reason is that the complete shape that the Amodal Head needs to predict includes invisible occluded areas, making the prediction more difficult than that of the Modal Head, and requiring stronger gradient supervision signals.

[0048] Model training is performed sequentially in three stages to avoid gradient conflicts between new modules and pre-trained backbones, allowing each module to converge gradually from easy to difficult. The first stage is the bone intervention warm-up stage: all parameters of the Hiera image encoder are frozen, and only the newly added parameters in the dual-branch memory management module and the occlusion graph module are trained. This allows these two modules to adapt to the feature distribution of the surgical scene in a fixed feature space. This stage uses the EndoVis2017 and EndoVis2018 datasets, employs the AdamW optimizer, and has an initial learning rate of [missing information]. Weight decay coefficient The batch size is 8, and the training lasts for 20 epochs. The second stage is the dual-head decoder training stage: the parameters of the last two stages of the Hiera encoder are unfrozen, and joint training with the Amodal Head is added to the weights of the first stage. The AIS-Lap dataset and synthetic occlusion augmentation data are introduced, focusing on optimizing the shape completion ability of the Amodal Head in occluded scenes. The initial learning rate is adjusted to... A cosine annealing scheduling strategy is adopted, with the minimum learning rate decaying to The training process lasts for 30 epochs. The third stage is the end-to-end fine-tuning stage: all network parameters are unfrozen, and the network is trained with a very small learning rate. The complete network is jointly optimized end-to-end to make the feature representations of each module compatible with each other. In this stage, the network is trained on all datasets for a total of 10 epochs.

[0049] During training, the following data augmentation operations were performed on the input frame sequence: random horizontal flip (flip probability 0.5); color jitter (brightness, contrast, and saturation randomly perturbed within a certain range) to simulate different endoscopic light source intensities; random Gaussian blur to simulate image acquisition noise and smoke obstruction; and random affine transformation (small-range rotation and translation) to simulate the slight movement of the endoscope during surgery. All of the above augmentation operations used the exact same random parameters for all frames within the same frame sequence to maintain temporal consistency between frames.

[0050] In one example, the method of the present invention is compared with SAM2 and other existing models, and the data is as follows: Table 1 Performance comparison between the method of this invention and the SAM2 series models On the EndoVis2017 dataset, the proposed method was compared with the SAM2 baseline. The J index improved from 85.45% to 90.12%, an increase of 4.67 percentage points; the Dice coefficient improved from 89.35% to 93.64%, an increase of 4.29 percentage points. These improvements are mainly attributed to the dual-branch memory management mechanism—the propagation memory bank (PMB) specifically stores high-confidence unoccluded frame features, effectively preventing occluded frame features from contaminating the memory bank. This provides higher-quality reference frames during the temporal feature aggregation stage, thereby significantly improving the fine segmentation capability of instrument boundaries.

[0051] On the EndoVis2018 dataset, the method of this invention ranked first among the three SAM series comparison methods with a CIoU of 87.3%. EndoVis2018 contains more instrument occlusion and overlap scenarios, which is the core challenge that this invention focuses on. Compared with the SAM2 baseline (82.2%), it is 5.1 percentage points higher, and compared with MedSAM2 (83.7%) and SurgSAM2 (84.4%), which are also based on SAM2 improvements, it is 3.6 and 2.9 percentage points higher, respectively. The reason is that: the occlusion relationship directed graph (ORG) module explicitly models the stacking direction of multiple instruments through graph convolutional networks, which makes up for the lack of perception of the topological relationship of occlusion between instruments in existing methods; the Amodal Head in the parallel dual-head mask decoder can simultaneously infer the complete shape of the occluded instrument, while methods such as MedSAM2 and SurgSAM2 only output the visible area mask. The occlusion frequency of EndoVis2018 is significantly higher than that of EndoVis2017. The improvement of this invention relative to the SAM2 baseline on this dataset (5.1%) is also greater than that of EndoVis2017 (4.67%), which is completely consistent with the technical route of this invention designed specifically for occlusion scenarios.

[0052] The present invention also provides a deep learning-based single-port laparoscopic surgical field occlusion segmentation system, which includes the following functional modules.

[0053] The image preprocessing and feature extraction module is responsible for quality assessment and image enhancement of the input surgical video frames. Severely blurred frames are filtered out by frame sharpness scoring. For frames that pass the quality assessment, contrast-limited adaptive histogram equalization, dehazing, and radial distortion geometric correction are performed sequentially. Then, a hierarchical visual encoder is used to extract multi-scale feature maps at four resolution scales. These feature maps are then fused across scales by a feature pyramid network to output enhanced feature maps with uniform resolution, providing high-quality visual feature input for subsequent modules.

[0054] The occlusion detection and dual-branch memory management module is responsible for determining the occlusion status of each device instance and maintaining two independent memory storage structures. This module calculates the occlusion confidence score based on the spatial consistency change of adjacent frame masks and the degree of visible area reduction, and uses a dual-threshold mechanism to robustly determine the start and end of occlusion events. The propagation memory bank writes high-confidence unoccluded frame features according to conditions after each frame inference, and maintains the normal state feature sequence of the device in a rolling manner. The occlusion protection memory bank immediately freezes and saves the complete morphological features of several frames before occlusion when an occlusion event is triggered, and clears and releases them after occlusion ends to prevent incomplete features during occlusion from contaminating the shape prior storage.

[0055] The memory attention module is responsible for enhancing the current frame's augmented feature map across frames using the contents of the dual-branch memory. This module concatenates the contents of the propagation memory with the contents of the occlusion protection memory, which are weighted by temporal decay, to form a joint key-value sequence. Using the spatial features of each location in the augmented feature map as the query, it performs multi-head cross-attention calculation, incorporating the appearance and morphological information of the device in historical frames into the current frame's features, and outputs the memory-enhanced feature map. The weights of the occlusion protection memory decrease exponentially with the duration of occlusion, allowing the model to fully utilize shape priors during short-term occlusion and rely on current visual features during long-term occlusion.

[0056] The occlusion relationship graph module is responsible for explicitly modeling the spatial stacking relationships between all instrument instances in the current video frame. This module initializes directed graph nodes with the appearance features and visible boundary features of each instance, jointly calculates the directed edge weights with the features of two instance nodes and the boundary features of overlapping regions, and aggregates the features of stacked neighbor nodes through a two-layer graph convolutional network. This allows each instrument node to absorb the structural information of the instrument instances pressing on it while acquiring its own features, and outputs occlusion stacked node features carrying stacking direction constraints, providing geometric direction guidance for complete morphological decoding.

[0057] The parallel dual-head mask decoding module is responsible for decoding the memory-enhanced feature map and simultaneously outputting the visible region mask and the complete shape mask. This module includes a parallel visible mask terminal and a complete mask terminal: the visible mask terminal outputs the segmentation result of the currently visible region of the instrument using the standard decoding path; the complete mask terminal adds shape prior features extracted from the occlusion protection memory and stacked node features output by the occlusion relationship graph module to the standard query token to form an extended query token, which outputs the complete shape mask of the instrument including the occluded region through an independent decoding path, and outputs the pixel-by-pixel difference between the complete shape mask and the visible region mask as the occluded region mask, ultimately providing three types of mask results for the surgical assistance display system.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning-based method for single-port laparoscopic surgical field occlusion segmentation, characterized in that, include: S1. Quality assessment and image enhancement processing are performed on the video frames of single-port laparoscopic surgery, and multi-scale feature extraction and fusion are performed using a hierarchical visual encoder to obtain enhanced feature maps. S2. For each device instance in the enhanced feature map, calculate the occlusion confidence score based on the spatial consistency change and relative area reduction of the visible mask in adjacent frames, and use the score exceeding a preset threshold as the trigger condition for the occlusion event. S3. A dual-branch memory management mechanism consisting of a propagation memory bank and an occlusion protection memory bank is adopted to manage the historical features of the device instance. The propagation memory bank continuously stores the device instance features of the unoccluded frames. The occlusion protection memory bank freezes and saves the complete morphological features of several frames before the occlusion occurs when the occlusion event is triggered, and releases them after the occlusion ends. S4. Using a weighted combination of propagation memory features and occlusion protection memory integrity features, the enhanced feature map is enhanced across frames through a memory attention mechanism to obtain a memory-enhanced feature map. The weight of the occlusion protection memory features decreases as the duration of occlusion increases. S5. Construct all instrument instances in the video frame into a directed graph, initialize node features with the appearance and boundary features of each instance, use the occlusion and stacking probability between nodes as the weight of the directed edges, and use a graph convolutional network to perform neighborhood aggregation inference to obtain the occlusion and stacking node features. S6. A parallel dual-head mask decoder is used to decode the memory-enhanced feature map. The visible mask terminal outputs the visible area mask of each instrument instance. The complete mask terminal uses the complete shape feature and the occlusion layer node feature as additional inputs to output the complete shape mask of the instrument, and outputs the difference between the complete shape mask and the visible area mask as the occluded area mask.

2. The method according to claim 1, characterized in that, In step S2, the occlusion confidence score is calculated as follows: For the i-th device instance in the current frame t, predict the mask using the visible area of ​​the current frame. Predicted mask of visible region from the previous frame The occlusion confidence score is defined by the weighted sum of the intersection-union complement and the relative reduction in mask pixel area. : ； Where the subscript t-1 represents the previous frame, for pixel area This is the predicted mask for the visible area of ​​the previous frame. for pixel area The cross-joint ratio (CLU) of the two frames' masks. This is the balance coefficient; when Exceeding the occlusion detection threshold When an occlusion event is triggered, record the start time of the occlusion. ; when Persistently below the recovery threshold When the occlusion exceeds a certain number of consecutive frames, the occlusion is determined to have ended and the occlusion protection memory is released. .

3. The method according to claim 2, characterized in that, In step S3, the occlusion protection memory, when the occlusion event is triggered, uses the occlusion start time as the starting time. Based on this, continuous freezing preservation The complete set of morphological features of a frame ,in For the i-th device instance in the th... The complete morphological features of a frame To preserve frame counts, the occlusion protection memory remains frozen during the duration of the occlusion event and is cleared and released after the occlusion ends.

4. The method according to claim 3, characterized in that, In step S4, the specific implementation of the memory attention mechanism is as follows: To enhance feature maps The feature vector of each spatial location is used as the query to propagate the content stored in the memory. With time-decayed weighted complete morphological feature set The concatenated content along the sequence dimension consists of keys and values; multi-head cross-attention computation is then performed. ； in, To enhance memory feature maps, , The duration of the occlusion is given by t, where t is the current time. This indicates concatenation along the sequence dimension. This represents multi-head cross-attention calculation with time-decaying weights. Defined as: ； in This is the decay rate hyperparameter.

5. The method according to claim 1, characterized in that, In step S5, the probability of occlusion stacking between nodes is calculated as follows: For each directed edge (i,j) in the directed graph, extract the node feature vectors of instrument instances i and j respectively. , And the occlusion boundary features obtained after convolutional pooling of the spatially overlapping regions of the two. The three features are concatenated and then mapped through a two-layer fully connected network to scalar edge weights representing the probability that instance i occludes instance j. : ； in and These are the learnable weight matrices and biases of a two-layer fully connected network, respectively. This is the Sigmoid activation function.

6. The method according to claim 5, characterized in that, In step S5, the node feature update rule for neighborhood aggregation inference in the graph convolutional network is as follows: Each node sums its own features with the features of its directed neighbors according to the corresponding edge weights, and then performs a learnable linear transformation and nonlinear activation to obtain the node features for the next layer; The layer update rules are as follows: ； in, For the first The node feature vector of instance i in layer , Let i be the set of all neighboring nodes of a directed graph that have directed edges pointing to node i. and For the first The learnable weight matrix and bias of the layer; After passing through a two-layer graph convolutional network, the output node features of each instance are the occlusion layer node features.

7. The method according to claim 4, characterized in that, In step S6, the complete masking terminal constructs extended query tokens for each instrument instance. As decoding input: Extract shape prior features from the complete morphological features stored in the occlusion protection memory bank. : ； shape prior features Features of overlapping nodes with occlusion splicing, via linear projection layer After projecting to the query token dimension, it is compared with the object query token. Add them together to form the extended query token: ； To extend query tokens To query and memorize the enhanced feature map as keys and values, a complete morphological mask of the device is obtained through multi-layer Transformer decoding. When the instance is not occluded Take the zero vector.

8. The method according to claim 1, characterized in that, In step S1, the quality assessment uses the Laplacian gradient variance as the frame sharpness score for the t-th frame image. Calculate frame sharpness score : ； in for The second-order Laplace gradient plot, For variance calculation; Below the preset resolution threshold The frame is determined to be a fuzzy frame. Fuzzy frames do not participate in the writing of the propagation memory and the occlusion protection memory, but they still participate in the inference process to maintain the temporal continuity. The frames that pass the quality assessment are sequentially processed by contrast-limited adaptive histogram equalization, dehazing based on smoke density estimation, and radial distortion geometric correction to obtain preprocessed images; The preprocessed image is used to extract feature maps at four resolution scales via a hierarchical visual encoder. s=1,2,3,4, and then the feature pyramid network performs top-down cross-scale fusion of feature maps at each scale to output an enhanced feature map with uniform resolution. .

9. The method according to claim 1, characterized in that, In step S5, the specific method for initializing node features with the appearance and boundary features of each instance is as follows: For the i-th device instance, its enhancement feature map The local feature maps extracted by the region of interest alignment operation are compressed to a fixed dimension using global average pooling to obtain the appearance feature vector. The visible region mask boundary is extracted by the gradient operator, flattened, and then mapped to a fixed dimension through a fully connected layer to obtain the boundary feature vector. The two are then combined to form the initial features of the node. As features of nodes in each layer of a graph convolutional network Initial input.

10. A deep learning-based single-port laparoscopic surgical field occlusion segmentation system, characterized in that, The system is used to implement the method according to any one of claims 1-9, the system comprising: The image preprocessing and feature extraction module is used to perform quality assessment and image enhancement processing on single-port laparoscopic surgery video frames, and uses a hierarchical visual encoder to perform multi-scale feature extraction and fusion to obtain enhanced feature maps. The occlusion event detection module is used to calculate the occlusion confidence score for each device instance in the enhanced feature map based on the spatial consistency change and relative area reduction of the visible mask in adjacent frames. The score exceeding a preset threshold is used as the trigger condition for the occlusion event. The dual-branch memory management module is used to manage the historical features of device instances using a dual-branch memory management mechanism consisting of a propagation memory bank and an occlusion protection memory bank. The propagation memory bank continuously stores the device instance features of unoccluded frames, while the occlusion protection memory bank freezes and saves the complete morphological features of several frames before the occlusion occurs when an occlusion event is triggered, and releases them after the occlusion ends. The memory attention module is used to perform cross-frame context enhancement on the enhanced feature map by weighted combination of propagation memory features and occlusion protection memory integrity features. The weight of occlusion protection memory integrity features decreases as the duration of occlusion increases. The occlusion relationship graph module is used to construct a directed graph of all instrument instances in the video frame. The node features are initialized with the appearance and boundary features of each instance. The occlusion and stacking probability between nodes is used as the weight of the directed edge. A graph convolutional network is used for neighborhood aggregation inference to obtain the occlusion and stacking node features. The parallel dual-head mask decoding module is used to decode the memory-enhanced feature map using a parallel dual-head mask decoder. The visible mask terminal outputs the visible area mask of each instrument instance, while the complete mask terminal outputs the complete shape mask of the instrument with the complete shape feature and the occlusion layer node feature as additional inputs and outputs the occluded area mask. The difference between the complete shape mask and the visible area mask is output as the occluded area mask.

Images