Laparoscope video semantic segmentation method based on weighted scanning strategy SSM

Abstract

The invention discloses a laparoscope video semantic segmentation method based on a weighted scanning strategy SSM, and the method comprises the steps: inputting a to-be-segmented laparoscope image into a laparoscope video semantic segmentation network model, so as to enable the model to output a segmentation prediction result; wherein the initial module performs initial feature extraction on a laparoscope image to obtain initial features; the double-branch encoder obtains local features and global features based on the initial features through local feature extraction and weighted multidirectional scanning strategies; performing feature fusion on the local features and the global features by using a cross attention mechanism to obtain first fusion features and second fusion features; the double-branch decoder decodes the first fusion feature and the second fusion feature to obtain a local decoding feature map and a global decoding feature map; and the gating mechanism module performs semantic segmentation by utilizing attention perception based on the local decoding feature map and the global decoding feature map to obtain a segmentation prediction result, so that an accurate and robust segmentation effect is realized.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a semantic segmentation method for laparoscopic videos based on the weighted scanning strategy SSM (State Space Model). Background Technology

[0002] Laparoscopic minimally invasive surgery videos are crucial visual records of surgical procedures, containing a wealth of surgical knowledge that can be used for various tasks such as surgical decision support, intraoperative image segmentation, and surgical navigation. Currently, intraoperative judgment and operation in laparoscopic surgery mainly rely on physicians' real-time observation of video footage for decision-making. Therefore, accurately identifying and segmenting key anatomical structures and surgical instruments from laparoscopic videos is a critical prerequisite for achieving intelligent surgical assistance. However, due to the similarity in morphology and structure of some tissues and organs in the surgical scene, effective differentiation is difficult; tissue deformation and local occlusion can easily occur due to changes in physician operation and lighting during surgery; and surgical instruments are typically distributed obliquely in the field of view. Furthermore, existing methods largely rely on physicians manually interpreting video content, resulting in low efficiency, high subjectivity, and difficulty in large-scale application. Therefore, based on deep learning methods, an efficient and accurate method for parsing laparoscopic surgical videos has been developed, which is of great significance for advancing the intelligent development of minimally invasive surgery.

[0003] Currently, deep learning-based automatic segmentation methods for laparoscopic images mainly fall into three categories. First, some methods use U-Net (a fully convolutional neural network) as the basic network framework, enhancing feature representation capabilities by introducing dense connections and attention mechanisms. However, due to the local limitations of convolutional computation, these methods cannot capture long-range dependency information. Second, some methods use Transformer (a deep learning architecture based on attention mechanisms) as the core architecture, utilizing self-attention mechanisms to extract global contextual information. However, the computational complexity of its self-attention mechanism increases quadratically with image resolution, limiting its practical application in high-resolution laparoscopic images. Third, some methods use Mamba (a structured state-space sequence model) as the basic framework, extracting global features with linear complexity. However, its core scanning strategy only performs spatial modeling along the horizontal and vertical directions, failing to fully capture oblique spatial distribution features and channel dependencies, thus limiting its application in laparoscopic surgical videos. Furthermore, some tissues and organs in laparoscopic surgical images exhibit high visual similarity, variable morphology, and are subject to interference from instrument occlusion and mirror reflections. In addition, the surgical instruments are mostly distributed at an angle, and the existing Mamba is unable to meet the segmentation requirements of the surgical instruments.

[0004] Therefore, how to provide a semantic segmentation method for laparoscopic videos that simultaneously possesses the ability to perceive local details, understand global context, and model multi-directional spatial features, thereby achieving accurate and robust segmentation results, has become an important issue. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a semantic segmentation method for laparoscopic videos based on the weighted scanning strategy (SSM).

[0006] The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a semantic segmentation method for laparoscopic videos based on a weighted scanning strategy (SSM), the semantic segmentation method for laparoscopic videos comprising: Obtain the laparoscopic image to be segmented; The laparoscopic image is input into a pre-trained laparoscopic video semantic segmentation network model so that the laparoscopic video semantic segmentation network model outputs segmentation prediction results; The laparoscopic video semantic segmentation network model includes an initialization module, a dual-branch encoder, a dual-branch decoder, and a gating mechanism module. The initial module is used to extract initial features from the laparoscopic image to obtain initial features; The dual-branch encoder is used to obtain local features and global features based on the initial features through local feature extraction and a weighted multi-directional scanning strategy; and to perform feature fusion on the local features and the global features using a cross-attention mechanism to obtain a first fused feature and a second fused feature. The dual-branch decoder is used to decode the first fused feature and the second fused feature respectively to obtain a local decoded feature map and a global decoded feature map; The gating mechanism module is used to perform semantic segmentation based on the local decoded feature map and the global decoded feature map, using attention perception to obtain segmentation prediction results.

[0007] Optionally, the dual-branch encoder includes multiple dual-branch encoder structural unit modules; each dual-branch encoder structural unit module includes a CB module, a local-global feature coupling unit module, and an HVB module; in each dual-branch encoder structural unit module, the input and output terminals of the CB module and the input and output terminals of the HVB module are all connected to the local-global feature coupling unit module.

[0008] Optionally, the CB module includes two cascaded residual sub-modules with identical structures.

[0009] Optionally, the HVB module includes a first normalization layer, an HSS2D layer, a second normalization layer, and a multilayer perceptron layer connected in sequence.

[0010] Optionally, the training method for the laparoscopic video semantic segmentation network model includes: Construct a dataset; the dataset includes multiple laparoscopic sample images and the standard segmentation results corresponding to each laparoscopic sample image; Each laparoscopic sample image is input into the initial laparoscopic video semantic segmentation network model, so that the initial laparoscopic video semantic segmentation network model outputs the predicted sample segmentation result corresponding to each laparoscopic sample image; A loss function is constructed based on the difference between the standard segmentation result and the predicted segmentation result corresponding to each laparoscopic sample image. The loss function is iteratively updated in the direction of reducing the difference until the number of iterations reaches a preset number or the loss function converges, thus obtaining the trained laparoscopic video semantic segmentation network model.

[0011] Optionally, the loss function includes a cross-entropy loss function and a deep supervision loss function; The cross-entropy loss function includes: ; in, This represents the cross-entropy loss function; Indicates the first Standard segmentation results corresponding to each laparoscopic sample image; Indicates the first Predicted sample segmentation results for each laparoscopic sample image; The deep supervision loss function includes: ; in, Represents the deep supervised loss function; This represents the segmentation loss value generated by the HVMamba branch; This represents the segmentation loss value generated by the CNN branch; This represents the segmentation loss value generated by the gating mechanism module.

[0012] Secondly, the present invention provides a semantic segmentation device for laparoscopic videos based on a weighted scanning strategy (SSM), the semantic segmentation device for laparoscopic videos comprising: The acquisition module is used to acquire the laparoscopic image to be segmented; The segmentation module is used to input the laparoscopic image into a pre-trained laparoscopic video semantic segmentation network model, so that the laparoscopic video semantic segmentation network model outputs segmentation prediction results; The laparoscopic video semantic segmentation network model includes an initialization module, a dual-branch encoder, a dual-branch decoder, and a gating mechanism module. The initial module is used to extract initial features from the laparoscopic image to obtain initial features; The dual-branch encoder is used to obtain local features and global features based on the initial features through local feature extraction and a weighted multi-directional scanning strategy; and to perform feature fusion on the local features and the global features using a cross-attention mechanism to obtain a first fused feature and a second fused feature. The dual-branch decoder is used to decode the first fused feature and the second fused feature respectively to obtain a local decoded feature map and a global decoded feature map; The gating mechanism module is used to perform semantic segmentation based on the local decoded feature map and the global decoded feature map, using attention perception to obtain segmentation prediction results.

[0013] Thirdly, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a computer program stored in memory, it implements the steps described in any of the above-mentioned laparoscopic video semantic segmentation methods based on the weighted scanning strategy (SSM).

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in any of the above-mentioned laparoscopic video semantic segmentation methods based on weighted scanning strategy (SSM).

[0015] The present invention provides a semantic segmentation method for laparoscopic video based on a weighted scanning strategy (SSM). The dual-branch encoder in the semantic segmentation network model of laparoscopic video can be used to obtain local and global features based on initial features through local feature extraction and a weighted multi-directional scanning strategy. It can capture spatial features from all directions, effectively solve the segmentation difficulties caused by the oblique distribution of instruments, and significantly improve the discrimination ability of the model.

[0016] Furthermore, the dual-branch encoder can utilize a cross-attention mechanism to fuse local and global features. The gating mechanism module can perform semantic segmentation based on local and global decoded feature maps using attention perception. By dynamically fusing local detail information and global contextual information perception capabilities, and adaptively incorporating complementary features from different semantic dimensions, it improves the segmentation accuracy of laparoscopic anatomical structures and surgical instruments.

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a laparoscopic video semantic segmentation method based on a weighted scanning strategy (SSM) provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the laparoscopic video semantic segmentation network model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the dual-branch encoder provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the dual-branch encoder unit module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the HSS2D module provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of multiple scanning strategies provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the state space module of the weighted scanning strategy provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of the local-global feature coupling unit module provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of the dual-branch decoder module provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the gate control mechanism module provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the training process of the laparoscopic video semantic segmentation network model provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0020] To address the shortcomings of existing semantic segmentation methods for laparoscopic videos, such as a lack of local detail perception, global context understanding, and multi-directional spatial feature modeling capabilities, which hinders the achievement of accurate and robust segmentation results, this invention provides a semantic segmentation method for laparoscopic videos based on a weighted scanning strategy (SSM). See [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating a laparoscopic video semantic segmentation method based on a weighted scanning strategy (SSM) according to an embodiment of the present invention, which specifically includes the following steps: Step S101: Obtain the laparoscopic image to be segmented.

[0021] In this embodiment of the invention, the laparoscopic image to be segmented can be obtained by acquiring laparoscopic surgical video frames.

[0022] For example, by setting the number of sampling intervals, keyframes can be extracted from a video sequence at equal time intervals to obtain the laparoscopic image to be segmented.

[0023] Step S102: Input the laparoscopic image into the pre-trained laparoscopic video semantic segmentation network model so that the laparoscopic video semantic segmentation network model outputs the segmentation prediction result. The laparoscopic video semantic segmentation network model includes an initial module, a dual-branch encoder, a dual-branch decoder, and a gating mechanism module. The initial module is used to extract initial features from the laparoscopic image to obtain initial features. The dual-branch encoder is used to obtain local features and global features based on the initial features through local feature extraction and a weighted multi-directional scanning strategy. It then uses a cross-attention mechanism to fuse the local features and global features to obtain a first fused feature and a second fused feature. The dual-branch decoder is used to decode the first fused feature and the second fused feature to obtain a local decoded feature map and a global decoded feature map, respectively. The gating mechanism module is used to perform semantic segmentation based on the local decoded feature map and the global decoded feature map using attention perception to obtain the segmentation prediction result.

[0024] In this embodiment of the invention, the initial module is used to extract initial features from the laparoscopic image to obtain initial features.

[0025] In this embodiment of the invention, the initial module includes multiple convolutional neural networks and a downsampling module, which are placed at the beginning of the dual-branch encoder to map the input laparoscopic image to the feature space to obtain an initial feature map.

[0026] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the laparoscopic video semantic segmentation network model provided in the embodiment of the present invention. The branch encoder is used to obtain local features and global features based on the initial features through local feature extraction and weighted multi-directional scanning strategy; and to use the cross attention mechanism to fuse the local features and global features to obtain the first fused feature and the second fused feature.

[0027] A dual-branch decoder is used to decode the first fused feature and the second fused feature respectively to obtain a local decoded feature map and a global decoded feature map.

[0028] In this embodiment of the invention, the dual-branch encoder includes a downsampling (max pooling) operation, and the dual-branch decoder includes a corresponding convolution module and an upsampling (bilinear interpolation) operation.

[0029] Furthermore, the dual-branch encoder includes a CNN branch based on a residual structure and a structured vision Mamba branch based on a weighted scanning strategy (SSM), used to extract local detail features and model long-range dependencies (i.e., global features), respectively, as well as a local-global feature coupling unit (LoWR) branch. The CNN branch includes multiple CB (CNN Block) modules, the Mamba branch includes multiple HVB (Hypertext Delivery Vehicle) modules, and the LoWR branch includes multiple LoWR modules. Therefore, a CB module, a LoWR module, and an HVB module can be considered as a dual-branch encoder structural unit module, and the dual-branch encoder includes multiple dual-branch encoder structural unit modules. In each dual-branch encoder structural unit module, the input and output terminals of the CB module and the HVB module are connected to the LoWR module.

[0030] See Figure 3 , Figure 3 This is a schematic diagram of the structure of the dual-branch encoder provided in an embodiment of the present invention. The dual-branch encoder provided in the embodiment of the present invention will be described in detail below: In this embodiment of the invention, a CNN branch includes a stem and a plurality of sequentially connected CB modules located at its output. Here, stem represents a series of cascaded CNN modules.

[0031] See Figure 4 , Figure 4 This is a schematic diagram of the dual-branch encoder structural unit module provided in an embodiment of the present invention. The CB module consists of two cascaded residual sub-modules with identical structures. The input features of each residual sub-module are subjected to three cascaded convolution operations and then added to its own identity mapping to output deep features, which can effectively alleviate the gradient vanishing problem in deep networks. Specifically, the first residual submodule in the CB module includes a first convolutional layer, a second convolutional layer, and a third convolutional layer; wherein, the input of the first convolutional layer is the input feature. The output of the first convolutional layer is connected to the input of the second convolutional layer. The output of the second convolutional layer is connected to the input of the third convolutional layer and outputs features to the LG-FCU (Local-Global Feature Coupling Unit). The output of the third convolutional layer is connected to the input of the next residual submodule and the input of the first convolutional layer. The first and third convolutional layers are both cnov1 convolutional layers, and the second convolutional layer is a conv3 convolutional layer.

[0032] The second residual submodule includes a fourth, fifth, and sixth convolutional layer. The input of the fourth convolutional layer is connected to the output of the third convolutional layer; the output of the fourth convolutional layer is connected to the input of the fifth convolutional layer and receives the currently output features from the LG-FCU; the output of the fifth convolutional layer is connected to the input of the sixth convolutional layer; and the output of the sixth convolutional layer is connected to the input of the fourth convolutional layer and outputs the second fused feature. The fourth and sixth convolutional layers are both cnov1 convolutional layers, and the fifth convolutional layer is a conv3 convolutional layer.

[0033] in, Figure 3 In the dual-branch encoder structure unit module, "×2" indicates two CB modules connected in series, and "×15" indicates 15 CB modules connected in series.

[0034] In this embodiment of the invention, the structured vision Mamba branch based on the weighted scanning strategy SSM includes a PatchEmbedding module and a plurality of HVB modules connected in sequence at its output.

[0035] The Patch Embedding module is used to segment the initial input features into fixed-size patches and map each patch to a fixed-dimensional vector representation.

[0036] Each HVB module contains a first normalized layer (LN), an HSS2D layer, a second normalized layer, and a multilayer perceptron layer (MLP) connected in sequence.

[0037] The first normalization layer inputs the global features output by the HVB module from the previous stage. The first normalized layer's output is connected to the input of the HSS2D, which in turn is connected to the input of the second normalized layer. The second normalized layer's input is also connected to the first normalized layer's input. The second normalized layer's output is connected to the input of the MLP, which in turn is connected to the output of the HSS2D and input to the LG-FCU, ultimately outputting the first fused feature. .

[0038] In this embodiment of the invention, the calculation process of the HSS2D module is as follows, see below. Figure 5 , Figure 5 Here is a schematic diagram of the structure of the HSS2D module provided in this embodiment of the invention: Using a single convolutional layer, the features input to the HSS2D module are divided into two groups according to their channels to obtain the first group of features. Second group of features ,in, The first output feature map is obtained through convolution and SiLU activation function. : ; right A second feature map is obtained by performing a weighted selective scanning operation based on attention perception to extract spatial features from all directions. The details are as follows: See Figure 6 , Figure 6 This is a schematic diagram of multiple scanning strategies provided in an embodiment of the present invention. First, eight intermediate feature maps are generated through a pre-set eight-scan sequence (forward horizontal, reverse horizontal, forward vertical, reverse vertical, forward diagonal, reverse diagonal, forward anti-diagonal, and reverse anti-diagonal). See also... Figure 7 , Figure 7 This is a schematic diagram of the state space module of the weighted scanning strategy provided in this embodiment of the invention. A weighted scanning order feature map is then obtained using a shared coordinate attention mechanism. The global dependencies are calculated using the S6 Block on the weighted scanning order feature map. .

[0039] The calculations performed in Block S6 are as follows: For parameter matrix The state transition matrix obtained after exponential mapping Optimizes as the model is trained: ; Discretize the mapping matrix B to obtain the discretized form It optimizes as the model is trained.

[0040] ; in, This indicates the discretization step size.

[0041] The state update equation is: ; in, Indicates the current hidden state; This indicates the hidden state at the previous moment; Represents the input sequence; The output equation is: ; in, Indicates the output sequence; and These are the state mapping matrix and the input mapping matrix, respectively, which are optimized during model training; ; in, This represents the output of the entire sequence, where L represents the output corresponding to different scan orders. In this case, only [the output is actually...]. and .

[0042] Use selective scan operation only on features A forward and reverse scan is performed along the channel dimension, and the channel dependencies are calculated using S6Bolck to obtain the third feature map. .

[0043] Will and respectively with The fourth feature map is obtained by performing element-wise multiplication. and the fifth feature map Then, point addition is performed along the channel dimension, and a convolution operation is used for feature fusion and channel dimension reduction to obtain the sixth feature map. : ; in, Dot product; This represents a dot plus a dot.

[0044] In this embodiment of the invention, the local-global feature coupling unit module is used to adaptively fuse the local detail information and global context feature modeling capabilities of the encoder.

[0045] In this embodiment of the invention, the local-global feature coupling unit module connects the input and output terminals of the HVB, and also connects the input and output terminals of the CB. The calculations in the local-global feature coupling unit module are as follows: See Figure 8 , Figure 8 This is a schematic diagram of the local-global feature coupling unit module provided in an embodiment of the present invention. It uses a cross-attention mechanism to calculate the fused features from local features to global features, obtaining the first fused feature. : ; ; ; ; ; in, express The key matrix; This indicates the global characteristics output by the HVB module in the previous stage; It is the encoder stage. ; Indicates multilayer sensor operation; express The query matrix; express The value matrix; This represents the local features of the first intermediate layer output by the CB module, which is the local features output by the second convolutional layer in the CB module. Used to convert attention scores into a probability weight distribution to compute a weighted sum of value vectors; express and The interaction correlation matrix; The spatial dimension representing global features; express and based on Integration features; This indicates the global characteristics output by the HVB module at the current stage; In this embodiment of the invention, a cross-attention mechanism is used to calculate the fusion feature from local features to global features, resulting in a second fusion feature. The specific calculation process is as follows: ; ; ; ; in, express The key matrix; This indicates that it is the local feature of the second intermediate layer output by the CB module at the current stage, that is, the local feature output by the fourth convolutional layer in the CB module; express The query matrix; express The value matrix; Spatial dimension representing local features; express and Integration features based on cross-attention relationships; express Integrating global features The final characteristic after; Finally based on After passing through the fifth and sixth convolutional layers, the second fused feature is output. .

[0046] In this embodiment of the invention, based on the above, the dual-branch module provided by this embodiment includes four dual-branch encoders, wherein the HVB module in the first dual-branch encoder ultimately outputs... The final output of the CB module The final output of the HVB module in the second dual-branch encoder The final output of the CB module The final output of the HVB module in the third dual-branch encoder The final output of the CB module The final output of the HVB module in the fourth dual-branch encoder The final output of the CB module .

[0047] See Figure 9 , Figure 9 This is a schematic diagram of the dual-branch decoder module provided in an embodiment of the present invention. The dual-branch decoder is a U-shaped decoder that receives... , , and Then, perform the following operations: In the top-level branch, After upsampling and convolution processing, and Feature splicing and fusion are performed, followed by upsampling and convolution processing to obtain the top-level branch features; the dense block includes a convolutional layer, a batch normalization module (BN), a convolutional layer and a batch normalization module connected in sequence.

[0048] In the middle-level branches, the characteristics of the top-level branches and Feature splicing and fusion are performed, followed by dense block processing, upsampling, and convolution to obtain mid-level branch features; In the bottom-level branches, the characteristics of the middle-level branches and Feature concatenation and fusion are performed, followed by dense block processing, upsampling, and convolution. Finally, the output is a global decoded feature map after passing through two convolutional layers. .

[0049] In the U-shaped decoder , , and Perform the same operation, and finally output the local decoded feature map. .

[0050] In this embodiment of the invention, the Attention-Aware Gating Mechanism (AGM) module performs semantic segmentation based on local and global decoded feature maps, using attention awareness to obtain segmentation prediction results, as follows: First, see Figure 10 , Figure 10This is a schematic diagram of the gating mechanism module provided in this embodiment of the invention. The local decoding feature map and the global decoding feature map are concatenated in the channel dimension. Parallel convolution operations are used to extract coarse-grained features and fine-grained features. The convolution kernel sizes are 3 and 5, respectively. The first decoding feature map is obtained by adding them element by element. ; Then, the second feature map is obtained through the global pooling operation avgPool. After performing convolution operations using two cascaded 1×1 convolutional layers, using Softmax The operation transforms the attention score into a probability weight distribution. and with and Perform dot product to obtain the fused decoded feature map. : ; in, , ,and , Used for predicting the final segmentation results.

[0051] Specifically, the gating mechanism module outputs the segmentation prediction result through a convolutional layer with 8 feature channels. The 8 feature channels correspond to the segmentation categories of the segmentation prediction result: background, abdominal wall, liver, fat, gallbladder, surgical instruments, gastrointestinal tract, and superclass.

[0052] The training process of the laparoscopic video semantic segmentation network model will be explained below. (See also...) Figure 11 , Figure 11 This is a schematic diagram illustrating the training process of the laparoscopic video semantic segmentation network model provided in this embodiment of the invention: First, a dataset is constructed, which includes multiple laparoscopic sample images and the standard segmentation results corresponding to each laparoscopic sample image.

[0053] A subset of the Cholec80 public dataset was used. Seventeen laparoscopic cholecystectomy videos were selected from this dataset, and 101 short video sequences (80 frames each, approximately 3 seconds) were extracted from them. The PixelAnnotationTool was used to annotate each frame, resulting in a total of 8080 annotated images, containing 13 semantic categories. To address the class imbalance issue caused by the low frequency of certain categories in the public dataset, a class merging strategy is adopted: categories that appear only in one or two different surgical procedures are uniformly assigned to the newly established Misc superclass; at the same time, the two surgical instrument categories of grasping forceps and L-shaped electrocoagulation hook are merged to improve the stability of model training.

[0054] For private datasets, videos of traditional manual laparoscopic and robot-assisted laparoscopic minimally invasive surgeries were collected as the original dataset, and important anatomical structures and surgical instruments in the dataset were delineated.

[0055] In the private dataset, the original laparoscopic surgery video (original frame rate of approximately 25fps) was downsampled in the temporal dimension. The video stream was read using the moviepy.editor library, and temporal resampling was performed using the VideoFileClip.set_fps() method. Finally, the sampling rate was uniformly set to 1 frame / second and re-encoded for output.

[0056] Based on the processing of the aforementioned private dataset, the sampling interval parameter T=30 seconds is set. In the 1fps video sequence obtained by downsampling, keyframes are extracted at equal time intervals (i.e., one frame is extracted every 30 seconds) for manual or automatic segmentation and annotation.

[0057] The preprocessed public and private datasets are collectively referred to as datasets, and are divided into datasets (13 surgical procedures, namely 1, 9, 18, 20, 24, 25, 26, 35, 43, 48, 55 and 28), validation sets (2 surgical procedures, namely 17 and 52) and test sets (2 surgical procedures, namely 12 and 27) according to the surgical procedure segments.

[0058] The final dataset includes multiple laparoscopic sample images and the standard segmentation results corresponding to each laparoscopic sample image.

[0059] Before optimizing the semantic segmentation network model for laparoscopic videos, we first set the initial learning rate, learning rate decay method, number of network iterations, learning rate scheduler, and optimizer. The initial learning rate used here is 5e-5, the learning rate scheduler uses a step decay strategy with a step size of 30, the number of network iterations is 150, and the Adam optimizer is used for parameter optimization. Next, the laparoscopic video semantic segmentation network model is trained. Each laparoscopic sample image is input into the initial laparoscopic video semantic segmentation network model, so that the initial laparoscopic video semantic segmentation network model outputs the predicted sample segmentation results corresponding to each laparoscopic sample image. The specific process can be found in the aforementioned steps for processing laparoscopic images using the laparoscopic video semantic segmentation network model, and will not be repeated here.

[0060] A loss function is constructed based on the difference between the standard segmentation result and the predicted segmentation result corresponding to each laparoscopic sample image. The loss function is iteratively updated in the direction of reducing the difference until the number of iterations reaches a preset number or the loss function converges, thus obtaining the trained laparoscopic video semantic segmentation network model.

[0061] In this embodiment of the invention, the loss function includes the cross-entropy loss function and the deep supervision loss function.

[0062] Cross-entropy loss function include: ; in, Indicates the first Standard segmentation results corresponding to each laparoscopic sample image; Indicates the first Predicted sample segmentation results for each laparoscopic sample image; Deep supervised loss function include: ; in, This represents the loss value of the segmentation result obtained from the HVMamba branch; This represents the loss value obtained from the segmentation result obtained by the CNN branch; This represents the loss value of the segmentation result obtained by the attention-aware gating mechanism module.

[0063] In this embodiment of the invention, during iterative training, the model performance is evaluated using a validation set after each round of training, and hyperparameters are adjusted based on the validation results to prevent overfitting. After the model training is complete, the segmentation effect of the final model is quantitatively evaluated using test set images independent of the training and validation sets. The specific evaluation coefficients are: (1) Intersection over Union (IoU) of a single class: ; (2) Single-category overlap coefficient (Dice): ; (3) Mean Intersection over Union (mIoU): ; (4) Mean Dice (mDice): ; in, A false positive is a feature that is not in the specified category but is predicted as belonging to that category. A false negative is a feature that is actually in that category but is predicted to be in another category. A true positive is a feature that perfectly matches the prediction. This represents the total number of categories.

[0064] The effectiveness and segmentation accuracy of the laparoscopic video semantic segmentation network model were verified through model evaluation.

[0065] In this embodiment of the invention, the dual-branch encoder in the laparoscopic video semantic segmentation network model can be used to obtain local and global features based on initial features through local feature extraction and weighted multi-directional scanning strategies. It can capture spatial features from all directions, effectively solve the segmentation difficulties caused by the oblique distribution of instruments, and significantly improve the model's discrimination ability.

[0066] Furthermore, the dual-branch encoder can utilize a cross-attention mechanism to fuse local and global features. The gating mechanism module can perform semantic segmentation based on local and global decoded feature maps using attention perception. By dynamically fusing local detail information and global contextual information perception capabilities, and adaptively incorporating complementary features from different semantic dimensions, it improves the segmentation accuracy of laparoscopic anatomical structures and surgical instruments.

[0067] Based on the same inventive concept, embodiments of the present invention also provide a semantic segmentation device for laparoscopic videos based on a weighted scanning strategy (SSM), the semantic segmentation device for laparoscopic videos comprising: The acquisition module is used to acquire the laparoscopic image to be segmented; The segmentation module is used to input the laparoscopic image into a pre-trained laparoscopic video semantic segmentation network model, so that the laparoscopic video semantic segmentation network model outputs segmentation prediction results; The laparoscopic video semantic segmentation network model includes an initial module, a dual-branch encoder, a dual-branch decoder, and a gating mechanism module; the initial module is used to extract initial features from the laparoscopic image to obtain initial features; The dual-branch encoder is used to obtain local features and global features based on the initial features through local feature extraction and a weighted multi-directional scanning strategy; and to perform feature fusion on the local features and the global features using a cross-attention mechanism to obtain a first fused feature and a second fused feature. The dual-branch decoder is used to decode the first fused feature and the second fused feature respectively to obtain a local decoded feature map and a global decoded feature map; The gating mechanism module is used to perform semantic segmentation based on the local decoded feature map and the global decoded feature map, using attention perception to obtain segmentation prediction results.

[0068] In this embodiment of the invention, the dual-branch encoder in the laparoscopic video semantic segmentation network model can be used to obtain local and global features based on initial features through local feature extraction and weighted multi-directional scanning strategies. It can capture spatial features from all directions, effectively solve the segmentation difficulties caused by the oblique distribution of instruments, and significantly improve the model's discrimination ability.

[0069] Furthermore, the dual-branch encoder can utilize a cross-attention mechanism to fuse local and global features. The gating mechanism module can perform semantic segmentation based on local and global decoded feature maps using attention perception. By dynamically fusing local detail information and global contextual information perception capabilities, and adaptively incorporating complementary features from different semantic dimensions, it improves the segmentation accuracy of laparoscopic anatomical structures and surgical instruments.

[0070] Optionally, the dual-branch encoder includes multiple dual-branch encoder structural unit modules; each dual-branch encoder structural unit module includes a CB module, a local-global feature coupling unit module, and an HVB module; in each dual-branch encoder structural unit module, the input and output terminals of the CB module and the input and output terminals of the HVB module are all connected to the local-global feature coupling unit module.

[0071] Optionally, the CB module includes two cascaded residual sub-modules with identical structures.

[0072] Optionally, the HVB module includes a first normalization layer, an HSS2D layer, a second normalization layer, and a multilayer perceptron layer connected in sequence.

[0073] Optionally, the training method for the laparoscopic video semantic segmentation network model includes: Construct a dataset; the dataset includes multiple laparoscopic sample images and the standard segmentation results corresponding to each laparoscopic sample image; Each laparoscopic sample image is input into the initial laparoscopic video semantic segmentation network model, so that the initial laparoscopic video semantic segmentation network model outputs the predicted sample segmentation result corresponding to each laparoscopic sample image; A loss function is constructed based on the difference between the standard segmentation result and the predicted segmentation result corresponding to each laparoscopic sample image. The loss function is iteratively updated in the direction of reducing the difference until the number of iterations reaches a preset number or the loss function converges, thus obtaining the trained laparoscopic video semantic segmentation network model.

[0074] Optionally, the loss function includes a cross-entropy loss function and a deep supervision loss function; The cross-entropy loss function includes: ; in, This represents the cross-entropy loss function; Indicates the first Standard segmentation results corresponding to each laparoscopic sample image; Indicates the first Predicted sample segmentation results for each laparoscopic sample image; The deep supervision loss function includes: ; in, Represents the deep supervised loss function; This represents the segmentation loss value generated by the HVMamba branch; This represents the segmentation loss value generated by the CNN branch; This represents the segmentation loss value generated by the gating mechanism module.

[0075] This invention also provides an electronic device, such as... Figure 12 As shown, it includes a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204. The processor 1201, communication interface 1202, and memory 1203 communicate with each other via the communication bus 1204. Memory 1203 is used to store computer programs; When the processor 1201 executes the program stored in the memory 1203, it implements the method steps of any of the above-mentioned laparoscopic video semantic segmentation methods based on the weighted scanning strategy (SSM).

[0076] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0077] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0078] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0079] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0080] The present invention also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program implements the method steps of any of the above-described laparoscopic video semantic segmentation methods based on the weighted scanning strategy (SSM).

[0081] Optionally, the computer-readable storage medium may be non-volatile memory (NVM), such as at least one disk storage device.

[0082] Optionally, the aforementioned computer-readable storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0083] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of the method described in any of the above-described laparoscopic video semantic segmentation methods based on weighted scanning strategy (SSM).

[0084] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0085] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0086] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0087] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0088] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0089] It should be noted that the device, electronic device and storage medium in the embodiments of the present invention are respectively the device, electronic device and storage medium for applying the above-mentioned laparoscopic video semantic segmentation method based on weighted scanning strategy (SSM). Therefore, all embodiments of the above-mentioned laparoscopic video semantic segmentation method based on weighted scanning strategy (SSM) are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0090] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for laparoscopic videos based on the weighted scanning strategy (SSM), characterized in that, The laparoscopic video semantic segmentation method includes: Obtain the laparoscopic image to be segmented; The laparoscopic image is input into a pre-trained laparoscopic video semantic segmentation network model so that the laparoscopic video semantic segmentation network model outputs segmentation prediction results; The laparoscopic video semantic segmentation network model includes an initialization module, a dual-branch encoder, a dual-branch decoder, and a gating mechanism module. The initial module is used to extract initial features from the laparoscopic image to obtain initial features; The dual-branch encoder is used to obtain local features and global features based on the initial features through local feature extraction and a weighted multi-directional scanning strategy; and to perform feature fusion on the local features and the global features using a cross-attention mechanism to obtain a first fused feature and a second fused feature. The dual-branch decoder is used to decode the first fused feature and the second fused feature respectively to obtain a local decoded feature map and a global decoded feature map; The gating mechanism module is used to perform semantic segmentation based on the local decoded feature map and the global decoded feature map, using attention perception to obtain segmentation prediction results.

2. The laparoscopic video semantic segmentation method according to claim 1, characterized in that, The dual-branch encoder includes multiple dual-branch encoder structural unit modules; each dual-branch encoder structural unit module includes a CB module, a local-global feature coupling unit module, and an HVB module; in each dual-branch encoder structural unit module, the input and output terminals of the CB module and the input and output terminals of the HVB module are all connected to the local-global feature coupling unit module.

3. The laparoscopic video semantic segmentation method according to claim 2, characterized in that, The CB module comprises two cascaded residual sub-modules with identical structures.

4. The laparoscopic video semantic segmentation method according to claim 2, characterized in that, The HVB module includes a first normalization layer, an HSS2D layer, a second normalization layer, and a multilayer perceptron layer connected in sequence.

5. The laparoscopic video semantic segmentation method according to claim 1, characterized in that, The training methods for the laparoscopic video semantic segmentation network model include: Construct a dataset; the dataset includes multiple laparoscopic sample images and the standard segmentation results corresponding to each laparoscopic sample image; Each laparoscopic sample image is input into the initial laparoscopic video semantic segmentation network model, so that the initial laparoscopic video semantic segmentation network model outputs the predicted sample segmentation result corresponding to each laparoscopic sample image; A loss function is constructed based on the difference between the standard segmentation result and the predicted segmentation result corresponding to each laparoscopic sample image. The loss function is iteratively updated in the direction of reducing the difference until the number of iterations reaches a preset number or the loss function converges, thus obtaining the trained laparoscopic video semantic segmentation network model.

6. The laparoscopic video semantic segmentation method according to claim 5, characterized in that, The loss function includes the cross-entropy loss function and the deep supervision loss function; The cross-entropy loss function includes: ； in, This represents the cross-entropy loss function; Indicates the first Standard segmentation results corresponding to each laparoscopic sample image; Indicates the first Predicted sample segmentation results for each laparoscopic sample image; The deep supervision loss function includes: ； in, Represents the deep supervised loss function; This represents the segmentation loss value generated by the HVMamba branch; This represents the segmentation loss value generated by the CNN branch; This represents the segmentation loss value generated by the gating mechanism module.

7. A semantic segmentation device for laparoscopic video based on a weighted scanning strategy (SSM), characterized in that, The laparoscopic video semantic segmentation device includes: The acquisition module is used to acquire the laparoscopic image to be segmented; The segmentation module is used to input the laparoscopic image into a pre-trained laparoscopic video semantic segmentation network model, so that the laparoscopic video semantic segmentation network model outputs segmentation prediction results; The laparoscopic video semantic segmentation network model includes an initialization module, a dual-branch encoder, a dual-branch decoder, and a gating mechanism module. The initial module is used to extract initial features from the laparoscopic image to obtain initial features; The dual-branch encoder is used to obtain local features and global features based on the initial features through local feature extraction and a weighted multi-directional scanning strategy; and to perform feature fusion on the local features and the global features using a cross-attention mechanism to obtain a first fused feature and a second fused feature. The dual-branch decoder is used to decode the first fused feature and the second fused feature respectively to obtain a local decoded feature map and a global decoded feature map; The gating mechanism module is used to perform semantic segmentation based on the local decoded feature map and the global decoded feature map, using attention perception to obtain segmentation prediction results.

8. The laparoscopic video semantic segmentation device based on weighted scanning strategy (SSM) according to claim 7, characterized in that, The dual-branch encoder includes multiple dual-branch encoder structural unit modules; each dual-branch encoder structural unit module includes a CB module, a local-global feature coupling unit module, and an HVB module; in each dual-branch encoder structural unit module, the input and output terminals of the CB module and the input and output terminals of the HVB module are all connected to the local-global feature coupling unit module.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a computer program stored in memory, implements the laparoscopic video semantic segmentation method based on the weighted scanning strategy (SSM) as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the laparoscopic video semantic segmentation method based on the weighted scanning strategy (SSM) as described in any one of claims 1-6.

Images