Robotic medical image segmentation and feature extraction method for precision surgery

Abstract

The present invention relates to the field of medical image segmentation and discloses a robotic medical image segmentation and feature extraction method for precision surgery. The method comprises: constructing a dynamic segmentation network model with a bidirectional attention architecture; an input preprocessing module for feature extraction and generation of a multi-scale feature pyramid; a Transformer encoding branch for temporal feature modeling and outputting temporal enhancement features; a convolutional encoding branch for enhancing anatomical features and surgical instrument features and outputting spatial enhancement features; a multi-stage feature fusion unit for performing multi-stage iterative fusion and outputting final fused features; a decoding output module for decoding and generating pixel-level segmentation masks for anatomical structures and surgical instruments; training the dynamic segmentation network model; and utilizing the trained dynamic segmentation network model to perform medical image segmentation and feature extraction based on real-time input medical image video sequences. The method achieves precise segmentation of anatomical tissue and dynamic instruments in surgical scenes.

Description

Technical Field

[0001] The present invention relates to the field of real-time medical image segmentation for robot-assisted surgery, and more specifically, to a robot-assisted medical image segmentation and feature extraction method for precision surgery. Background Art

[0002] With the advancement of robotic-assisted minimally invasive laparoscopic surgery, accurate surgical medical image segmentation has become crucial. However, existing methods face two major challenges: first, the limitations of static images, including the ambiguity of local features between different anatomical structures and the fine-grained structural complexity of deformable instruments; second, the complexity of dynamic videos, such as blurring caused by rapid instrument motion and inevitable cross-occlusion. Although existing methods primarily focus on spatial feature extraction, they ignore the temporal dependencies in surgical video streams, resulting in poor performance in surgical scene segmentation tasks. Summary of the Invention

[0003] The purpose of this invention is to propose a robotic medical image segmentation and feature extraction method for precision surgery. By combining temporal dynamics and structural asymmetry, accurate segmentation and feature extraction of anatomical tissues and dynamic instruments in surgical scenes can be achieved, thereby providing reliable support for precision surgery.

[0004] To achieve the above objectives, the present invention proposes a robotic medical image segmentation and feature extraction method for precision surgery, comprising:

[0005] A dynamic segmentation network model with a bidirectional attention architecture is constructed, and the dynamic segmentation network model includes an input preprocessing module, a Transformer encoding branch, a convolutional encoding branch, a multi-stage feature fusion unit and a decoding output module; wherein the input preprocessing module is used to extract features from video frames and generate a multi-scale feature pyramid; the Transformer encoding branch is used to perform temporal feature modeling based on the input feature pyramid and output temporal enhancement features; the convolutional encoding branch is used to enhance anatomical features and surgical instrument features through convolution operations based on the input feature pyramid and output spatial enhancement features; the multi-stage feature fusion unit is used to perform multi-stage iterative fusion on the outputs of the Transformer encoding branch and the convolutional encoding branch and output the final fused features; the decoding output module is used to input the fused features into the Transformer decoder for decoding to generate pixel-level segmentation masks of anatomical structures and surgical instruments;

[0006] Training the dynamic segmentation network model using a training data set;

[0007] During robot-assisted surgery, the trained dynamic segmentation network model is used to perform medical image segmentation and feature extraction based on real-time input medical image video sequences, and the segmentation mask output by the model is transmitted to the surgical robot control system for surgical safety warning and navigation assistance.

[0008] Optionally, the input preprocessing module includes:

[0009] The ResNet-50 backbone network is used to extract features from the input continuous video frames and generate feature maps of different resolutions for each frame. The feature maps contain the spatial feature information of each frame.

[0010] The temporal dimension expansion unit is used to splice the feature maps of multiple consecutive frames according to the time dimension to form a spatiotemporal feature tensor;

[0011] A 3D convolution layer is used to apply a 3D convolution operation to the spatiotemporal feature tensor to gradually downsample and generate a multi-scale feature pyramid, wherein the multi-scale feature pyramid includes multiple feature layers of different resolutions, each feature layer containing time dimension information.

[0012] Optionally, the Transformer encoding branch includes a temporal query propagator, a multi-head attention layer, and a dimensionality expansion layer connected in sequence;

[0013] The time query propagator includes:

[0014] The feature projection unit is used to splice the continuous multi-frame features of a certain level in the feature pyramid along the time dimension, project the spliced ​​features through a learnable key-value weight matrix, and generate a key-value matrix;

[0015] The temporal query generation unit is used to perform convolution operations on the spliced ​​features to extract high-activation areas, use Top-K to select key content features to generate content queries, and generate position queries through forward and backward propagation based on the position embedding of the previous, intermediate, and next frames of the input;

[0016] The multi-head attention layer is used to perform multi-head attention calculation and feature refinement based on the content query, the location query and the key-value matrix;

[0017] The dimension expansion layer is used to perform channel expansion on the refined features and output enhanced features with temporal consistency.

[0018] Optionally, the content query generation formula is:

[0019]

[0020] in, The query content vector is used to capture key area features; is the feature map of the t-th frame; For continuous splicing along the time dimension T Frame features; Conv represents 2D convolution, the convolution kernel size is 3×3, and the number of output channels is D ; Indicates that the previous K high-response regions to generate sparse content query vectors.

[0021] Optionally, the generation formula of the location query is:

[0022]

[0023] in, No. t The position query vector of the frame is used to encode the spatiotemporal position information; For continuous splicing along the time dimension T Frame features; is a learnable weight matrix; Conv represents 2D convolution; represents a fully connected layer, keeping the dimension of the location query as D ; and Respectively represent t+1 Frame and t-1 The position query vector of the frame.

[0024] Optionally, the multi-head attention calculation formula is:

[0025]

[0026] in, is the query content vector; No. t The position query vector of the frame; K is the bond matrix generated by feature projection; is the transposed matrix of K; V is the value matrix generated by feature projection; Represents the time aggregation of location query; ⊙ represents element-by-element multiplication; is the scaling factor used to stabilize the gradient; It represents normalization along the key dimension and outputs the attention weight matrix; Attention(Q,K,V) represents multi-head attention calculation and outputs the attention weighted feature matrix, where each row corresponds to the enhanced features of a key area.

[0027] Optionally, the convolutional coding branch includes an aggregated asymmetric feature pyramid module and a feature compression layer;

[0028] The aggregated asymmetric feature pyramid module includes:

[0029] The spatiotemporal aggregation layer is used to perform convolution operations on the continuous multi-frame features of a certain level in the feature pyramid to obtain spatiotemporal aggregation features;

[0030] an anatomical perception enhancement branch, configured to perform a symmetric convolution operation on the spatiotemporal aggregation feature map to enhance the anatomical structure features of irregular polygons and output multi-scale anatomical features;

[0031] an instrument perception enhancement branch, configured to perform a convolution operation on the spatiotemporal aggregation feature map using an asymmetric strip convolution pair to enhance regular strip-shaped instrument features and output multi-scale instrument features;

[0032] The attention weighted fusion layer is used to perform attention fusion calculation on the enhanced anatomical features, instrument features and original aggregate features to generate spatial enhancement features;

[0033] The feature compression layer is used to compress the spatial enhancement features to the same channel dimension as the Transformer encoding branch.

[0034] Optionally, the calculation formula of the spatiotemporal aggregation layer is:

[0035]

[0036] in, It is the spatiotemporal aggregation feature; For continuous splicing along the time dimension T Frame features; Conv 5×5×T represents a convolution kernel of size 5×5×T;

[0037] The calculation formula of the anatomical perception enhancement branch is:

[0038]

[0039] The calculation formula of the device perception enhancement branch is:

[0040]

[0041] in, k m is the spatial size of the symmetric convolution kernel; t m is the depth of the time dimension.

[0042] Optionally, the attention fusion calculation formula is:

[0043]

[0044] Among them, ⨂ represents element-by-element multiplication; Conv1×1×T Represents spatiotemporal compression convolution to generate attention maps; Spatial enhancement features for attention-weighted fusion.

[0045] Optionally, the multi-stage feature fusion unit adds the outputs of the Transformer encoding branch and the convolutional encoding branch element by element, and generates the final fused features after multiple parameter-sharing residual iterations.

[0046] The beneficial effects of the present invention are:

[0047] The present invention constructs a dynamic segmentation network model with a bidirectional attention architecture. In the Transformer encoding branch, the global temporal modeling of the Transformer is integrated with the local spatial perception of the convolution to achieve collaborative optimization of spatiotemporal features (based on the temporal query propagator, a dynamic query mechanism is used to model cross-frame dependencies, and the frame feature representation is enhanced by using temporal consistency constraints to solve the problems of instrument motion blur and occlusion). In the convolutional encoding branch, anatomical features and surgical instrument features are enhanced through convolution operations, and spatial enhancement features are extracted (the feature expression of anatomical tissue (irregular morphology) and surgical instruments (long strip structure) is enhanced by using symmetric and asymmetric convolution separation through the aggregation of asymmetric feature pyramid modules). Then, the features extracted by the Transformer encoding branch and the convolutional encoding branch are iteratively fused in multiple stages through a multi-stage feature fusion unit to achieve progressive optimization of spatial accuracy and temporal consistency, and the final fused features are output. Finally, the fused features are decoded by a decoding output module to generate pixel-level segmentation masks of the anatomical structure and surgical instruments, thereby achieving accurate segmentation of anatomical structures and instruments in surgical videos.

[0048] The system of the present invention has other features and advantages that will be apparent from or will be described in detail in the accompanying drawings and subsequent detailed description incorporated herein, which together serve to explain the specific principles of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings, in which like reference numerals generally represent like components.

[0050] Figure 1 A schematic diagram of a dynamic segmentation network model in one embodiment of the present invention is shown.

[0051] Figure 2 FIG. 4 is a schematic diagram showing a time query propagator (TQP) in one embodiment of the present invention.

[0052] Figure 3 A schematic diagram of an aggregated asymmetric feature pyramid module (AAFP) in one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0053] The present invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention may be implemented in various forms and is not limited to the embodiments set forth herein. Rather, these embodiments are provided to make the present invention more thorough and complete and to fully convey the scope of the present invention to those skilled in the art. Example

[0054] This embodiment provides a robotic medical image segmentation and feature extraction method for precision surgery, including:

[0055] S1: Construct a dynamic segmentation network model with a bidirectional attention architecture. The dynamic segmentation network model includes an input preprocessing module, a Transformer encoding branch, a convolutional encoding branch, a multi-stage feature fusion unit, and a decoding output module. The input preprocessing module is used to extract features from video frames and generate a multi-scale feature pyramid. The Transformer encoding branch is used to perform temporal feature modeling based on the input feature pyramid and output temporal enhancement features. The convolutional encoding branch is used to enhance anatomical features and surgical instrument features through convolution operations based on the input feature pyramid and output spatial enhancement features. The multi-stage feature fusion unit is used to perform multi-stage iterative fusion on the outputs of the Transformer encoding branch and the convolutional encoding branch and output the final fused features. The decoding output module is used to input the fused features into the Transformer decoder for decoding to generate pixel-level segmentation masks of anatomical structures and surgical instruments.

[0056] Specifically, if Figure 1 As shown in the figure, a dynamic segmentation network model with a bidirectional attention architecture is constructed, consisting of a Transformer encoding branch and a convolutional encoding branch. Global temporal dependencies and local spatial features are iteratively integrated through multiple stages. Dynamic spatiotemporal queries are generated through the Temporal Query Propagator (TQP). Combined with content and location queries, multi-head attention weights are calculated to generate occlusion-resistant spatiotemporal feature tubes. The Aggregated Asymmetric Feature Pyramid (AAFP) module performs symmetric and asymmetric strip convolution operations on anatomical structures and surgical instruments, respectively, to fuse the spatiotemporal aggregate features. The outputs of the Transformer and convolutional branches are then element-wise added together, and the final fused features are generated through M parameter-sharing residual iterations. Finally, the fused features are input into the Transformer decoder to generate pixel-level segmentation masks.

[0057] Furthermore, in this embodiment, the input preprocessing module includes:

[0058] The ResNet-50 backbone network (without the fully connected layers) is used to extract features from continuous video input frames and generate feature maps of different resolutions for each frame. The feature maps contain the spatial feature information of each frame. For example: a single frame image is input → ResNet-50 → 2D feature maps of different levels (such as C1: 1 / 4, C2: 1 / 8, C3: 1 / 16, C4: 1 / 32) are output (resolution gradually reduced, such as 1 / 4, 1 / 8 of the original image resolution).

[0059] The temporal dimension expansion unit is used to splice the feature maps of multiple consecutive frames according to the time dimension to form a spatiotemporal feature tensor; for example, the ResNet output features of consecutive T frames are spliced ​​according to the time dimension to form a spatiotemporal feature tensor;

[0060] 3D convolution layers apply 3D convolution operations to the spatiotemporal feature tensor, gradually downsampling it to generate a multi-scale feature pyramid. A multi-scale feature pyramid consists of multiple feature layers at different resolutions, each containing information about the temporal dimension. For example, applying multiple sets of 3D convolutions (with a kernel size of 3×3×3) to the spatiotemporal feature tensor generates a multi-scale spatiotemporal feature pyramid with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32.

[0061] In one example, the input preprocessing module uses a ResNet-50 network combined with 3D convolution to extract multi-scale features from input video frames, outputting a multi-scale feature pyramid with resolutions of {1 / 4, 1 / 8, 1 / 16, 1 / 32} and feature dimensions of {256, 512, 1024, 2048}. This preprocessing reduces the computational burden, preserves multi-scale spatial information, and provides foundational features for subsequent encoding stages.

[0062] Furthermore, in this embodiment, the Transformer encoding branch includes a sequentially connected temporal query propagator (TQP), a multi-head attention layer, and a dimensionality expansion layer. The Transformer encoding branch dynamically generates spatiotemporal feature pipes through the TQP to model the temporal dependencies between video frames and address motion blur and occlusion. The multi-head attention layer captures global contextual semantic information. By extracting anti-occlusion temporal consistency features, the model's tracking capabilities for dynamic targets (such as surgical instruments) are enhanced.

[0063] like Figure 2 As shown, the Time Query Propagator (TQP) includes:

[0064] Feature projection unit: It is used to splice the continuous multi-frame features of a certain level (such as 1 / 4 level features) in the feature pyramid along the time dimension, and project the spliced ​​features through the learnable key-value weight matrix to generate a key-value matrix; for example, input continuous T Frame features (such as T =5), extracting a multi-scale feature pyramid through the input preprocessing module , and then concatenate the multi-frame features along the time dimension to obtain the connection features ; Then through the learnable key weight matrix Sum weight matrix Project the splicing features to generate a cross-frame key value matrix K and W Through multi-frame key-value projection, even if the target is occluded in a certain frame (such as surgical forceps covered by tissue), its position and semantics can still be restored through the projected features of adjacent frames.

[0065] Temporal query generation unit: used to perform convolution operations on the spliced ​​features to extract high activation areas, and use Top-K to select key content features to generate content queries, and based on the position embedding of the previous frame, intermediate frame and next frame of the input, generate position queries through forward propagation and backward propagation; Temporal queries include content queries and position queries, and each selected temporal query (Q Temp ) corresponds to a space-time feature tube;

[0066] The formula for generating content query is:

[0067]

[0068] in, The query content vector is used to capture key area features; is the feature map of the t-th frame; For continuous splicing along the time dimension T Frame features; Conv represents 2D convolution, the convolution kernel size is 3×3, and the number of output channels is D ; Indicates that the previous K high-response regions to generate sparse content query vectors.

[0069] The formula for generating the location query is:

[0070]

[0071] in, No. t The position query vector of the frame is used to encode the spatiotemporal position information; For continuous splicing along the time dimension T Frame features; is a learnable weight matrix; Conv represents 2D convolution; represents a fully connected layer, keeping the dimension of the location query as D ; and Respectively represent t+1 Frame and t-1 The position query vector of the frame.

[0072] The multi-head attention layer is used to perform multi-head attention calculations and feature refinement based on content query, location query, and key-value matrix. The multi-head attention calculation formula is:

[0073]

[0074] in, is the query content vector; No. t The position query vector of the frame; K is the bond matrix generated by feature projection; is the transposed matrix of K; V is the value matrix generated by feature projection; Represents the time aggregation of location query; ⊙ represents element-by-element multiplication; is the scaling factor used to stabilize the gradient; It represents normalization along the key dimension and outputs the attention weight matrix; Attention(Q,K,V) represents multi-head attention calculation and outputs the attention weighted feature matrix, where each row corresponds to the enhanced features of a key area.

[0075] The dimension expansion layer is used to expand the channels of the refined features and output enhanced features with temporal consistency.

[0076] Furthermore, in this embodiment, the convolutional coding branch includes an aggregated asymmetric feature pyramid module (AAFP) and a feature compression layer. By aggregating the asymmetric feature pyramid module (AAFP), the convolutional coding branch uses symmetric convolution to enhance local features of irregular anatomical structures (such as the liver and gallbladder), while simultaneously extracting details of long instruments (such as surgical forceps) through asymmetric strip convolution. This optimizes spatial detail features and distinguishes object categories with significant morphological differences.

[0077] like Figure 3 As shown, the aggregated asymmetric feature pyramid module includes:

[0078] Spatiotemporal aggregation layer: It is used to perform convolution operations on the continuous multi-frame features of a certain level in the feature pyramid (such as the 1 / 4 level features of the same input as the Transformer encoding branch) to obtain spatiotemporal aggregation features; for example, 5×5×T convolution is used to fuse spatiotemporal information. The calculation formula of the spatiotemporal aggregation layer is:

[0079]

[0080] in, It is the spatiotemporal aggregation feature; For continuous splicing along the time dimension T Frame features; Conv 5×5×T represents a convolution kernel of size 5×5×T;

[0081] Anatomy-aware enhancement branch: It is used to perform symmetric convolution operations on the spatiotemporal aggregation feature map to enhance the anatomical structure features of irregular polygons and output multi-scale anatomical features. The calculation formula of the anatomical enhancement branch is:

[0082] ;

[0083] Device perception enhancement branch: This branch is used to perform convolution operations on the spatiotemporal aggregation feature map using asymmetric strip convolution pairs to enhance the regular strip-shaped device features and output multi-scale device features.

[0084] The calculation formula of the device perception enhancement branch is:

[0085]

[0086] in, k m is the spatial size of the symmetric convolution kernel; t m is the depth of the time dimension.

[0087] Attention weighted fusion layer: used to perform attention fusion calculation on the enhanced anatomical features, instrument features and original aggregate features to generate spatial enhancement features; the attention fusion calculation formula is:

[0088]

[0089] Among them, ⨂ represents element-by-element multiplication; Conv 1×1×T Represents spatiotemporal compression convolution to generate attention maps; Spatial enhancement features for attention-weighted fusion.

[0090] The feature compression layer is used to compress the spatial enhancement features to the same channel dimension as the Transformer encoding branch.

[0091] Furthermore, in this embodiment, a multi-stage feature fusion unit adds the outputs of the Transformer encoding branch and the convolutional encoding branch element-by-element, generating the final fused features through multiple parameter-sharing residual iterations. Through multi-stage (e.g., four-stage) residual connections, the global temporal features of the Transformer branch are fused layer by layer with the local spatial features of the convolutional branch, achieving feature complementarity and progressive optimization. By combining global semantics with local details, a more robust spatiotemporal fusion feature is generated.

[0092] Specifically, the number of stages in multi-stage feature fusion matches the number of levels in the feature pyramid (e.g., 4 levels: 1 / 4, 1 / 8, 1 / 16, 1 / 32), and the processing order is progressively refined from deep layers (low resolution) to shallow layers (high resolution). For example, the input features of the current layer are multi-scale features of the 1 / 8 layer (spatiotemporal feature concatenation). The Transformer encoding branch generates a dynamic query, models cross-frame dependencies, and calculates global spatiotemporal weights through a multi-head attention mechanism, outputting temporally enhanced features. The convolutional encoding branch optimizes anatomical and instrumental features through symmetric / asymmetric convolutions, compressing the enhanced features to the same dimension as the temporally enhanced features, outputting spatially enhanced features. The features of the two branches are then fused through element-wise addition. Cross-stage transfer is then achieved through residual connections, where the fused features output by the current layer are passed to the next stage (e.g., the 1 / 4 layer) via parameter-sharing residual connections. The global temporal features of the Transformer branch are fused layer by layer with the local spatial features of the convolutional branch, outputting a multi-level feature pyramid, achieving feature complementarity and progressive optimization.

[0093] Furthermore, in this embodiment, the decoding output module upsamples the fused features to the original resolution and generates pixel-level segmentation masks through the segmentation head; and the cross entropy loss and Dice loss are combined to optimize model training.

[0094] Specifically, the decoder output module inputs the fused features (1 / 4, 1 / 8, 1 / 16, and 1 / 32 layers) output by multi-stage feature fusion units. Each layer of features contains temporal information (the result of spatiotemporal interactions across T consecutive frames). The decoder employs a progressive upsampling and cross-layer skip connection architecture to gradually recover spatial details and integrate global semantics. For example, deep features are first initialized, inputting the deepest layer features (1 / 32 layer, with the lowest resolution but richest semantic information). 3×3 convolutions are then used to compress the number of channels and reduce computational overhead. Layer-by-layer upsampling and feature fusion are then performed, from deep to shallow layers, and concatenated with the bidirectional attention features from the corresponding layers. After fusion, asymmetric 1×3 and 3×1 convolutions are used to enhance the edges of slender instruments. Residual connections are used to preserve the original features and prevent vanishing gradients. Temporal attention weights are introduced during fusion to suppress sudden noise and ensure smooth transitions between segmentation results (e.g., continuous instrument trajectory). Finally, bilinear interpolation or transposed convolution is used to upsample the highest resolution decoded features (1 / 4 layer) to the input image size. The loss calculation can use a hybrid loss function:

[0095]

[0096] in, and They are cross entropy loss and Dice loss respectively; is the cross entropy loss weight, which is used to control the contribution of cross entropy loss to the total loss; is the Dice loss weight, which is used to control the contribution of Dice loss to the total loss.

[0097] S2: Use the training dataset to train the dynamic segmentation network model;

[0098] Specifically, the dynamic segmentation network model in this embodiment is a fully supervised model, whose training relies on pixel-level annotated temporal segmentation masks. Training data can be constructed independently or directly used from existing public datasets, such as the EndoVis2018 dataset (containing 15 laparoscopic video sequences, 1280×1024 resolution, and 12-category annotations). The model training process focuses on learning spatiotemporal features and optimizing surgical scene characteristics, combining traditional methods such as data preprocessing, hybrid loss functions, and multi-stage training strategies. The training process is not detailed here.

[0099] S3: During robot-assisted surgery, the trained dynamic segmentation network model is used to perform medical image segmentation and feature extraction based on real-time input medical image video sequences. The segmentation mask output by the model is then transmitted to the surgical robot control system for surgical safety warning and navigation assistance.

[0100] Specifically, during robotic-assisted surgery (such as cholecystectomy), the endoscopic video stream is captured in real time and scaled to the model input size, with T consecutive frames (e.g., T = 5 frames) cached as the input window. A trained dynamic segmentation network model is then used to perform real-time image segmentation and feature extraction. The segmentation mask is then output to the surgical robot control system, providing surgical safety warnings. This includes overlaying the segmentation results on the endoscopic image to highlight key structures (e.g., the gallbladder trigone is marked in red, and the instrument outline is in blue). Furthermore, the segmentation mask is fed into a safety assessment module to calculate the distance between the instrument and vulnerable tissue in real time and trigger a warning (e.g., an alarm if the distance is < 2 mm).

[0101] While various embodiments of the present invention have been described above, the above description is intended to be illustrative, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A robotic medical image segmentation and feature extraction method for precision surgery, characterized in that: include: A dynamic segmentation network model with a bidirectional attention architecture is constructed, and the dynamic segmentation network model includes an input preprocessing module, a Transformer encoding branch, a convolutional encoding branch, a multi-stage feature fusion unit and a decoding output module; wherein the input preprocessing module is used to extract features from video frames and generate a multi-scale feature pyramid; the Transformer encoding branch is used to perform temporal feature modeling based on the input feature pyramid and output temporal enhancement features; the convolutional encoding branch is used to enhance anatomical features and surgical instrument features through convolution operations based on the input feature pyramid and output spatial enhancement features; the multi-stage feature fusion unit is used to perform multi-stage iterative fusion on the outputs of the Transformer encoding branch and the convolutional encoding branch and output the final fused features; the decoding output module is used to input the fused features into the Transformer decoder for decoding to generate pixel-level segmentation masks of anatomical structures and surgical instruments; Training the dynamic segmentation network model using a training data set; During robot-assisted surgery, the trained dynamic segmentation network model is used to perform medical image segmentation and feature extraction based on real-time input medical image video sequences. The segmentation mask output by the model is then transmitted to the surgical robot control system for surgical safety warning and navigation assistance. The input preprocessing module includes: The ResNet-50 backbone network is used to extract features from the input continuous video frames and generate feature maps of different resolutions for each frame. The feature maps contain the spatial feature information of each frame. The temporal dimension expansion unit is used to splice the feature maps of multiple consecutive frames according to the time dimension to form a spatiotemporal feature tensor; A 3D convolution layer is used to apply a 3D convolution operation to the spatiotemporal feature tensor to gradually downsample and generate a multi-scale feature pyramid, wherein the multi-scale feature pyramid includes multiple feature layers of different resolutions, each feature layer contains time dimension information; The Transformer encoding branch includes a temporal query propagator, a multi-head attention layer, and a dimension expansion layer connected in sequence; The time query propagator includes: The feature projection unit is used to splice the continuous multi-frame features of a certain level in the feature pyramid along the time dimension, project the spliced ​​features through a learnable key-value weight matrix, and generate a key-value matrix; The temporal query generation unit is used to perform convolution operations on the spliced ​​features to extract high-activation areas, use Top-K to select key content features to generate content queries, and generate position queries through forward and backward propagation based on the position embedding of the previous, intermediate, and next frames of the input; The multi-head attention layer is used to perform multi-head attention calculation and feature refinement based on the content query, the location query and the key-value matrix; The dimension expansion layer is used to perform channel expansion on the refined features and output enhanced features with temporal consistency.

2. The method according to claim 1, characterized in that The generation formula of the content query is: in, The query content vector is used to capture key area features; is the feature map of the t-th frame; For continuous splicing along the time dimension T Frame features; Conv represents 2D convolution, the convolution kernel size is 3×3, and the number of output channels is D ; Indicates that the previous K high-response regions to generate sparse content query vectors.

3. The method according to claim 2, characterized in that The generation formula of the location query is: in, No. t The position query vector of the frame is used to encode the spatiotemporal position information; For continuous splicing along the time dimension T Frame features; is a learnable weight matrix; Conv represents 2D convolution; represents a fully connected layer, keeping the dimension of the location query as D ; and Respectively represent t+1 Frame and t-1 The position query vector of the frame.

4. The method according to claim 3, characterized in that The multi-head attention calculation formula is: in, is the query content vector; No. t The position query vector of the frame; K is the bond matrix generated by feature projection; is the transposed matrix of K; V is the value matrix generated by feature projection; Represents the time aggregation of location query; ⊙ represents element-by-element multiplication; is the scaling factor used to stabilize the gradient; It represents normalization along the key dimension and outputs the attention weight matrix; Attention(Q,K,V) represents multi-head attention calculation and outputs the attention weighted feature matrix, where each row corresponds to the enhanced features of a key area.

5. The method according to claim 1, characterized in that The convolutional coding branch includes an aggregated asymmetric feature pyramid module and a feature compression layer; The aggregated asymmetric feature pyramid module includes: The spatiotemporal aggregation layer is used to perform convolution operations on the continuous multi-frame features of a certain level in the feature pyramid to obtain spatiotemporal aggregation features; The anatomical perception enhancement branch is used to perform symmetric convolution operations on the spatiotemporal aggregation feature map to enhance the anatomical structure features of irregular polygons and output multi-scale anatomical features; an instrument perception enhancement branch, configured to perform a convolution operation on the spatiotemporal aggregation feature map using an asymmetric strip convolution pair to enhance regular strip-shaped instrument features and output multi-scale instrument features; The attention weighted fusion layer is used to perform attention fusion calculation on the enhanced anatomical features, instrument features and original aggregate features to generate spatial enhancement features; The feature compression layer is used to compress the spatial enhancement features to the same channel dimension as the Transformer encoding branch.

6. The method according to claim 5, characterized in that The calculation formula of the spatiotemporal aggregation layer is: in, It is the spatiotemporal aggregation feature; For continuous splicing along the time dimension T Frame features; Conv 5×5×T represents a convolution kernel of size 5×5×T; The calculation formula of the anatomical perception enhancement branch is: The calculation formula of the device perception enhancement branch is: in, k m is the spatial size of the symmetric convolution kernel; t m is the depth of the time dimension.

7. The method according to claim 6, characterized in that The attention fusion calculation formula is: Among them, ⨂ represents element-by-element multiplication; Conv 1×1×T Represents spatiotemporal compression convolution to generate attention maps; Spatial enhancement features for attention-weighted fusion.

8. The method according to claim 1, characterized in that The multi-stage feature fusion unit adds the outputs of the Transformer encoding branch and the convolutional encoding branch element by element, and generates the final fused features after multiple parameter-sharing residual iterations.

Images