Surgical instrument endoscope image segmentation method and system based on graph dynamics enhancement

Abstract

The application discloses a surgical instrument endoscope image segmentation method and system based on graph dynamics enhancement, proposes a graph dynamics enhancement basic model combined with topological structure reasoning, boundary dynamics evolution, attention mechanism and multi-scale dynamic fusion unit, and through the collaborative work of a topological coupling reasoning field, a boundary dispersion dynamic field and a multi-scale dynamic collaborative field, the segmentation precision and robustness of the instrument form continuity and boundary fineness in the endoscope surgery image are improved. The application shows significant performance advantages in endoscope surgery instrument segmentation, has good expansibility and practical application potential, is suitable for segmentation processing of complex occlusion, reflection interference and multi-instrument intersection characteristics in endoscope surgery, and provides efficient and accurate image segmentation support for accurate positioning, tracking and intraoperative navigation of surgical instruments.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing, specifically relating to a method and system for segmenting endoscopic images of surgical instruments based on graph dynamics enhancement. Background Technology

[0002] In the field of medical imaging, accurate image segmentation plays a crucial role in diagnosis and surgery. For example, in endoscopic surgery, precise instrument segmentation not only helps surgeons accurately assess the boundaries and morphology of instrument areas but also supports intraoperative scene understanding and surgical guidance. In endoscopic imaging, accurate differentiation between instrument areas and background tissues aids in real-time navigation, significantly improving surgical efficiency and safety. However, current image segmentation methods still suffer from insufficient accuracy and robustness when dealing with complex imaging environments, optical distortions, artifacts, and noise interference. This is particularly true in segmentation tasks involving multiple intersecting instruments, occlusion, or highly reflective areas, where existing models struggle to achieve high-quality segmentation.

[0003] Currently, segmentation models based on encoder-decoder structures such as U-Net are widely used in medical image processing. However, due to their limited receptive field, they struggle to capture long-range dependencies and multi-scale features, resulting in limited performance in noisy and low-contrast endoscopic images. Meanwhile, traditional graph structure modeling methods, such as graph neural networks, can extract global topological information from images and effectively handle complex morphologies, but they have limitations in processing multi-scale and local boundary details. Furthermore, although Transformer-based segmentation models can handle long-range dependencies, the stability and detail resolution of feature extraction remain insufficient in images with complex endoscopic anatomy and rich artifacts. Summary of the Invention

[0004] Existing segmentation methods have significant limitations in handling complex noise, topological complexity, and ambiguous boundary features, particularly in segmenting multi-instrument intersections and occlusion regions in endoscopic surgical instrument images, where existing methods struggle to meet accuracy requirements. This invention proposes a Graph-DynamicsAugmented Foundation Model (GDA-FM) for endoscopic surgical instrument image segmentation. This method proposes a multi-module fusion-based foundation model to address the aforementioned problems in existing technologies.

[0005] The technical solution included in this invention to solve its technical problem is as follows:

[0006] In a first aspect, embodiments of this application provide a method for segmenting endoscopic images of surgical instruments based on graph dynamic enhancement, comprising the following steps:

[0007] First, image data was collected from multiple publicly available surgical instrument endoscopic medical image datasets.

[0008] The collected image data is organized and divided to obtain training and test sets.

[0009] Image data is preprocessed and standardized to ensure consistency under different imaging devices or imaging conditions. Based on its ground truth mask, three types of mask prompts—boxes, points, and noise—are automatically generated to provide standardized images and diverse priors for subsequent model input.

[0010] Then, a graph dynamics enhancement model, GDA-FM, combining topological structure and multi-scale dynamics, is constructed. The model consists of three parts: an image encoder, a cue encoder, and a mask decoder. The mask decoder includes three core modules: a topologically coupled inference field (TRF), a boundary dispersive dynamic field (BDDF), and a multi-scale dynamic cooperative field (MDSF).

[0011] The constructed graph dynamical augmentation base model GDA-FM was trained using the preprocessed dataset.

[0012] Finally, the weights obtained from training are applied to the test set to evaluate the segmentation performance.

[0013] In one possible implementation, the graphical dynamics-enhanced fundamental model GDA-FM, which combines topological structure and multi-scale dynamics, operates as follows:

[0014] First, image data and three types of masking cues—boundaries, points, and noise—are input into the model. The image data is processed by an image encoder based on a ViT (Vision Transformer) architecture pre-trained on large-scale natural images, while the cues are encoded into conditional vectors by the cue encoder. Features extracted by the image encoder are input into the topologically coupled inference field of the mask decoder. Multi-layer graph convolution transfers the topological dependencies of each component, outputting structural enhancement features. The boundary dispersion dynamic field takes the structural enhancement features output from the topologically coupled inference field as input, introduces a differentiable Korteweg–de Vries equation, and simulates the nonlinear propagation and dispersion smoothing of the boundary using a fourth-order Runge–Kutta discrete model, ultimately obtaining boundary optimization features. The structural enhancement features and boundary optimization features are fused in a multi-scale dynamic collaborative field. The resulting unified feature is concatenated with the conditional vector obtained from the cue encoder along the channel dimension. An upsampling module restores the original image size to obtain the final mask.

[0015] Secondly, embodiments of this application provide a surgical instrument endoscopic image segmentation system based on graph dynamics enhancement, comprising the following modules:

[0016] The data preprocessing module is used to collect image data from multiple publicly available surgical instrument endoscopic medical image datasets; organize and divide the collected image data to obtain training and test sets; preprocess the image data, standardize the data to ensure consistency under different imaging equipment or imaging conditions, and automatically generate three types of mask prompts—boxes, points, and noise—based on the ground truth mask, providing standardized images and diverse priors for subsequent model input.

[0017] Model building module: Used to construct the graph dynamics enhancement foundation model GDA-FM, which combines topological structure and multi-scale dynamics. The model consists of three parts: an image encoder, a cue encoder, and a mask decoder. The mask decoder includes three core modules: the topologically coupled inference field (TRF), the boundary dispersive dynamic field (BDDF), and the multi-scale dynamic cooperative field (MDSF).

[0018] Model Training Module: The preprocessed dataset from the data preprocessing module is used to train the constructed graph dynamical augmentation model GDA-FM. The trained weights are then applied to the test set to evaluate the segmentation performance.

[0019] Image segmentation module: Implements the graph dynamics augmentation base model GDA-FM based on the pre-trained graph dynamics augmentation base model.

[0020] First, image data and three types of masking prompts (boxes, points, and noise) are input into a pre-trained graph dynamics enhancement model. Image data is processed by an image encoder based on a ViT (Vision Transformer) architecture pre-trained on large-scale natural images, while the prompts are encoded into conditional vectors by the prompt encoder. Features extracted by the image encoder are input into the topologically coupled inference field of the mask decoder. Multi-layer graph convolution transfers the topological dependencies of each component, outputting structural enhancement features. The boundary dispersion dynamics field takes the structural enhancement features output from the topologically coupled inference field as input, introduces differentiable Korteweg–de Vries equations, and simulates the nonlinear propagation and dispersion smoothing of the boundary using fourth-order Runge–Kutta discretization, ultimately obtaining optimized boundary features. The structural enhancement features and boundary optimization features are fused in a multi-scale dynamic collaborative field. The resulting unified feature is concatenated with the conditional vectors obtained from the prompt encoder along the channel dimension. An upsampling module restores the original image size, yielding the final mask.

[0021] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory;

[0022] The memory is used to store computer programs.

[0023] When the processor executes the program stored in the memory, it implements any of the surgical instrument endoscopic image segmentation methods described in this application.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the surgical instrument endoscopic image segmentation methods described in this application.

[0025] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the surgical instrument endoscopic image segmentation methods described in this application.

[0026] Beneficial effects of this invention:

[0027] This invention, based on a deep learning framework, addresses the major challenges in current medical image processing, including noise interference, topological complexity, and semantic ambiguity, which lead to low segmentation accuracy and negatively impact surgical guidance and navigation. The proposed multi-faceted fundamental model significantly improves the accuracy and robustness of medical image segmentation by integrating topological feature propagation, dynamic boundary evolution, and contextual fusion. Its innovative multi-dimensional feature coupling technique enhances the ability to recognize complex topological structures and semantic information, and pre-training and refined training improve the model's generalization ability and applicability. The application of this invention not only improves the accuracy of medical image segmentation and significantly enhances the efficiency and effectiveness of clinical surgery, bringing significant technological advancements to the field of medical imaging, but also ultimately improves the patient's medical experience and surgical outcomes.

[0028] The model constructed in this invention employs a Topologically Coupled Inference Field (TRF) to suppress topological breaks through graph neural networks and global feature propagation; a Boundary Dispersive Dynamical Field (BDDF) to enhance fine boundary representation using structural and dynamic domain features; and a Multi-Scale Dynamic Cooperative Field (MDSF) to selectively enhance local features through multi-scale fusion, thereby improving the detection sensitivity for multiple instrument intersections. Through the independent processing and collaborative operation of these modules, this invention achieves accurate instrument segmentation in complex surgical settings, demonstrating its broad application potential in the field of medical image segmentation. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0030] Figure 1This is a schematic diagram of the overall network framework in an embodiment of the present invention, including detailed structural diagrams of the Topology-coupled Reasoning Field (TRF) and the Boundary Dispersive Dynamics Field (BDDF).

[0031] Figure 2 This is a detailed structural diagram of the Multi-scale Dynamic Synergy Field (MDSF) in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0033] like Figure 1As shown, this invention proposes a multi-module fusion foundation model implementation method for endoscopic image segmentation of surgical instruments. This method proposes an image graph-dynamics augmented foundation model (GDA-FM) that combines topological inference, boundary dynamics evolution, attention mechanisms, and multi-scale dynamic fusion units. GDA-FM improves the segmentation accuracy and robustness of instrument morphology coherence and boundary refinement in endoscopic surgical images by introducing three core modules (Topological Coupled Inference Field (TRF), Boundary Dispersion Dynamic Field (BDDF), and Multi-Scale Dynamic Cooperative Field (MDSF)) working collaboratively. The model comprises the following modules: a Topology-coupled Reasoning Field (TRF), employing graph neural networks and topological propagation to capture long-range structural information and suppress fragmentation in surgical instrument images; a Boundary Dispersive Dynamics Field (BDDF), combining attention from the structural and dynamic domains to accurately capture boundary details of the image; and a Multi-scale Dynamic Synergy Field (MDSF), adaptively fusing features from different scales to effectively address the multi-scale instrument morphology in surgical images. This invention demonstrates significant performance advantages in endoscopic surgical instrument image segmentation, possessing good scalability and practical application potential. It is suitable for segmenting complex occlusions, reflective interference, and multiple instrument intersections in endoscopic surgery, providing efficient and accurate image segmentation support for precise instrument positioning, tracking, and intraoperative navigation. The method designs various modular components to extract global and local features of the image in the structural and dynamic domains, thereby improving segmentation accuracy.

[0034] This application provides a method for segmenting endoscopic images of surgical instruments based on graph dynamics enhancement, including the following steps:

[0035] Step 1: Collect image data from multiple publicly available surgical instrument endoscopic medical image datasets, including Kvasir-Instrument, Synthetic MICCAI 2020, and UW-Sinus-Surgery-C / L datasets. Each dataset contains image-annotated instrument regions to ensure the model has comprehensive training and testing sample sources.

[0036] Step 2: Organize and divide the collected image data to form training and test sets.

[0037] Step 3: Preprocess the image data from Step 2. The data undergoes standardization to ensure consistency across different imaging devices or imaging conditions. This includes uniformly adjusting the image size (e.g., adjusting the image size to 1024×1024) and normalizing the pixel values ​​to the range [0,1]. Based on the ground truth mask of the image data, three types of mask prompts—boundary, point, and noise—are automatically generated to provide standardized images and diverse priors for subsequent model input.

[0038] Step 4: To achieve high-precision medical image segmentation, a Graph-Dynamics Augmented Foundation Model combining topological structure and multi-scale dynamics is proposed. It comprises three core modules: Topology-coupled Reasoning Field (TRF), Boundary Dispersive Dynamics Field (BDDF), and Multi-scale Dynamic Synergy Field (MDSF). For example... Figure 1 The framework shown consists of three parts: an image encoder, a cue encoder, and a mask decoder. The topologically coupled inference field and the boundary dispersive dynamic field are serially embedded in the mask decoder. The multi-scale dynamic cooperative field (MDSF) is used to perform multi-scale dynamic fusion of the two outputs to balance morphological integrity and boundary fineness.

[0039] When inputting data images, the preprocessed image along with three types of cues—boundaries, points, and noise masks—is fed into the model. The image is processed by an image encoder based on a ViT (Vision Transformer) architecture pre-trained on large-scale natural images. The cues are encoded into conditional vectors by the cues encoder. Features extracted by the image encoder first enter a topologically coupled inference field. This field, combined with multi-layer graph convolution, propagates the topological dependencies of the device's components globally, yielding structural enhancement features. Subsequently, the boundary dispersive dynamic field, using these structural enhancement features as input, introduces a differentiable Korteweg–de Vries equation and employs fourth-order Runge–Kutta discretization for several evolution steps in the feature domain. Through nonlinear propagation and dispersion smoothing, high-frequency artifacts are suppressed, and local fractures are repaired, resulting in optimized boundary features. Finally, the structural enhancement features and boundary optimization features are fused in the Multi-Scale Dynamic Cooperative Field (MDSF). The MDSF uses multi-scale selective convolution and channel gating to dynamically weight the two features according to channels and scale, and outputs a unified feature. This unified feature is concatenated with the conditional vector obtained by the cue encoder in the channel dimension, and a low-resolution mask is generated by the mask prediction head. The mask is then restored to the original image size by the upsampling module to obtain the final mask.

[0040] The structure and function of each core module are described in detail below, and the relevant mathematical formulas are given.

[0041] In one possible implementation, in the topologically coupled inference field (TRF), the feature map extracted by the image encoder is first denoted as... (in For batch size, For the number of channels, , (For spatial dimensions), and rearranged into a node representation matrix according to batch, channel, and spatial position. The order of the nodes corresponds one-to-one with the original spatial coordinates:

[0042]

[0043] Next, for each sample, the similarity is calculated based on the inner product of the channel dimension. Constructing Adjacency Tensors Specifically, for any two spatial locations within the same sample, their channel vectors are taken, multiplied element-wise along the channel dimension, and summed. The resulting scalar serves as a similarity measure between the two locations. All related measures are collected around each location, first exponentialized, then normalized by the sum of these values, ensuring that each weight is positive and summed to one, thus obtaining the adjacency weight tensor.

[0044]

[0045] in, , For channel-dimensional dot product operation, and Let represent the channel vectors of sample b at the i-th and j-th spatial locations (nodes), respectively.

[0046] The adjacency tensor is normalized using softmax to obtain graph structure features, reflecting the relative feature correlations between nodes. Subsequently, a multi-node propagation is performed through a graph convolution update operation to segment the graph structure features into connected regions, preserving local topological information and capturing global dependencies. The graph convolution update operation is expressed as:

[0047]

[0048] in, It is a non-linear activation function. For graph convolution kernel, These are the characteristics after propagation.

[0049] Finally, the updated features are rearranged according to the spatial index of the nodes to restore the feature map with the same height and width as the original. The number of channels is equal to the number of output channels set by the TRF, and this is output as the structure enhancement feature to achieve the integration of global and local features.

[0050]

[0051] in, Number of TRF output channels To convert back to the spatial domain feature representation, it achieves the capture of global complex structures and the adaptation of local morphology.

[0052] In one possible implementation, the boundary dispersive dynamical field (BDDF) first incorporates the output characteristics of the TRF. Considered as a continuous boundary field that varies with simulation time Its evolution is described using a differentiable KdV law:

[0053]

[0054] in, , and These are the coefficients controlling the nonlinear propagation term and the dispersion term, respectively. Used to control the evolution rate of the boundary. This represents the nonlinear propagation at the boundary, used to simulate the propagation trend at the mask boundary. As a dispersion term, it acts on the high-frequency region at the boundary, producing a harmonic noise suppression effect.

[0055] For each time step The evolution process is discretized, and the fourth-order Runge-Kutta method is used for approximate solution to update the boundary features. An evolutionary update process can be represented as:

[0056]

[0057] Each of the sub-items is:

[0058]

[0059]

[0060]

[0061]

[0062] in, The nonlinear propagation operator is approximated as:

[0063]

[0064] Through this dynamic evolution mechanism, the model achieves dynamic optimization in the structural domain, ensuring high accuracy in instrument positioning and reducing noise interference.

[0065] In one possible implementation, such as Figure 2 As shown, in the multi-scale dynamic cooperative field MDSF, parallel branching is first used to process the TRF-enhanced structural features. Boundary features of BDDF evolution Each branch includes convolution operations with different kernel sizes (providing different receptive fields) to extract complementary multi-scale features. After convolution, group normalization and non-linear activation are used to adjust the feature distribution. The structural and boundary features after adjusting the feature distribution are element-wise added to achieve preliminary fusion, resulting in the fused features. :

[0066] Next, MDSF adaptively balances the proportion of structural and boundary features through an attention mechanism. Specifically, for fused features... Apply global average pooling to obtain channel descriptors , ( (where the number of channels is used), and then the gating weight vector is extracted through a two-layer fully connected network. The first layer of the fully connected network will... Dimensionality reduction and ReLU activation The second fully connected network is then upgraded to the original number of channels, and finally passed through the Sigmoid function. Received This refers to the gating weights for each channel, used to control the proportion of structural / boundary information fused:

[0067]

[0068] in and This is a learnable weight matrix. Then, Attention coefficients are applied to structural features and boundary features respectively to achieve dynamic weighted fusion and obtain the fused output:

[0069]

[0070] in This indicates that the channel is multiplied element by element ( (Extended to the feature map dimension). The above formula realizes channel-wise adaptive modulation of structural and boundary features: if a certain channel value A higher proportion of boundary features indicates that structural features dominate that channel, while a lower proportion indicates that boundary features increase their proportion. MDSF also introduces a residual channel alignment mechanism: it maps the input structural features to the same channel dimension as the fused output using a 1×1 convolution, and then aligns them with the fused output. Element-wise addition creates residual connections to preserve the original information. This residual path ensures no information is lost during fusion and facilitates gradient propagation.

[0071] In summary, the MDSF module achieves organic coupling between structural semantics and boundary details through multi-scale convolution selection and attention fusion. Its multi-scale variable receptive field design enables neurons in each channel to adaptively select appropriate convolution kernel responses, thereby dynamically adjusting the fusion ratio of structural and boundary features for different parts; the dynamic weight mask generated by Sigmoid gating further ensures the balanced fusion of structural-boundary information in both spatial and channel dimensions.

[0072] Step 5: Train GDA-FM using the preprocessed and data augmentation dataset from Step 3. This framework is implemented in PyTorch and trained on dual NVIDIA 4090 GPUs. The optimizer is Adam, and the initial learning rate is set to 1×10⁻⁶. −4 The batch size is 4, and the input resolution is uniformly 1024×1024. Data augmentation includes: random scaling (including scaling that preserves proportions), random cropping and padding, and random horizontal flipping. After training, the predicted mask is interpolated back to the original size for evaluation.

[0073] During training, a weighted combination loss of binary cross-entropy and Dice similarity coefficient was used to ensure high accuracy and robustness of the model in instrument segmentation. The model's performance was evaluated on three benchmark datasets: Kvasir-Instrument, Synthetic MICCAI 2020, and UW-Sinus-Surgery-C / L.

[0074] Step 6: Apply the trained weights to the test set to evaluate the segmentation performance. We use key evaluation metrics to assess the model, including accuracy, Dice similarity coefficient (DSC), geometric mean (G-mean), and intersection over union (IoU). Accuracy measures the overall correctness of the classification and is calculated using the following formula:

[0075]

[0076] TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively.

[0077] DSC is an index that measures the similarity between two sample sets. The calculation formula is:

[0078]

[0079] in and These represent the sizes of the two sample sets, This represents their intersection. The value of DSC ranges from 0 to 1, with a higher value indicating a higher similarity.

[0080] G-Mean is a comprehensive metric for classifier performance, particularly suitable for handling class imbalance problems. Its calculation formula is:

[0081]

[0082] Sensitivity represents the proportion of correctly identified positive examples, and Specificity represents the proportion of correctly identified negative examples. A higher G-Mean value indicates better overall performance of the classifier when dealing with imbalanced datasets.

[0083] IoU is used to measure the degree of overlap between two point sets, and the calculation formula is:

[0084]

[0085] The value of IoU ranges from 0 to 1, with higher values ​​indicating a greater degree of overlap.

[0086] Based on the experimental results, the multi-module fusion basic model framework of this invention exhibits the best performance across almost all categories and metrics, demonstrating its significant advantages and strong competitiveness in surgical instrument endoscopic image segmentation tasks. This result further validates the effectiveness and robustness of the multi-module fusion model in handling complex medical image segmentation tasks.

[0087] surface Comparison of experimental results on the Kvasir-Instrument dataset

[0088] model Accuracy (%)↑ DSC (%)↑ G-Mean (%)↑ IoU (%)↑ AttentionUNet 98.48% 90.53% 94.19% 85.13% BAT 99.04% 94.99% 97.33% 91.26% DAttUNet 97.73% 89.02% 93.08% 83.27% DconnNet 99.00% 94.71% 97.49% 90.91% H_vmunet 98.51% 90.67% 95.80% 84.52% MDViT 98.19% 87.91% 92.54% 81.16% MRUNet 98.02% 87.83% 92.32% 81.09% R2U_Net 95.07% 81.11% 91.21% 73.36% SF_UNet 98.79% 92.14% 95.32% 87.44% SegNet 97.03% 87.60% 93.13% 80.71% TransUNet 96.54% 83.22% 93.03% 74.50% UNet 97.84% 88.54% 93.68% 82.87% UNext 98.57% 90.39% 93.87% 85.05% u-kan 98.68% 91.90% 94.52% 87.08% GDA-FM 99.19% 96.20% 97.85% 93.09%

[0089] The quantitative results for the Kvasir-Instrument dataset are shown in Table 1. The results demonstrate that the proposed GDA-FM(ours) performs well on multiple metrics. The quantitative results for the Synthetic MICCAI 2020 dataset are shown in Table 2.

[0090] surface Comparison of experimental results for the Synthetic MICCAI 2020 dataset

[0091] model Accuracy (%)↑ DSC (%)↑ G-Mean (%)↑ IoU (%)↑ AttentionUNet 98.30% 91.74% 93.02% 85.02% BAT 98.18% 93.95% 95.45% 89.30% DAttUNet 98.22% 91.45% 92.80% 84.52% DconnNet 99.27% 94.51% 95.90% 89.29% H_vmunet 97.63% 87.78% 90.50% 78.80% MDViT 97.97% 89.64% 92.14% 81.57% MRUNet 98.33% 91.91% 93.53% 85.25% R2U_Net 98.14% 91.06% 92.88% 83.89% SF_UNet 97.88% 89.97% 92.01% 82.14% SegNet 98.08% 90.48% 91.65% 83.11% TransUNet 97.69% 88.46% 89.88% 79.79% UNet 98.33% 91.80% 92.89% 85.11% UNext 97.92% 89.77% 91.48% 81.87% u-kan 98.08% 90.58% 92.02% 83.21% GDA-FM 98.91% 94.73% 95.58% 90.11%

[0092] surface Comparison of experimental results on the UW-Sinus-Surgery-C / L dataset

[0093] model Accuracy (%)↑ DSC (%)↑ G-Mean (%)↑ IoU (%)↑ AttentionUNet 97.71% 79.45% 81.90% 73.56% BAT 98.17% 84.67% 86.75% 79.64% DAttUNet 96.71% 73.43% 76.68% 65.72% DconnNet 97.99% 84.04% 87.08% 79.63% H_vmunet 98.75% 83.06% 85.62% 78.91% MDViT 98.37% 84.58% 86.49% 79.40% MRUNet 98.04% 80.23% 82.76% 75.20% R2U_Net 93.31% 70.57% 79.92% 61.71% SF_UNet 98.81% 84.31% 85.96% 80.38% SegNet 97.27% 75.66% 78.31% 68.96% TransUNet 92.81% 58.85% 67.44% 46.54% UNet 96.09% 73.08% 77.45% 64.74% UNext 97.98% 80.30% 82.53% 75.10% u-kan 98.83% 84.49% 86.12% 79.14% GDA-FM 98.91% 84.79% 86.96% 80.88%

[0094] To verify the impact of each module on the model, we evaluated the influence of key modules in GDA-FM on the overall model performance. The experimental results are shown in Table 4. By ablating the TRF Module, BDDF Field, and MDSF Field one by one, we were able to quantify the contribution of each module to the model.

[0095] surface Ablation experiments on key modules of the Graph-Dynamics Augmented FoundationModel on the Kvasir-Instrument dataset. √ indicates that the module is retained, and unchecked positions indicate that the module is removed from the model.

[0096] TRF BDDF MDSF Accuracy (%)↑ DSC (%)↑ G-Mean (%)↑ IoU (%)↑ 97.94% 94.03% 95.11% 90.18% √ √ 98.78% 94.90% 96.80% 91.08% √ √ 99.01% 95.50% 97.73% 91.92% √ √ 98.26% 95.37% 97.45% 91.77% √ √ √ 99.19% 96.20% 97.85% 93.09%

[0097] Experiments show that the model performs best when all modules work together. In the ablation experiments of each module, the removal of the TRFModule had the greatest impact on model performance, indicating that this graph-based processing method of the TRFModule plays a crucial role in capturing and coordinating global topological information. The absence of the BDDF Field led to a significant decrease in G-mean and Accuracy from 99.19 to 98.78, implying that the interaction between dynamic feature fusion and boundary evolution in the BDDF Field plays a significant role in the detection instrument and boundary accuracy. The removal of the MDSF module, however, caused DSC to decrease to 95.37 and IoU to increase to 91.77.

[0098] In summary, we propose a framework called Graph-Dynamics Augmented Foundation Model (GDA-FM), which combines graph structure propagation and dynamic boundary evolution to introduce a multi-scale fusion kernel. This kernel coordinates the balanced flow of features across the structural and dynamic domains, generating information-rich feature representations and improving the segmentation accuracy of endoscopic surgical instrument images. Specifically, the TRF Module integrates topological and structural information to obtain global morphological information. The BDDF Field utilizes the joint evolution mechanism of the structural and dynamic domains to capture the local heterogeneity and complex morphological features of the surgical instrument lesion region. The MDSF Field adaptively fuses the most suitable scale features to handle instrument structures at different scales, enhancing the model's adaptability to multi-scale features. Experiments show that the GDA-FM framework exhibits state-of-the-art performance in endoscopic surgical instrument image segmentation and provides important support for surgical navigation and instrument tracking in clinical practice.

[0099] This application also provides a graph-dynamic enhancement-based surgical instrument endoscopic image segmentation device, including the following modules:

[0100] The data preprocessing module is used to collect image data from multiple publicly available surgical instrument endoscopic medical image datasets; organize and divide the collected image data to obtain training and test sets; preprocess the image data, standardize the data to ensure consistency under different imaging equipment or imaging conditions, and automatically generate three types of mask prompts—boxes, points, and noise—based on the ground truth mask, providing standardized images and diverse priors for subsequent model input.

[0101] Model building module: Used to construct the graph dynamics enhancement foundation model GDA-FM, which combines topological structure and multi-scale dynamics. The model consists of three parts: an image encoder, a cue encoder, and a mask decoder. The mask decoder includes three core modules: the topologically coupled inference field (TRF), the boundary dispersive dynamic field (BDDF), and the multi-scale dynamic cooperative field (MDSF).

[0102] Model Training Module: The preprocessed dataset from the data preprocessing module is used to train the constructed graph dynamical augmentation model GDA-FM. The trained weights are then applied to the test set to evaluate the segmentation performance.

[0103] Image segmentation module: Implements the graph dynamics augmentation base model GDA-FM based on the pre-trained graph dynamics augmentation base model.

[0104] First, image data and three types of masking prompts (boxes, points, and noise) are input into a pre-trained graph dynamics enhancement model. Image data is processed by an image encoder based on a ViT (Vision Transformer) architecture pre-trained on large-scale natural images, while the prompts are encoded into conditional vectors by the prompt encoder. Features extracted by the image encoder are input into the topologically coupled inference field of the mask decoder. Multi-layer graph convolution transfers the topological dependencies of each component, outputting structural enhancement features. The boundary dispersion dynamics field takes the structural enhancement features output from the topologically coupled inference field as input, introduces differentiable Korteweg–de Vries equations, and simulates the nonlinear propagation and dispersion smoothing of the boundary using fourth-order Runge–Kutta discretization, ultimately obtaining optimized boundary features. The structural enhancement features and boundary optimization features are fused in a multi-scale dynamic collaborative field. The resulting unified feature is concatenated with the conditional vectors obtained from the prompt encoder along the channel dimension. An upsampling module restores the original image size, yielding the final mask.

[0105] This application also provides an electronic device, including a processor and a memory.

[0106] The memory is used to store computer programs.

[0107] When the processor executes a program stored in the memory, it implements any of the methods described in this application.

[0108] In one possible implementation, the electronic device of this application embodiment further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0109] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0110] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0111] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0112] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0113] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the methods described in this application.

[0114] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in this application.

[0115] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0116] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0117] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0118] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for segmenting endoscopic images of surgical instruments based on graph-dynamic enhancement, characterized in that, Includes the following steps: First, image data was collected from multiple publicly available surgical instrument endoscopic medical image datasets; The collected image data is organized and divided to obtain training and testing sets; The image data is preprocessed and standardized to ensure consistency under different imaging devices or imaging conditions. Based on its truth mask, three types of mask prompts are automatically generated: bounding boxes, points, and noise. Then, a graphical dynamics-enhanced foundational model, GDA-FM, combining topological structure and multi-scale dynamics is constructed. The model consists of three parts: an image encoder, a cue encoder, and a mask decoder. The mask decoder includes three core modules: topologically coupled inference field (TRF), boundary dispersive dynamic field (BDDF), and multi-scale dynamic cooperative field (MDSF). The preprocessed dataset was used to train the constructed graph dynamical augmentation base model GDA-FM; Finally, the weights obtained from training are applied to the test set to evaluate the segmentation performance; The specific operation of the graph dynamics-enhanced foundational model GDA-FM, which combines topological structure and multi-scale dynamics, is as follows: First, image data and three types of masking prompts (boxes, points, and noise) are input into the model. The image data is processed by an image encoder based on a ViT architecture pre-trained on large-scale natural images, while the prompts are encoded into conditional vectors by the prompt encoder. The features extracted by the image encoder are input into the topologically coupled inference field of the mask decoder. The topological dependencies of each component of the device are passed through multi-layer graph convolution, and the structure enhancement features are output. The boundary dispersion dynamic field takes the structure enhancement features output by the topologically coupled inference field as input, introduces a differentiable Korteweg–de Vries equation, and simulates the nonlinear propagation and dispersion smoothing of the boundary through fourth-order Runge–Kutta discrete simulation, finally obtaining the boundary optimization features. The structure enhancement features and the boundary optimization features are fused in a multi-scale dynamic collaborative field. The resulting unified features are concatenated with the conditional vectors obtained by the prompt encoder in the channel dimension. The original image size is restored through an upsampling module to obtain the final mask.

2. The method for segmenting endoscopic images of surgical instruments based on graph dynamics enhancement according to claim 1, characterized in that, In Topologically Coupled Inference Field (TRF), the feature maps extracted by the image encoder are first rearranged into a node representation matrix according to batch, channel, and spatial location. Next, an adjacency tensor is constructed for each sample based on the channel dimension inner product similarity. Specifically, for any two spatial locations within the same sample, their channel vectors are taken, multiplied element-wise along the channel dimension, and summed. The resulting scalar is used as a similarity measure between the two locations. All related measures are collected around each location, first exponentialized, and then normalized according to the sum of these values, so that the weights of each group are positive and summed to one, thus obtaining the adjacency tensor. The adjacency tensor is normalized by softmax to obtain graph structure features, which reflect the relative feature correlation between nodes. Then, multi-node propagation is performed through graph convolution update operation to segment the graph structure features into connected regions in order to preserve local topological information and capture global dependencies. Finally, the updated features are rearranged according to the spatial index of the nodes to restore the feature map with the same height and width as the original. The number of channels is equal to the number of output channels set by TRF, which is used as the output of enhanced structural features to achieve the integration of global and local features.

3. The method for segmenting endoscopic images of surgical instruments based on graph dynamics enhancement according to claim 2, characterized in that, The boundary dispersive dynamic field (BDDF) first treats the output characteristics of the topologically coupled inference field (TRF) as a continuous boundary field that varies with simulation time; and uses a differentiable KdV law to describe its evolution. For each time step, the evolution process is discretized, and the boundary features are updated by approximating the solution using the fourth-order Runge-Kutta method.

4. The method for segmenting endoscopic images of surgical instruments based on graph dynamics enhancement according to claim 3, characterized in that, In the Multi-Scale Dynamic Cooperative Field (MDSF), the structural features enhanced by the Transformer-Reinforced Functional Field (TRF) and the boundary features evolved by the BDDF are processed in parallel branches. Each branch includes convolution operations with different kernel sizes to extract complementary multi-scale features. After convolution, group normalization and nonlinear activation are used to adjust the feature distribution. The structural and boundary features with adjusted feature distributions are then element-wise added to achieve preliminary fusion, resulting in the fused features. Then, MDSF adaptively balances the proportion of structural and boundary features through an attention mechanism; Specifically, global average pooling is applied to the fusion features to obtain channel descriptors, and then gate weight vectors are extracted through two fully connected networks. The first fully connected network reduces the dimensionality of the channel descriptors and activates them with ReLU, while the second fully connected network increases the dimensionality to the original number of channels. Finally, the gate weights of each channel are obtained through the Sigmoid function to control the fusion ratio of structural / boundary information. Subsequently, the gating weights are applied as attention coefficients to the structural features and boundary features respectively to achieve dynamic weighted fusion and obtain the fused output; MDSF also introduces a residual channel alignment mechanism: the structural features of the input are mapped to the same channel dimension as the fused output through 1×1 convolution, and then added element-wise to the fused output to achieve residual connections to preserve the original information.

5. A surgical instrument endoscopic image segmentation device based on graph-dynamic enhancement, characterized in that, Includes the following modules: Data preprocessing module: used to collect image data from multiple publicly available surgical instrument endoscopic medical image datasets; organize and divide the collected image data to obtain training and test sets; preprocess the image data, and standardize the data to ensure consistency under different imaging equipment or imaging conditions, and automatically generate three types of mask prompts: bounding boxes, points, and noise based on its ground truth mask; Model building module: used to build the graphical dynamics enhancement basic model GDA-FM, which combines topological structure and multi-scale dynamics; The model consists of three parts: an image encoder, a cue encoder, and a mask decoder. The mask decoder includes three core modules: topologically coupled inference field (TRF), boundary dispersive dynamic field (BDDF), and multi-scale dynamic cooperative field (MDSF). Model training module: The preprocessed dataset from the data preprocessing module is used to train the constructed graph dynamical augmentation model GDA-FM; the trained weights are then applied to the test set to evaluate the segmentation performance. Image segmentation module: Implements the graph dynamical augmentation base model GDA-FM based on the pre-trained graph dynamical augmentation base model; First, image data and three types of masking prompts (boxes, points, and noise) are input into a pre-trained graph dynamics enhancement model. The image data is processed by an image encoder based on a ViT architecture pre-trained on large-scale natural images, while the prompts are encoded into conditional vectors by a prompt encoder. Features extracted by the image encoder are input into the topologically coupled inference field of the mask decoder. The topological dependencies of each component of the device are passed through multi-layer graph convolution, outputting structural enhancement features. The boundary dispersion dynamics field takes the structural enhancement features output by the topologically coupled inference field as input, introduces a differentiable Korteweg–de Vries equation, and simulates the nonlinear propagation and dispersion smoothing of the boundary through fourth-order Runge–Kutta discretization, finally obtaining boundary optimization features. The structural enhancement features and boundary optimization features are fused in a multi-scale dynamic collaborative field. The resulting unified features are concatenated with the conditional vectors obtained by the prompt encoder in the channel dimension. The original image size is restored through an upsampling module to obtain the final mask.

6. An electronic device, comprising a processor and a memory; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the surgical instrument endoscopic image segmentation method according to any one of claims 1-4.

7. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the surgical instrument endoscopic image segmentation method according to any one of claims 1-4.

8. A computer program product containing instructions that, when run on a computer, causes the computer to perform the surgical instrument endoscopic image segmentation method according to any one of claims 1-4.

Images