Image-based surgical instrument real-time segmentation method

Abstract

The invention discloses a surgical instrument real-time segmentation method based on an image, and the method comprises the steps: 1, constructing a surgical instrument real-time segmentation method network based on the image, 2, constructing a surgical instrument image segmentation data set of a deep learning network for image segmentation, and 3, carrying out the enhancement processing of the surgical instrument image data set, 4, using the surgical instrument to enhance the image segmentation data set, and training the image-based surgical instrument real-time segmentation method network, and 5, inputting the surgical instrument image collected in real time into the image-based surgical instrument real-time segmentation method network obtained through training in the step 4, and obtaining the image-based surgical instrument real-time segmentation method network. A mark image for accurately detecting the posture of the surgical instrument is output in real time; the method has the effect on common interference factors in a real operation scene, still can accurately identify and segment the surgical instrument under the interference condition, and has the characteristics of high precision, high real-time performance and strong robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image detection and segmentation, and in particular to a real-time segmentation method for surgical instruments based on images. Background Art

[0002] With the rapid development of robot-assisted surgery technology, surgical operations are gradually shifting from traditional open surgery to minimally invasive surgery. Medical robots can provide a more stable operating platform, allowing doctors to complete complex surgical steps with higher precision. However, in complex surgical environments, medical robots rely on precise target detection and real-time instrument recognition to ensure the safety and accuracy of operations.

[0003] In robot-assisted surgery systems, accurate identification and segmentation of surgical instruments play a vital role in improving surgical safety and effectiveness. During surgery, doctors need to make decisions quickly in a dynamic and complex environment, and real-time identification and positioning of various surgical instruments (such as scalpels, scissors, tweezers, etc.) are crucial to the success of the operation. However, traditional surgical instrument segmentation methods mainly rely on image processing-based rules or manual annotation, facing problems such as low accuracy, low efficiency, and sensitivity to environmental changes. These methods often make misjudgments or misses when faced with complex surgical scenarios, posing risks to the safety of surgery.

[0004] Since the early 1980s, surgical tool recognition and segmentation technology has been adopted in clinical practice. Dr. Roberts, MD, first applied this technology in actual clinical practice. He innovatively combined surgical microscopes and CT images, indirectly promoting the development of surgical navigation. Heilbrun et al. used the principle of machine vision to perform three-dimensional positioning through infrared cameras. Lee et al. first used RGB color space for surgical instrument detection. Subsequently, Doignon et al. used the HSV color model to distinguish the hue and brightness of the tip of the surgical tool and its surrounding environment, making color comparison easier. In addition, other technologies including random forests, SVM, and AdaBoost have also been developed and used for surgical tool recognition.

[0005] Although early surgical tool recognition and segmentation techniques have limitations, such as low accuracy of feature representation and slow response speed, their application in the medical field has been significantly enhanced with the advancement of multi-level learning technology, especially the introduction of models composed of convolutional layers, activation functions, and pooling layers. The emergence of this model has greatly enhanced the efficiency and accuracy of surgical instrument recognition and segmentation. For example, Garcia-Peraza-Herrera and his team developed an algorithm that combines a fully convolutional network system and dynamic tracking by optical flow, which enhances the recognition of spatiotemporal continuity of surgical tools, but consumes high computing resources. In addition, Chen et al. proposed a model composed of convolutional layers, activation functions, and normalization layers combined with line recognition. It has an impressive accuracy of 91.9% and a detection speed of 21 frames per second. However, the segmentation efficiency still needs to be optimized in surgical scenarios with high real-time requirements. Although the FasterR-CNN model developed by Jin and his team with end-to-end training has increased the accuracy to 81.8% and a speed of 5 frames per second. In addition, Azqueta and his team developed a strategy that integrates domain adaptation and semantic-level segmentation methods. Although this strategy has opened up a new path for the automatic identification of surgical tools and the evaluation of their effects, its lack of diversity in training samples limits its breadth of application.

[0006] With the rapid development of computer vision and deep learning technology, researchers have begun to explore the use of deep learning models for automatic segmentation of surgical instruments. Deep learning models can learn the complex features of surgical instruments through large-scale data training, thereby achieving higher accuracy. However, most of the existing deep learning models are often complex in structure, consume a lot of computing resources, and are difficult to meet the needs of real-time processing. During surgery, real-time performance is crucial, and doctors often need to obtain the segmentation results of instruments within a few milliseconds. Therefore, it is particularly important to design a lightweight, fast and accurate deep learning model. The present invention aims to provide an image-based real-time segmentation method for surgical instruments, which can efficiently complete the segmentation of surgical instruments, solve the shortcomings of existing methods, and meet the requirements of operating rooms for high-precision and high-real-time segmentation. Summary of the invention

[0007] The purpose of the present invention is to provide an image-based real-time segmentation method for surgical instruments to solve the problems of difficulty in maintaining the robustness and stability of instrument segmentation due to factors such as instrument jitter, local occlusion, insufficient lighting, smoke interference, etc. caused by limited field of view in the neurosurgical operating area and complex collaboration of multiple instruments.

[0008] To this end, the technical solution of the present invention is as follows:

[0009] A real-time segmentation method for surgical instruments based on an image, comprising the following steps:

[0010] Step 1, constructing an image-based surgical instrument real-time segmentation method network, which is formed by sequentially adding an FPN module, a bidirectional convolutional LSTM module and a CBAM module between the encoding module and the decoding module of the ConvNext-tiny network;

[0011] Step 2: Construct a surgical instrument image segmentation dataset using a deep learning network for image segmentation;

[0012] Step 3, performing enhancement processing on the surgical instrument image dataset obtained in step 2;

[0013] Step 4: Use the surgical instrument enhanced image segmentation dataset obtained in step 3 to train the deep learning network for surgical instrument image segmentation constructed in step 1;

[0014] Step 5: Input the surgical instrument image acquired in real time into the image-based surgical instrument real-time segmentation method network trained in step 4 to output a marked image with the surgical instrument posture accurately detected in real time.

[0015] Furthermore, in step 1, the number of double convolutional network layers constituting the bidirectional convolutional LSTM module is 10 to 15 layers.

[0016] Furthermore, the specific implementation steps of step S2 are:

[0017] Step 2.1, obtaining images of different surgical instruments during the operation to form an original image data set;

[0018] Step 2.2: Use the image annotation software labelme to manually annotate each image in the original image dataset to obtain annotated surgical instrument image segmentation dataset; the annotation method is: mark the outer edge of the surgical instrument in the image point by point to construct a closed contour line reflecting the shape of the surgical instrument. Each annotated image also generates segmentation information, including the surgical instrument category and the coordinates of all annotated points; images that do not contain surgical instruments are not annotated, and no segmentation information is generated accordingly.

[0019] Further, in step 3, the enhancement processing includes at least one of random cropping, color adjustment, contrast transformation, brightness transformation, noise disturbance, scaling, translation, and rotation.

[0020] Furthermore, the steps of step 4 are:

[0021] In step 4.1, the surgical instrument image dataset obtained in step 3 is divided into a training set, a validation set, and a test set in a ratio of 8:1:1;

[0022] Step 4.2: Input the training set obtained by step 4.1 into the deep learning network for surgical instrument image segmentation constructed by step 1, and train the image segmentation network with the labeled image in the segmentation data set as input and the image segmentation information as output; at the same time, during the training process, use the verification set to perform ablation experiments on the network after each round of training, and finally take the weight corresponding to the round with the best recognition accuracy in the training as the optimal weight and save it, and the training is completed.

[0023] Compared with the existing technology, this image-based real-time segmentation method for surgical instruments can effectively deal with common interference factors in real surgical scenes, such as instrument occlusion, jitter, partial incomplete display, and insufficient ambient light. Even under these interference conditions, the method can still accurately identify and segment surgical instruments, showing high adaptability in complex and changing surgical environments, with the characteristics of high precision, high real-time performance, and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the image-based real-time segmentation method for surgical instruments of the present invention;

[0025] Figure 2 A schematic diagram of the network structure of the image-based real-time segmentation method for surgical instruments of the present invention;

[0026] Figure 3 It is a schematic diagram of the FPN module structure of the network of the image-based real-time segmentation method of surgical instruments of the present invention;

[0027] Figure 4 It is a schematic diagram of the CBAM module structure of the network of the image-based real-time segmentation method for surgical instruments of the present invention;

[0028] Figure 5 Schematic diagrams of eight types of surgical instrument images respectively included in the image-based real-time segmentation method for surgical instruments of the present invention;

[0029] Figure 6 A schematic diagram of a labeled image of a segmented data set in the image-based real-time segmentation method for surgical instruments of the present invention;

[0030] Figure 7 These are example diagrams of the original labeled image before and after being processed using the Mosaic data enhancement method in the image-based real-time segmentation method of the present invention;

[0031] Figure 8 These are example diagrams of the original labeled image before and after being processed by the Albumentations image enhancement library in the image-based real-time segmentation method of the present invention;

[0032] Fig. 9Schematic diagram of the confusion matrix for binary classification;

[0033] Fig.10 A schematic diagram of a PR curve plotted by precision and recall rate for different types of surgical instruments in an embodiment of the present invention;

[0034] Fig.11 It is a schematic diagram of the intersection over union (IoU) of the evaluation index in the semantic segmentation in the real-time segmentation method of surgical instruments based on images of the present invention;

[0035] Fig.12 This is a comparison chart of the missed detection of the ConvNext-tiny network and the segmentation results of this application;

[0036] Fig.13 This is a comparison chart of the incomplete segmentation of the ConvNext-tiny network and the segmentation results of this application;

[0037] Fig.14 A schematic diagram of a mixed matrix of various types of surgical instruments in the image-based real-time segmentation method of surgical instruments of the present invention;

[0038] FIG. 15( a ) is a diagram showing the segmentation result of an image with tissue and organ occlusion in an embodiment of the present invention;

[0039] FIG15( b ) is a diagram showing the segmentation result of an image in which there are crossovers of surgical instruments in an embodiment of the present invention;

[0040] FIG15( c ) is a diagram showing the segmentation result of an image in which the surgical instrument is shaking according to an embodiment of the present invention;

[0041] FIG. 15( d ) is a diagram showing the segmentation result of an image in which the surgical instrument is not fully displayed according to an embodiment of the present invention;

[0042] FIG. 15( e ) is a diagram showing the segmentation result of an image with smoke interference in an embodiment of the present invention;

[0043] FIG15( f ) is a diagram showing the segmentation result of an image in a dark environment according to an embodiment of the present invention;

[0044] Fig.16 This is a visualization diagram of the surgical instrument segmentation result obtained by using the method of the present application in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention in any way.

[0046] See also Figure 1 ,The specific implementation steps of the image-based real-time segmentation method for surgical ,instruments are described as follows.

[0047] Step 1: Construct an image-based real-time segmentation method network for surgical instruments;

[0048] See also Figure 2 ,The surgical instrument image segmentation network is based on the existing ConvNext-tiny network and is improved by adding FPN module, bidirectional convolution LSTM module and CBAM module.

[0049] Specifically, the surgical instrument image segmentation network is composed of an encoding module, an FPN module, a bidirectional convolutional LSTM module, a CBAM module and a decoding module connected in sequence; among them, the encoding module and the decoding module adopt the encoding module and the decoding module of the ConvNext-tiny network, and the FPN module, the bidirectional convolutional LSTM module and the CBAM module are existing modules.

[0050] In this step 1, an FPN module is added after the encoding module of the ConvNext-tiny network, that is, the input end of the FPN module is connected to the output end of the encoding module; then, the feature information output by the encoding module is input into the FPN module, so that the feature information extracted by the encoding module is processed at different scales at the same time, so that in the subsequent decoding process, the coarse-grained global information and the fine-grained local information are fused; then, a bidirectional convolution LSTM module (BidirectionalConvLSTM) is added after the FPN module, that is, the output end of the FPN module is connected to the input end of the bidirectional convolution LSTM module; then, the added bidirectional convolution LSTM module is used to allow the structure to be decoded. The purpose of combining forward and backward features to capture richer temporal information is to effectively process the dependencies between consecutive frames and improve the segmentation effect; finally, a CBAM module is added after the bidirectional convolutional LSTM module, that is, the output end of the bidirectional convolutional LSTM module is connected to the input end of the CBAM module, so that the output results of each bidirectional convolutional network in the bidirectional convolutional LSTM module are input to the CBAM module; then, by introducing the CBAM attention mechanism before decoding, the model can dynamically adjust the weight of each time step according to the context of the frame, and use the characteristics of each frame to generate the corresponding attention weight, so as to more accurately fuse the information between frames; then, better cope with the differences in image quality or clinical features of different frames, and improve the effectiveness of temporal information fusion, such as Figure 4 shown.

[0051] Therefore, the specific structure of the network of the image-based real-time segmentation method of surgical instruments of the present application includes:

[0052] (1) Image input, the image input size is 256×256×3;

[0053] (2) The encoder part consists of a PatchEmbedding module, three ConvNext blocks, a Downsampling module, three ConvNext blocks, a Downsampling module, and nine ConvNext blocks connected in sequence; each ConvNext module has the same architecture, consisting of sequentially connected depthwise convolution, layer normalization, point-by-point convolution, GELU activation function, and residual connection;

[0054] (3) A newly added improved part is composed of an FPN module, a bidirectional convolutional LSTM module and a CBAM module in sequence; wherein the FPN module is composed of a backbone network, a lateral link, an upsampling, a feature fusion and an output layer; the bidirectional convolutional LSTM module is composed of F layers of double convolutional networks; each layer of the double convolutional network has the same structure, and is composed of a forward ConvLSTM and a backward ConvLSTM; the CBAM module is composed of a channel attention module and a spatial attention module; the channel attention module is composed of a pooling layer, a fully connected layer and a channel weighting; the spatial attention module is composed of a pooling layer, a convolutional layer and a spatial weighting; in the development process of the improved network structure, the value of the number of layers F of the double convolutional network in the bidirectional convolutional LSTM module that can effectively achieve the image segmentation effect is 10 to 15; in this embodiment, F = 12;

[0055] (4) The decoder part consists of the first UpConv, the first DoubleConv module, the second UpConv, the second DoubleConv module and the Output Layer module; the feature map output by the first UpConv and DoubleConv is 192×32×32; the feature map output by the second UpConv and DoubleConv is 96×64×64; the feature map output by the Output Laye module is 1×256×256, and finally the surgical instrument image segmentation result map is output.

[0056] Step 2: Construct a surgical instrument image segmentation dataset using a deep learning network for image segmentation:

[0057] Specifically, the implementation steps of step 2 are as follows:

[0058] Step 2.1, obtain images of different surgical instruments during the operation to form an original image dataset. The number of images in the dataset should meet the requirements of network training;

[0059] In this embodiment, the images in this step are based on 23 glioma craniotomy videos provided by hospital experts, and frame extraction is performed to form an original intracranial surgical instrument image dataset, totaling 5091 annotated video frames; see Figure 5As shown in the partial images, the original intracranial surgical instrument image dataset includes bipolar coagulation forceps 1, surgical scissors, aspirator, bipolar coagulation forceps 2, scalpel, dura mater forceps, tissue forceps and tissue scissors, a total of 8 types of surgical instruments; among them, bipolar coagulation forceps 1 and bipolar coagulation forceps 2 are two different styles of bipolar coagulation forceps.

[0060] Step 2.2: Use the image annotation software labelme to manually annotate each image obtained in step 2.1 in turn to obtain an annotated surgical instrument image segmentation dataset, which provides high-quality training data support for the image segmentation task.

[0061] Among them, the labeling method is: mark the outer edge of the surgical instrument in the image point by point to construct a closed contour line reflecting the shape of the surgical instrument. Each labeled image also generates segmentation information, including the surgical instrument category and the coordinates of all labeled points, so that the trained surgical instrument real-time segmentation method network can clearly identify the shape and contour of the instrument even in a complex background; images that do not contain surgical instruments are not labeled, and no segmentation information is generated accordingly.

[0062] Finally, the labeled images generated by annotating the original image set and their corresponding image segmentation information together constitute a complete segmentation dataset.

[0063] Step 3: Enhance the surgical instrument image dataset obtained in step 2.

[0064] Specifically, the implementation steps of step 3 are as follows:

[0065] In this step, four surgical instrument images in the dataset are first selected. Through random cropping and splicing operations, these four images are spliced ​​into a new image according to specific rules. This splicing method makes the target objects present different scales and positions in the image, increasing the diversity of training data; in this way, four images are used each time BatchNorm is calculated, which indirectly increases the batch size and reduces the memory requirement. The spliced ​​images are input into the network as enhanced data for training, which improves the model's ability to learn small targets, such as Figure 7 shown.

[0066] In this implementation step, the Albumentations image enhancement library is used to perform multiple enhancements on each image to increase the diversity of the data set and improve the robustness of the network. The specific enhancement operations include: randomly adjusting the hue, saturation and brightness of the image to simulate the changes in the image under different lighting environments and enhance the network's adaptability to lighting changes; enhancing the image contrast to 1.5 times of the original image, or reducing it to 0.9 times, so as to help the network better learn the features under different contrast conditions and enhance the ability to extract detail information; randomly enhancing the image brightness to 1.5 times or reducing it to 0.9 times to simulate the image performance under different brightness conditions and improve the model's sensitivity to brightness changes; adding Gaussian noise to the image to simulate the low-quality images that may be encountered in the actual environment, which helps to improve the noise robustness of the network. Randomly cropping parts of the image and re-adjusting the position and scale of the target helps the network adapt to the different scales and perspectives of the target and improves the model's tolerance to spatial transformations. Randomly rotating the image with a maximum rotation angle of 100 degrees enables the network to learn the rotation-invariant features of the image and enhance its ability to recognize targets at different angles, such as Figure 8 shown.

[0067] In step 3, all images in the surgical instrument image dataset were processed using a variety of data enhancement techniques, thereby increasing the number of images in the dataset. This operation effectively prevented the overfitting problem of the lightweight deep learning network for surgical instrument image segmentation during training and significantly improved the generalization ability of the network; after being processed by a variety of enhancement methods, the total number of images in the dataset reached 7103, corresponding to a total of 15706 surgical instruments.

[0068] Step 4: Use the surgical instrument enhanced image segmentation dataset obtained in step 3 to train the deep learning network for surgical instrument image segmentation constructed in step 1, so as to achieve the goal of real-time detection of surgical instrument posture.

[0069] Specifically, the implementation steps of step 4 are as follows:

[0070] In step 4.1, the surgical instrument image dataset obtained in step 3 is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0071] Step 4.2: Input the training set obtained by step 4.1 into the deep learning network for surgical instrument image segmentation constructed by step 1, and train the image segmentation network with the labeled image in the segmentation data set as input and the image segmentation information as output; at the same time, during the training process, use the verification set to perform ablation experiments on the network after each round of training, and finally take the weight corresponding to the round with the best recognition accuracy in the training as the optimal weight and save it, and the training is completed.

[0072] In this embodiment, the training parameters are shown in Table 1 below.

[0073] Table 1:

[0074] Parameter name Parameter Value Training size 256*256 Batch size 8 Iterations 100 Initial learning rate 0.0001

[0075] Step 5: Input the surgical instrument image acquired in real time into the deep learning network for surgical instrument image segmentation trained in step 4, and output a marked image with the surgical instrument posture accurately detected.

[0076] Furthermore, in order to evaluate the effectiveness of this method in detecting the segmentation results of surgical instruments, the segmentation accuracy and speed of the method of the present application were evaluated respectively; specifically, the evaluation indicators included precision (Precision), recall rate (Recall), average precision (Average Precision), mean average precision (mean Average Precision, mAP), Dice score, intersection over union (IoU) and the number of detected images per second (FPS).

[0077] like Fig. 9 The figure shows a schematic diagram of the confusion matrix for binary classification; in the figure, T and F refer to the positive and negative samples of the true value, and P and N refer to the positive and negative predicted values. Specifically, T represents the "correct" result, which refers to the situation where the model prediction is consistent with the actual situation, F represents the "wrong" result, which refers to the situation where the model prediction is inconsistent with the actual situation, P usually refers to the category predicted by the model as "interesting". In surgical instrument segmentation, the positive example usually refers to the surgical instrument area, and N refers to the category predicted by the model as "non-interesting", which usually refers to the background area or non-surgical instrument area. There are four classification results in total; among them, True Positive (TP): the model predicts a positive example (such as a surgical instrument), and it is actually a positive example; True Negative (TN): the model predicts a negative example (such as a non-surgical instrument or background), and it is actually a negative example; False Positive (FP): the model predicts a positive example (such as a surgical instrument), but it is actually a negative example (such as a non-surgical instrument or background); False Negative (FN): the model predicts a negative example (such as the background or other non-instrument parts), but it is actually a positive example (i.e., the surgical instrument part).

[0078] Precision is a measure of how many of the regions predicted by the model as "surgical instruments" are correct. In other words, precision focuses on how many of the regions predicted by the model as surgical instruments are actually surgical instruments. Precision measures the accuracy of the model when predicting the positive class (i.e. surgical instruments). High precision means that the model rarely mistakenly predicts non-instrument parts as instrument areas, that is, the model has fewer false positives. Its expression is:

[0079]

[0080] Recall, also known as sensitivity, is a measure of the proportion of correct predictions made by the model among all regions that are actually "surgical instruments". In short, recall focuses on how many regions of actual surgical instruments are identified by the model. Recall measures the proportion of correct predictions made by the model among all regions that are actually surgical instruments. A high recall means that the model misses fewer regions of surgical instruments, that is, fewer false negatives. Its expression is:

[0081]

[0082] like Fig.10 The figure shows a schematic diagram of the PR curves of different categories of surgical instruments drawn by precision and recall in an embodiment of the present invention. It can be seen from the figure that the PR curves of the eight surgical instruments are all close to the upper right corner, indicating that the model has both high precision (Precision) and high recall rate (Recall), proving that the method of the present invention has excellent segmentation performance in complex surgical scenarios.

[0083] The average precision is the average of the precisions of all categories (in multi-category tasks) or different prediction thresholds. It can be obtained by calculating different average precisions (AP), where the average precision of each category or each different threshold reflects the performance of the model under that category or threshold, and its expression is:

[0084]

[0085] The mean average precision (mAP) is the average precision of all categories, that is, the average AP value of each category of all categories. It is an indicator to measure the overall detection or segmentation performance of the model. It is usually used in multi-category tasks and can reflect the comprehensive performance of the model in different categories. Its expression is:

[0086]

[0087] The Dice score is an indicator that measures the model segmentation performance by calculating the overlap between the predicted area and the true label area. The value range is between [0,1]. The larger the value, the higher the similarity between the predicted result and the true value. Its expression is:

[0088]

[0089] Wherein, k is the number of surgical instrument categories. In this embodiment, k is 8;

[0090] Intersection over Union (IoU) is a commonly used evaluation metric, especially in object detection, semantic segmentation, and instance segmentation tasks. IoU measures the overlap between the predicted area and the true area, and is one of the important criteria for evaluating the accuracy of segmentation or detection results. The intersection over union (IoU) measures the segmentation or detection effect of the model by calculating the ratio of the intersection and union of the predicted area and the true label area. Its expression is:

[0091]

[0092] like Fig.11 The figure shows a schematic diagram of the intersection over union (IoU) during semantic segmentation of an image containing surgical instruments using the method of the present invention. The red annotation line is based on actual manual annotation of the image, and the green annotation line is the annotation line generated by the method of the present invention. The intersection over union of the two is 0.89, which is close to the maximum extreme value of 1. It can be seen that the degree of overlap between the evaluation prediction area and the real area is relatively high, proving that the image segmentation effect of the method of the present invention is reliable.

[0093] The number of detected images per second (FPS, Frames Per Second) is a standard indicator for measuring the processing speed of an image processing system (such as image segmentation, object detection, video analysis, etc.). It indicates how many frames of images the system can process per second (that is, how many images are detected, analyzed or segmented); FPS is an important indicator for evaluating the real-time performance and processing efficiency of the model, especially in real-time applications or video processing tasks. FPS (Frames Per Second) is the number of image frames processed per unit time (usually 1 second); for tasks such as image segmentation and object detection, FPS indicates the number of images that the model can analyze or process per second. Its expression is:

[0094]

[0095] In order to verify the superiority of the method of this application compared with the existing technology, we will evaluate the above indicators while conducting the same indicator evaluation on other existing technology methods and form a comparative analysis.

[0096] First, in order to verify the effectiveness of the method of this application, we used the same intracranial surgical instrument dataset to conduct ablation experiments on the method of this application. The specific test results are shown in Table 2 below.

[0097] The specific test results of the ablation experiment are shown in Table 2 below.

[0098] Table 2:

[0099]

[0100] In the network model category of Table 2, "ConvNext-tiny" means using the ConvNext-tiny network as the image segmentation network; "ConvNext-tiny+FPN" means that based on the ConvNext-tiny network, only the FPN module is added between the encoding module and the decoding module to form a network as the image segmentation network; "ConvNext-tiny+CBAM" means that based on the ConvNext-tiny network, only the CBAM module is added between the encoding module and the decoding module to form a network as the image segmentation network; "ConvNext-tiny+FPN+CBAM" means that based on the ConvNext-tiny network, only the FPN module and the CBAM module are added in sequence between the encoding module and the decoding module to form a network as the image segmentation network; "Conv Next-tiny+FPN+CBAM+BidirectionalConvLSTM" represents the image segmentation network of the present application.

[0101] From the experimental results in Table 2, it can be seen that the introduction of the FPN attention mechanism alone can effectively enhance the network's ability to extract important features, and its precision, recall rate, mAP and FPS are all improved; when the CBAM module is introduced into the network alone, although the recall rate has decreased to a certain extent, the other indicators have all improved; when the above two modules are introduced into the network at the same time, the FPS has a certain decrease, but the prediction is more accurate, and the other indicators have all improved; finally, based on the above three improvements to the ConvNext-tiny network, the image segmentation network of the present application has a precision of 2.4%, a recall rate of 2.6%, a mAP of 3.4%, and a FPS of 11.9% compared with the original ConvNext-tiny network.

[0102] Furthermore, in order to intuitively observe the detection effect of the improved algorithm, some detection results before and after the improvement are visualized and compared.

[0103] like Fig.12As shown, the comparison between the segmentation results of the present application and the ConvNext-tiny network is shown, wherein the left figure is the segmentation result obtained by training the ConvNext-tiny network, and the right figure is the segmentation result after training the image segmentation network proposed in the present application; from the comparison of the two figures, it can be seen that the ConvNext-tiny network has a missed detection problem in the segmentation process, that is, the part shown in the box in the right figure is not detected and marked in the left figure, while the improved network of the present application not only effectively solves the missed detection problem, but also significantly improves the segmentation accuracy.

[0104] like Fig.13 As shown, the comparison between the segmentation results of the present application and the ConvNext-tiny network is shown, wherein the left figure is the segmentation result obtained by training the ConvNext-tiny network, and the right figure is the segmentation result after training the image segmentation network proposed in the present application; from the comparison of the two figures, it can be seen that the ConvNext-tiny network has a segmentation defect problem in the segmentation process, that is, relative to the part shown in the right icon box, the part shown in the left icon box does not achieve a complete segmentation mark of the surgical instrument. It can be seen that the improved network of the present application not only effectively solves the segmentation defect problem, but also significantly improves the segmentation accuracy.

[0105] Fig.14 The confusion matrix of each category of surgical instruments is shown, which is a summary of the model's classification results for each category of surgical instruments in the dataset. This matrix reveals the categories of surgical instruments that the detection network is easily confused about when predicting, which can not only help us identify the errors that the model makes in classification, but also further understand the specific types of errors.

[0106] As shown in Table 3, the detection results of the accuracy, recall rate and average detection precision of eight surgical instruments included in the surgical instrument dataset using the method of the present application are shown.

[0107] Table 3:

[0108]

[0109] From the test results in Table 3, it can be seen that the accuracy of the eight surgical instruments is between 94.2% and 98.1%, the recall rate is between 92.5% and 98.4%, and the average detection precision is between 90.8% and 99.5%.

[0110] As shown in Figure 15(a), the segmentation result diagram of the image in the embodiment of the present invention is when there is tissue and organ occlusion; Figure 15(b) is the segmentation result diagram of the image in the embodiment of the present invention when there is a surgical instrument crossing; Figure 15(c) is the segmentation result diagram of the image in the embodiment of the present invention when there is a surgical instrument shaking; Figure 15(d) is the segmentation result diagram of the image in the embodiment of the present invention when there is an incomplete display of the surgical instrument; Figure 15(e) is the segmentation result diagram of the image in the embodiment of the present invention when there is smoke interference; Figure 15(f) is the segmentation result diagram of the image in the embodiment of the present invention when there is a dark environment; the segmentation result diagrams obtained in the above different situations clearly show that the method of the present application can still accurately detect multiple surgical instruments under interference conditions and maintain a high segmentation accuracy. This shows that the method has good adaptability in complex and changeable surgical environments, which is of great value for applications in actual clinical surgery and medical fields.

[0111] As shown in Table 4 below, the results of the surgical instrument posture estimation method of the present application and other surgical instrument posture estimation methods after training using the same data set; among them, the six other surgical instrument posture estimation methods listed in Table 4 are all well-known methods, and specific references are attached.

[0112] Table 4:

[0113] Method Precision(%) Recall(%) mAP(%) FPS <![CDATA[CGBA-Net [1] ]]> 85.1 87.2 78.3 42 <![CDATA[SANet [2] ]]> 87.9 89.5 85.3 71 <![CDATA[PraNet [3] ]]> 91.4 92.1 89.6 45 <![CDATA[MaskDINO [4] ]]> 95.3 93.6 95.3 89 <![CDATA[FastSegFormer [5] ]]> 96.3 93.7 94.1 109 <![CDATA[YOLOV8 [6] ]]> 95.7 93.5 96.9 113 This application 96.9 94.1 98.0 117

[0114] It can be clearly seen from the comparison results in Table 4 that, compared with other algorithms, the image segmentation effect achieved by the method of the present application has achieved the best result, with an accuracy of 96.9%, a recall rate of 94.1%, and mAP and FPS of 98.0% and 117 frames / second respectively.

[0115] like Fig.16 As shown in step S5 of the method of the present application, the surgical instrument image collected in real time is input into the image segmentation network trained by step 4, and the marked images of the different surgical instrument postures accurately detected can be output in real time, including the posture diagram of bipolar coagulation forceps 1 (tool 1), the posture diagram of surgical scissors (tool 2), the posture diagram of the suction device (tool 3), the posture diagram of bipolar coagulation forceps 2 (tool 4), the posture diagram of the scalpel (tool 5), the posture diagram of dura mater forceps (tool 6), the posture diagram of tissue forceps (tool 7), and the posture diagram of tissue scissors (tool 8).

[0116] In summary, this application proposes a surgical instrument segmentation algorithm based on the original ConvNext-tiny algorithm. The FPN module is introduced into its backbone network to fuse coarse-grained global information with fine-grained local information by simultaneously processing features at different scales. The BidirectionalConvLSTM module is introduced to capture timing information in continuous frames. The CBAM module is introduced to further filter out important timing features, allowing the network to pay more attention to key time-dependent information. For example, in continuous frames, for features with small or similar changes, the channel attention module will automatically reduce their weights and enhance channels with key timing information, thereby improving the segmentation accuracy of the model. The effective combination of the three enables surgical instrument segmentation.

[0117] References:

[0118] [1]Y.Wang, "CGBA-Net: context-guided bidirectional attention network for surgical instrument segmentation," Int J.Comput Assist Radiol.Surg., vol.18, no.10, pp.1769-1781.Oct.2023.

[0119] [2]J.Wei et al., "Shallow attention network for polyp segmentation,"

[0120] International conference on medical image computing and computer-assisted intervention.pp 699-708, 2021.

[0121] [3]D.Fan et al., "PraNet: parallel reverse attention network for polypsegmentation." Medical image computing and computer assisted intervention-

[0122] MICCAI 2020,pp 263-273,2020.

[0123] [4]W.Lu et al.,“High-Quality Object Detection Method for UAV ImagesBased on Improved DINO and Masked Image Modeling,”Remote Sens.,vol.15,no.19,2023,Art.no.4740.

[0124] [5]X.Cai et al.,“FastSegFormer:Aknowledge distillation-based methodfor real-time semantic segmentation of surface defects in navel oranges,”Comput.Electron Agr.,vol.217,Feb.2024,Art.no.108604.

[0125] [6]J.Terven,D.Cordova-Esparza,“Acomprehensive review of YOLO:FromYOLOv1 to YOLOv8 and beyond,”arXiv 2023.

Claims

1. A real-time segmentation method for surgical instruments based on an image, characterized in that the steps include: Step 1, constructing an image-based surgical instrument real-time segmentation method network, which is formed by sequentially adding an FPN module, a bidirectional convolutional LSTM module and a CBAM module between the encoding module and the decoding module of the ConvNext-tiny network; Step 2: Construct a surgical instrument image segmentation dataset using a deep learning network for image segmentation; Step 3, performing enhancement processing on the surgical instrument image dataset obtained in step 2; Step 4: Use the surgical instrument enhanced image segmentation dataset obtained in step 3 to train the image-based surgical instrument real-time segmentation method network constructed in step 1; Step 5: Input the surgical instrument image acquired in real time into the image-based surgical instrument real-time segmentation method network trained in step 4 to output a marked image with the surgical instrument posture accurately detected in real time.

2. The image-based real-time segmentation method for surgical instruments according to claim 1, characterized in that: In step 1, the number of double convolutional network layers constituting the bidirectional convolutional LSTM module is 10 to 15 layers.

3. The image-based real-time segmentation method for surgical instruments according to claim 1, characterized in that: The specific implementation steps of step S2 are: Step 2.1, obtaining images of different surgical instruments during the operation to form an original image data set; Step 2.2: Use the image annotation software labelme to manually annotate each image in the original image dataset to obtain annotated surgical instrument image segmentation dataset; the annotation method is: mark the outer edge of the surgical instrument in the image point by point to construct a closed contour line reflecting the shape of the surgical instrument. Each annotated image also generates segmentation information, including the surgical instrument category and the coordinates of all annotated points; images that do not contain surgical instruments are not annotated, and no segmentation information is generated accordingly.

4. The image-based real-time segmentation method for surgical instruments according to claim 1, characterized in that: In step 3, the enhancement processing includes at least one of random cropping, color adjustment, contrast transformation, brightness transformation, noise disturbance, scaling, translation, and rotation.

5. The image-based real-time segmentation method for surgical instruments according to claim 1, characterized in that: The steps for step 4 are: In step 4.1, the surgical instrument image dataset obtained in step 3 is divided into a training set, a validation set, and a test set in a ratio of 8:1:1; Step 4.2: Input the training set obtained by step 4.1 into the deep learning network for surgical instrument image segmentation constructed by step 1, and train the image segmentation network with the labeled image in the segmentation data set as input and the image segmentation information as output; at the same time, during the training process, use the verification set to perform ablation experiments on the network after each round of training, and finally take the weight corresponding to the round with the best recognition accuracy in the training as the optimal weight and save it, and the training is completed.

Images