Real-time surgical tool detection method applied to robot-assisted surgery video analysis

Abstract

The application belongs to the technical field of computer-aided surgery, and particularly relates to a real-time surgical tool detection method applied to robot-assisted surgery video analysis, comprising the following steps: collecting a computer-aided surgery video and processing to obtain a surgery image; performing classification prediction, regression prediction and center degree prediction on the surgery image in a pixel-by-pixel prediction manner; obtaining a candidate bounding box of a surgical tool according to the regression prediction, obtaining a prediction score of the candidate bounding box by using the classification prediction and the center degree prediction, and processing the prediction score by using a post-processing non-maximum suppression algorithm to obtain a final bounding box. The application adopts an anchor-free convolutional neural network architecture, avoids complex anchor calculation, and significantly reduces the number of parameters. In combination with an attention mechanism, the expression capability of the network is enhanced, and the detection accuracy of the network is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer-aided surgery, and particularly relates to a real-time surgical tool detection method applied to robot-assisted surgery video analysis. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Computer-aided surgery is a new surgical aid method, which uses computer technology to assist surgeons or surgical robots to complete surgical operations. Unlike traditional surgery, computer-aided surgery can reduce surgical accidents caused by human subjective consciousness, fatigue and other factors, and can reduce the probability of medical staff being infected. In particular, the surgical tool detection technology in the computer-aided surgery system can assist surgeons in judging and deciding the surgical process by detecting and identifying the position and category of minimally invasive surgical tools in the human body, has a certain navigation function, and makes the surgery safer.

[0004] Real-time surgical tool detection is one of the core technologies in the field of computer-aided surgery, which aims to mark the specific position of surgical tools in the image with a box and give the category of the surgical tools, so as to provide real-time position information of different tools for surgeons or assisting robots, improve surgical efficiency, and reduce the occurrence of complications. Because the surgical tool detection technology is used in surgical video images, it is necessary to achieve both speed and accuracy in the detection process of surgical tools, that is, real-time and accuracy are the requirements of surgical tool detection.

[0005] Unlike general target detection tasks, actual images used for surgical tool detection often have blood mist, blur, and too fast moving speed, which are not conducive to surgical tool detection, which will lead to a decrease in surgical tool detection accuracy, thereby affecting the surgical navigation process and causing harm to the human body. On the other hand, real-time is necessary to help implement surgical navigation through surgical tool detection. If the real-time of surgical tool detection is not achieved, the doctor's vision in the surgical process will be delayed, causing unnecessary harm to the human body. However, the current surgical tool detection method is very time-consuming, and a large number of anchor boxes need to be generated as prior boxes, and even the prior boxes need to be mapped back to the image feature map, which increases the amount of calculation and is very time-consuming, and cannot achieve real-time effect. SUMMARY

[0006] In order to overcome the shortcomings of the prior art, the present application provides a real-time surgical tool detection method applied to robot-assisted surgery video analysis, which improves the speed and accuracy of surgical tool detection.

[0007] To achieve the above object, one or more embodiments of the present application provide the following technical solutions:

[0008] The first aspect of the present application provides a real-time surgical tool detection method applied to robot-assisted surgery video analysis, comprising:

[0009] Collecting a computer-assisted surgery video and processing to obtain a surgery image;

[0010] Classifying and predicting the surgery image in a pixel-by-pixel prediction manner, and performing regression prediction and center degree prediction;

[0011] Obtaining a candidate bounding box of the surgery tool according to the regression prediction, obtaining a prediction score of the candidate bounding box by using the classification prediction and the center degree prediction, and processing the prediction score by using a post-processing non-maximum suppression algorithm to obtain a final bounding box.

[0012] The second aspect of the present application provides a real-time surgical tool detection method applied to robot-assisted surgery video analysis, comprising:

[0013] Collecting a computer-assisted surgery video and processing to obtain a surgery image;

[0014] Classifying and predicting the surgery image in a pixel-by-pixel prediction manner, and performing regression prediction and center degree prediction;

[0015] Obtaining a bounding box of the surgery tool according to the regression prediction, and obtaining a final bounding box by using a post-processing non-maximum suppression algorithm.

[0016] The third aspect of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the steps of the real-time surgical tool detection method applied to robot-assisted surgery video analysis according to the first aspect of the present application.

[0017] The fourth aspect of the present application provides an electronic device, which comprises a memory, a processor, and a program stored in the memory and executable on the processor, and the processor executes the program to realize the steps of the real-time surgical tool detection method applied to robot-assisted surgery video analysis according to the first aspect of the present application.

[0018] The above one or more technical solutions have the following beneficial effects:

[0019] 1. The present application adopts an anchor-free convolutional neural network architecture, avoids complex anchor frame calculation, and significantly reduces the number of parameters. In combination with an attention mechanism, the expression ability of the network is enhanced, and the detection accuracy of the network is improved.

[0020] 2、The application uses a bridge module to build a collapsed bridge network to replace the traditional backbone network to extract features, and uses a simple multi-span stage path aggregation network to perform feature fusion, thereby reducing the complexity of the network, improving the detection speed while ensuring high detection accuracy, and meeting the real-time detection requirements.

[0021] Advantages of the additional aspects of the application will be partially given in the following description, partially will become apparent from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0022] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the application, and together with the description of the exemplary embodiments of the application and explanations thereof serve to explain the application, and do not constitute an improper limitation of the application.

[0023] Figure 1 It is the overall convolutional neural network framework diagram of example one;

[0024] Figure 2 It is the detailed structure diagram of the bridge module of example one;

[0025] Figure 3 It is the detailed structure diagram of the attention head of example one. DETAILED DESCRIPTION

[0026] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0027] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application.

[0028] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0029] Example one

[0030] The embodiment discloses a real-time surgical tool detection method applied to robot-assisted surgical video analysis, and the overall concept is:

[0031] Collect computer-assisted surgical video and process to obtain surgical images;

[0032] The surgical images are classified and predicted in a pixel-by-pixel prediction manner, and regression prediction and centrality prediction are performed;

[0033] According to the regression prediction, a candidate bounding box of the surgical tool is obtained, the classification prediction and the center degree prediction are used to obtain a prediction score of the candidate bounding box, and a post-processing non-maximum suppression algorithm is used to process the prediction score to obtain a final bounding box.

[0034] The specific steps include:

[0035] S1: collect a computer-aided surgery video, and frame the surgery video to obtain surgery images;

[0036] S2: initialize a neural network framework for training;

[0037] S3: input the surgery images obtained in step S1 into the neural network framework, and pre-process the surgery images;

[0038] S4: train the neural network framework to obtain classification prediction, regression prediction and center degree prediction of the surgical tool in the surgery images in step S3;

[0039] S5: according to the regression prediction, a candidate bounding box of the surgical tool is obtained, the classification prediction and the center degree prediction are used to obtain a prediction score of the candidate bounding box, and a post-processing non-maximum suppression algorithm is used to process the prediction score to obtain a final bounding box.

[0040] In a specific embodiment, step S1 specifically includes:

[0041] S11: collect the entire surgery process video by using an endoscope when the computer-aided surgery is performed, and the speed is 25 fps;

[0042] S12: further downsample the video collected in step S11 at a speed of 25 fps to 1 fps and save it as a surgery image; it should be noted that in the specific implementation, the original video is downsampled to the frame speed of the manually marked surgery video; the downsampled original video is downsampled to enrich the time information between video segments, which is beneficial to the accuracy of the surgery tool detection.

[0043] S13: repeat step S12 until all surgery videos are converted into surgery images.

[0044] Step S2 specifically includes:

[0045] S21: the types of the surgery tools appearing in the surgery video are C, where C=6, indicating that six types of surgery tools appear in the surgery video being processed. The size of the surgery image is 854*480, and in order to improve the training speed of the neural network, the resolution of the surgery image is uniformly set to 512*512.

[0046] S22: The surgical image obtained in step S21 is first subjected to feature extraction by the backbone network, and then the obtained low-level features and high-level features are subjected to feature fusion to improve the utilization rate of feature information. Finally, the output head is used to output classification prediction and regression prediction. The network framework is shown in FIG. 8. Figure 1

[0047] S23: The backbone network in step S22 is a collapse bridge network. First, a common 6x6 convolution module is used to expand the channel number and reduce the image size, and then four bridge modules are used for effective feature extraction. In this process, maximum pooling is used to reduce the feature resolution to reduce the parameter amount. The Nx before the bridge module is the number of residual block repeated stacks, and the number after each stage is the channel number output after the corresponding module. For more details of the bridge module, see FIG. 9. Figure 2 First, the input features are processed by N / 2 residual blocks, and then the obtained features are branched. One branch continues to be processed by N / 2 residual blocks and a 1x1 convolution module, and the other branch is only processed by a 1x1 convolution. Then, the two obtained feature layers and the feature layer obtained by inputting through a 1x1 convolution are spliced, and finally a 1x1 convolution module is used to change the channel to output the desired channel number. The residual block only contains a 1x1 convolution and a 3x3 convolution, see FIG. 10. Figure 2

[0048] S24: The feature fusion in step S22 is performed by a simple multi-stage path aggregation network. First, it uses a spatial pyramid module to process the final output features of the backbone network to improve the receptive field of the network. The spatial pyramid module uses four different scale convolution kernel sizes of 1x1, 5x5, 9x9, and 13x13 for maximum pooling processing. In order to reduce the parameter amount, a 1x1 convolution module is used before and after the spatial pyramid module to flexibly adjust the channel number.

[0049] After the spatial pyramid pool, four different scale prediction paths are introduced by upsampling and downsampling to detect surgical tools of different scales. Each path will perform feature fusion between high-level features and low-level features twice to improve information utilization. Taking the second path as an example, first, a 1x1 convolution is used to adjust the channel number of the output features of the spatial pyramid and the final output features of the backbone network, so that their size and channel number are the same, and then the features are fused with the features propagated upwards from the next path. The fused features are propagated upwards by maximum pooling on one hand, and processed by a 3x3 convolution on the other hand, and then fused with the features propagated downwards from the previous path and the features propagated upwards from the next path. Finally, the fused features are downsampled on one hand and input to the attention head for prediction on the other hand. The processing of other paths is similar, and more details are shown in FIG. 11.​​Figure 1 You can find out by reading it.

[0050] S25: The output head in step S22 uses an attention head to output predicted content. It employs three independent branches for classification prediction, regression prediction, and centrality prediction, respectively. "Centrality" describes the normalized distance from a location to the center of the object it is responsible for, reducing the weight of bounding boxes far from the object center, suppressing low-quality predicted boxes, and improving detection accuracy. Each branch first uses two 3×3 convolutions, then combines them with an attention mechanism module. This approach doesn't introduce too many parameters and enhances the network's ability to focus on useful features while suppressing the expression of useless features, thus promoting subsequent regression and classification and improving detection accuracy. More details are available in the appendix. Figure 3 This can be understood from the above. It is worth noting that the four prediction paths of the multi-stage path aggregation network share a single attention head, so each path introduces an adaptive parameter to meet the prediction needs at different scales.

[0051] Step S3 specifically includes:

[0052] S31: Process the input surgical images in batches;

[0053] S32: Preprocess the input batch of images, i.e., use random flipping, random adjustment of image hue / saturation / contrast, random erasure of a rectangular area in the image, etc. as data augmentation to enrich the training samples.

[0054] S33: Repeat step S32 until all batch data has been processed.

[0055] Step S4 specifically includes:

[0056] S41: Train the neural network framework to obtain surgical tool classification prediction, regression prediction and centrality prediction in the surgical image in step S3. Use centrality prediction to suppress low-quality prediction boxes and retain high-quality prediction boxes.

[0057] Step S5 specifically includes:

[0058] S51: Design a total loss function as follows:

[0059]

[0060] Where L cls It is classification loss, L ctn It is the centrality loss, L reg This is the regression loss, and λ is the balancing weight, set to 1. N pos p represents the number of positive samples. x,y t x,y and ctn x,y Representing feature maps F respectivelyi the classification prediction score, the regression prediction and the centerness prediction of the (x, y) position of the feature map F and respectively represent the classification label, the regression label and the centerness label of the (x, y) position of the feature map F i is an indicator function, which is 1 if the position has a target, i.e. is 1, otherwise 0.

[0061] To solve the imbalance problem between positive and negative samples, the classification loss L cls uses the focal loss function. While the regression loss L reg uses the GIOU loss function, which is defined as follows:

[0062]

[0063] where IoU is the ratio of the intersection part of the area of the predicted bounding box and the area of the real box to the union area of the two, which is used to evaluate the degree of overlap between the predicted box and the real box. A c is the area of the smallest rectangle enclosing the real box and the predicted box, and u represents the union area of the real box and the predicted box.

[0064] The centerness is a normalized distance from the position to the center of the object responsible for the position, i.e. its range is from 0 to 1, so the centerness loss L ctn uses the binary cross-entropy loss function.

[0065] S52: Repeat step S51, constantly learn and train the network, so that the value of the loss function in step S51 becomes smaller and smaller to a certain value and tends to be constant until the loss function curve of the convolutional neural network is fitted.

[0066] The loss function curve fitting represents the success of the neural network training. Given an input image, we pass it through the network by forward propagation to obtain the classification prediction, regression prediction and centerness prediction of each position on the feature map.

[0067] According to the regression prediction, the candidate bounding box of the surgical tool is obtained, the classification prediction obtains the classification score of the candidate bounding box, and the centerness prediction is multiplied by the classification score to obtain the final prediction score, so as to reduce the prediction score of the low-quality candidate bounding box far away from the center of the object. And use the post-processing non-maximum suppression algorithm to process the final prediction score to obtain the final prediction bounding box.

[0068] ​​In the process of target detection, a large number of candidate bounding boxes will be generated at the position of the same target, and these candidate bounding boxes may overlap with each other, at which time we need to use non-maximum suppression to find the best target bounding box and eliminate redundant bounding boxes; the specific algorithm process includes:

[0069] Sort the final prediction scores of all candidate bounding boxes, and select the highest prediction score and the corresponding candidate box;

[0070] Iterate through the remaining candidate boxes, and if the overlap area (IOU) with the current highest score candidate box is greater than a certain threshold, the candidate box is deleted.

[0071] Continue to select a highest score from the unprocessed candidate boxes, and repeat the above process.

[0072] Embodiment two

[0073] The embodiment discloses a real-time surgical tool detection system applied to computer-aided surgical video analysis, comprising:

[0074] A surgical image acquisition module configured to acquire a computer-aided surgical video and process the surgical video to obtain a surgical image;

[0075] A surgical tool prediction module configured to perform classification prediction, regression prediction and center prediction on the surgical image in a pixel-by-pixel prediction manner;

[0076] A surgical tool bounding box acquisition module configured to obtain candidate bounding boxes of the surgical tool according to the regression prediction, obtain prediction scores of the candidate bounding boxes using the classification prediction and the center prediction, process the prediction scores using a post-processing non-maximum suppression algorithm, and obtain final bounding boxes.

[0077] Embodiment three

[0078] The purpose of the embodiment is to provide a computer-readable storage medium.

[0079] The computer-readable storage medium stores a computer program, and the program is executed by a processor to implement the steps in the real-time surgical tool detection method applied to robot-assisted surgical video analysis according to Embodiment 1 of the present disclosure.

[0080] Embodiment four

[0081] The purpose of the embodiment is to provide an electronic device.

[0082] The electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor, and the processor executes the program to implement the steps in the real-time surgical tool detection method applied to robot-assisted surgical video analysis according to Embodiment 1 of the present disclosure.

[0083] The steps involved in the apparatuses of embodiments two, three and four above correspond to the method of embodiment one, and the detailed description can be found in the relevant description of embodiment one. The term "computer readable storage medium" should be understood to include a single medium or multiple media, one or more sets of instructions; and should also be understood to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by a processor and that causes the processor to perform any one of the methods of the present application.

[0084] Those skilled in the art should understand that the modules or steps of the present application described above can be implemented by a general computer device, or alternatively, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0085] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A real-time surgical tool detection method applied to robot-assisted surgical video analysis, characterized in that, include: Acquire computer-assisted surgical videos and process them to obtain surgical images; Surgical images are classified, regressed, and centrality predicted using a pixel-by-pixel prediction method. Candidate bounding boxes for surgical tools are obtained based on regression prediction. The prediction scores of the candidate bounding boxes are obtained using classification prediction and centrality prediction. The prediction scores are then processed using a post-processing nonmaximum suppression algorithm to obtain the final bounding boxes. After using a lightweight neural network framework to train a backbone network, the surgical images are used to extract and fuse features. The output head is then used for classification prediction, regression prediction, and centerness prediction. The backbone network is a bridge network, comprising one convolutional module and four bridge modules; The specific steps of the bridge module are as follows: First, the input features are processed through N / 2 residual blocks. Then, the resulting features are branched. One branch continues to be processed using N / 2 residual blocks and a 1×1 convolution module, while the other branch only performs a single 1×1 convolution. The feature layers obtained from these two branches are then concatenated with the feature layer obtained from the input after a single 1×1 convolution. Finally, a 1×1 convolution module is used to change the number of channels, outputting the desired number of channels.

2. The real-time surgical tool detection method for robot-assisted surgical video analysis as described in claim 1, characterized in that, The feature fusion is performed using a multi-stage path aggregation network, which includes: first, processing the final output features of the backbone network using a spatial pyramid module; then, deriving four prediction paths of different scales through upsampling and downsampling to detect surgical tools of different scales; each path will undergo feature fusion between high-level and low-level features twice; finally, the fused features are upsampled and propagated downwards on one hand, and input into the output head for prediction on the other hand.

3. The real-time surgical tool detection method for robot-assisted surgical video analysis as described in claim 2, characterized in that, The output head is an attention head, which outputs predicted content, including three independent branches for classification prediction, regression prediction, and centrality prediction.

4. The real-time surgical tool detection method for robot-assisted surgical video analysis as described in claim 1, characterized in that, The step of processing the final prediction score using a post-processing nonmaximum suppression algorithm to obtain the final predicted bounding box specifically includes: Sort all candidate bounding boxes by their final prediction scores, and select the candidate box with the highest prediction score and its corresponding score. Iterate through the remaining candidate boxes. If the overlap area with the current highest-scoring candidate box is greater than a certain threshold, delete the candidate box. Continue selecting the highest-scoring box from the unprocessed boxes and repeat the above process.

5. The real-time surgical tool detection method for robot-assisted surgical video analysis as described in claim 1, characterized in that, The training of the neural network includes: establishing a loss function for the neural network, making the value of the loss function smaller and smaller until it tends to remain constant, until the loss function curve of the convolutional neural network is fitted, which represents the completion of neural network training; The total loss function is shown below: ， in, It is classification loss. It is a centrality loss. It is a regression loss. This is the balancing weight, set to 1; Indicates the number of positive samples. , and Representing feature maps respectively The classification prediction score, regression prediction, and centrality prediction at the (x,y) position. , and Representing feature maps respectively The classification label, regression label, and centrality label at the (x,y) position; It is an indicator function; if there is a target at that position, i.e. ,but It is 1 if it is 1, otherwise it is 0.

6. A real-time surgical tool detection system applied to computer-aided surgical video analysis, characterized in that: include: The surgical image acquisition module is configured to: acquire computer-assisted surgical videos and process them to obtain surgical images; The prediction module for surgical tools is configured to perform classification prediction, regression prediction, and centrality prediction on surgical images in a pixel-by-pixel prediction manner. The bounding box acquisition module for surgical tools is configured to: obtain candidate bounding boxes for surgical tools based on regression prediction; obtain prediction scores for candidate bounding boxes using classification prediction and centrality prediction; and process the prediction scores using a post-processing non-maximum suppression algorithm to obtain the final bounding box. After using a lightweight neural network framework to train a backbone network, the surgical images are used to extract and fuse features. The output head is then used for classification prediction, regression prediction, and centerness prediction. The backbone network is a bridge network, comprising one convolutional module and four bridge modules; The specific steps of the bridge module are as follows: First, the input features are processed through N / 2 residual blocks. Then, the resulting features are branched. One branch continues to be processed using N / 2 residual blocks and a 1×1 convolution module, while the other branch only performs a single 1×1 convolution. The feature layers obtained from these two branches are then concatenated with the feature layer obtained from the input after a single 1×1 convolution. Finally, a 1×1 convolution module is used to change the number of channels, outputting the desired number of channels.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the real-time surgical tool detection method for robot-assisted surgical video analysis as described in any one of claims 1-5.

8. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the real-time surgical tool detection method for robot-assisted surgical video analysis as described in any one of claims 1-5.

Images