Instrument tracking method for surgical navigation

Abstract

The invention discloses an instrument tracking method for surgical navigation, and belongs to the technical field of surgical navigation, and the method comprises the following steps: S1, data collection and preprocessing: collecting surgical instrument image data, storing the surgical instrument image data as a unified local file in a structured manner, and preprocessing the data; s2, performing data annotation and data enhancement, and performing data annotation on a data set obtained after preprocessing by using an image annotation tool LabelImg; s3, constructing a detection model, wherein the constructed model comprises an input end, a backbone network, a neck network and a prediction network; s4, training a detection model; s5, surgical instrument detection; s6, surgical instrument tracking; and S7, surgical path planning. By adopting the instrument tracking method for surgical navigation, the real-time performance, the accuracy and the robustness of surgical instrument detection can be improved, and the instrument tracking method has an important effect on the development of surgical robots.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical navigation, and in particular to an instrument tracking method for surgical navigation. Background Art

[0002] Surgery is a highly complex medical procedure during which surgeons need to precisely operate various instruments to treat patients' problems. The success of an operation often depends on the skills and experience of the surgeon. During surgery, doctors usually use microscopes, endoscopes, or other imaging devices to observe the surgical area. However, the position and status of surgical instruments are usually not clearly visible, which increases the difficulty of the operation. The position and movement of surgical instruments may obscure other instruments or the surgical area, making them difficult to identify. This will undoubtedly increase the complexity and risk of the operation. Surgical instrument posture detection is an indispensable technology in robot-assisted surgery, which allows doctors to obtain more comprehensive visual information, assists doctors in better completing the operation, and further improves the safety of the operation.

[0003] The task of object detection is to find all objects of interest in an image and determine their categories and locations. It is one of the core issues in the field of computer vision. Before the emergence of deep learning, traditional object detection algorithms were mainly based on manually designed features. Compared with traditional feature-based object detection methods, deep learning-based object detection methods can learn both low-level and high-level features of images, and have important applications in scientific research and actual industrial production. Deep learning is a type of multi-layer neural network algorithm that can automatically learn the internal structure of data hidden in training data through supervised, semi-supervised or unsupervised training methods, and can convert pixel data in the input image into higher-order and more abstract hierarchical features.

[0004] The target detection algorithms based on deep learning are mainly divided into two categories, one is the two-stage detection algorithm, and the other is the single-stage detection algorithm. The two-stage detection algorithm converts the detection problem into a classification problem of the local image in the generated proposed area through explicit region proposals. The detection accuracy is high, but the detection speed is too slow to meet the real-time requirements; the single-stage detection algorithm can directly identify the category and location information of the object from the input image without the need to explicitly extract the candidate area. Compared with the two-stage algorithm, the detection speed of the single-stage detection algorithm has been greatly improved, which can meet the real-time requirements, but the detection accuracy is slightly lower. Summary of the invention

[0005] The purpose of the present invention is to provide an instrument tracking method for surgical navigation, which can improve the real-time, accuracy and robustness of surgical instrument detection and reduce surgical risks.

[0006] To achieve the above object, the present invention provides an instrument tracking method for surgical navigation, comprising the following steps:

[0007] S1, data collection and preprocessing, collects surgical instrument image data by crawling images of surgical instruments on web pages using Python and taking images from relevant surgical instrument books, stores them as unified local files in a structured manner, and preprocesses the data;

[0008] S2, data labeling and data enhancement, use the image labeling tool LabelImg to label the data set obtained after preprocessing; use image processing software or image processing library to enhance the image data set;

[0009] S3, detection model construction, the constructed model includes four parts: input end, backbone network, neck network and prediction network;

[0010] S4, detection model training;

[0011] S5. Surgical instrument testing;

[0012] S6, surgical instrument tracking;

[0013] S7. Surgical pathway planning.

[0014] Preferably, in step S1, preprocessing includes removing noise, removing abnormal data, and removing duplicate data to improve data quality and reliability.

[0015] Preferably, in step S2, when annotating data, the following method is used to determine whether the image meets the annotation standard: if the surgical instrument is displayed in no less than half of the image, all displayed parts of the surgical instrument are framed by a minimum rectangular frame, and the type of the surgical instrument is marked; if the surgical instrument is displayed in less than half of the image, it is not marked, and the marked content will be saved in a txt document with the name corresponding to the image, which contains the category name of the surgical instrument, the coordinates of the center point of the marked frame, and the length and width of the marked frame. The coordinates of the center point and the length and width of the marked frame are normalized to obtain an image data set.

[0016] Preferably, in step S3, the image data set obtained in step S2 is used as the input image of the input end, 4 images are selected from the input end, and randomly spliced ​​so that the model can recognize the target in a small range. The specific steps are as follows:

[0017] S301, randomly select the coordinates of the image stitching reference points, and randomly select four images;

[0018] S302, the four pictures are resized and scaled according to the reference points, and are placed at the upper left, upper right, lower left, and lower right positions of the large picture of the specified size of the model;

[0019] S303, mapping the relationship to the image label according to the size transformation method of each image;

[0020] S304: splice the large images according to the specified horizontal and vertical coordinates, and process the coordinates of the detection frame that exceeds the boundary.

[0021] Preferably, in step S3, the backbone network (Backbone) part adopts three main functional units: Focus slicing, spatial pyramid pooling (SPP) and cross stage partial network (CSP), and integrates the attention mechanism Squeeze-and-Excitation (SE) into the SPP module, wherein the attention mechanism SE includes compression operation, excitation operation and feature recalibration.

[0022] Focus slicing divides the input data into four parts, and the resolution of each part becomes half of that of the input. The cross-stage local network (CSP) is a set of cross-stage residual units. It divides the input into two branches, performs convolution on each branch, and then connects the two branches (Concat). Two CSP structures are designed, including CSP1_X and CSP2_X. CSP1_X is used in the backbone network and has a residual unit (Residual unit) to reduce the number of parameters and calculations of the model and better process edge information and high-frequency features. The CSP2_X structure is used in the neck network. The difference between CSP2_X and CSP1_X is that the residual unit is replaced with multiple convolution modules to enhance the network's feature extraction capabilities;

[0023] The main function of the SPP module is to fuse local and global features. It uses the maximum pooling methods of 1×1, 5×5, 9×9 and 13×13 to splice the feature maps after pooling operations at different scales, and then fuse the information from different scales together.

[0024] The compression operation performs global average pooling on the C×H×W (where C represents the number of channels, H represents the height, and W represents the width) features obtained by the SPP module to obtain a feature map of size C×1×1; the excitation operation uses a fully connected neural network to perform nonlinear transformation on the feature map of size C×1×1 after the compression operation; the feature recalibration uses the result obtained by the excitation operation as the weight and then multiplies it by the input feature.

[0025] Preferably, the neck network (Neck) mainly realizes the fusion of shallow-level graphic features and deep-level semantic features, wherein the path aggregation network is replaced by a weighted bidirectional feature pyramid network.

[0026] Preferably, the prediction network (Head) corresponds to the Detect module network, and the Detect module network structure is composed of three 1*1 convolutions, corresponding to three detection feature layers.

[0027] Preferably, the model training in step S4 includes training parameters, evaluation indicators, and modifying the parameter file of the model before training.

[0028] Preferably, in step S5, the surgical instrument image acquired by the camera in real time is detected according to the training result of step S4, and the coordinates (bounding box) and confidence (indicating the confidence level of the detection) of each detected target are output, and unreliable detection results are filtered out according to the confidence threshold.

[0029] In step S6, DeepSort algorithm is used to track surgical instruments. The tracking of DeepSort algorithm is completed by combining spatial position information and appearance features. For each detected target, DeepSort extracts its appearance features and associates them with the targets in the previous frame. Kalman filtering is used in DeepSort algorithm to predict and update the target state, and Hungarian algorithm is used to process and match the detection results in each frame to ensure the consistency of tracking.

[0030] The DeepSort algorithm is used to track and output the bounding boxes, identifiers, and category information of all tracked targets in each frame. The bounding box of each target is drawn in the image, and the identifier of the target is marked on the box for visualization, and the tracking results of each frame are maintained. For undetected targets, its trajectory is tracked according to the prediction of the Kalman filter until it is detected again or lost. If the target is not detected in multiple frames, it will be marked as lost and tracking will stop.

[0031] In step S7, a 3D model of the patient's anatomical structure is reconstructed using preoperative images (such as CT or MRI), and key organs, lesions, blood vessels, etc. are marked with different colors;

[0032] Rapidly-exploring Random Trees (RRT) are used to plan the path according to the three-dimensional workspace of the marked surgical area. According to the position of the instrument obtained by surgical instrument tracking and the changes in the surgical area, the path is locally replanned when an obstacle is detected entering the current path.

[0033] Therefore, the present invention adopts the above-mentioned instrument tracking method for surgical navigation, adopts a single-stage detection algorithm, introduces an attention mechanism, improves the feature extraction ability of the model, and thus better distinguishes surgical instruments with high similarity. It also optimizes the backbone network, improves the feature fusion strategy, and uses a weighted bidirectional feature pyramid network for multi-scale feature fusion, thereby improving the detection accuracy and practicality of surgical instruments in various interference environments.

[0034] At the same time, the surgical navigation method based on improved single-stage instrument detection is of great significance in surgical operations, as it can significantly improve the safety and accuracy of operations. By automatically identifying surgical instruments, the workflow of the operating room can be optimized, human recording errors can be reduced, and a more intuitive operating interface and decision support can be provided to surgeons during the operation. It also improves the accuracy of surgical video and image analysis, helping doctors to better identify and track surgical instruments, while providing rich data support for medical teaching and surgical training. The surgical navigation method based on improved single-stage instrument detection not only reduces the cognitive burden of doctors, but also helps doctors take timely measures through real-time feedback and alarm functions, significantly promoting the intelligent and refined development of surgical operations.

[0035] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of an embodiment of an instrument tracking method for surgical navigation of the present invention;

[0037] Figure 2 It is an algorithm structure diagram of an embodiment of an instrument tracking method for surgical navigation of the present invention;

[0038] Figure 3 It is the Focus structure diagram;

[0039] Figure 4 It is the SPP structure diagram;

[0040] Figure 5 It is the SE structure diagram;

[0041] Figure 6 This is the BiFPN structure diagram. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0043] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0044] Embodiment 1

[0045] The present invention provides an instrument tracking method for surgical navigation, the flow chart is as follows: Figure 1 As shown, the following steps are included:

[0046] S1. Data collection and preprocessing: surgical instrument image data is collected by crawling surgical instrument images on web pages using Python, taking images from relevant surgical instrument books, etc., and stored as a unified local file in a structured manner; a large number of images of different surgical instruments are collected, which have problems such as clutter, repetitiveness, and incompleteness. The collected images are processed through data preprocessing to remove noise, abnormal data, and duplicate data, thereby improving data quality and reliability.

[0047] S2. Data labeling and data enhancement: Use the image labeling tool LabelImg to label the data set obtained after preprocessing, and use the following method to determine whether the image meets the labeling criteria. If more than half of the surgical instrument is displayed in the image, all the displayed parts of the surgical instrument are framed by the smallest rectangular frame and the type of the surgical instrument is marked; if less than half of the surgical instrument is displayed in the image, it is not marked.

[0048] The specific annotation method is: select yolo format annotation in the image annotation tool LabelImg, drag the mouse to select the target, select the category label of the surgical instrument, and click Save to complete the annotation of a picture. The annotated content will be saved in a txt document with the corresponding name of the image. The document also contains the category name of the surgical instrument, the center point coordinates of the marked frame, the length and width of the marked frame, and the center point coordinates and length and width of the marked frame are normalized. There is no corresponding txt document for unlabeled images, so delete the unlabeled images;

[0049] Use image processing software or image processing libraries, such as the PIL library, to perform data augmentation on the labeled image dataset using methods such as mirror flipping, rotation, scaling, translation, cropping, and color transformation.

[0050] S3. Detection model construction, the algorithm structure diagram is as follows Figure 2 As shown in the figure, the constructed model mainly consists of four parts: input end, backbone network, neck network and prediction network; the model uses Mosaic data enhancement at the input end, using 4 pictures, randomly scaled, randomly cropped, and randomly arranged for splicing, so that the model can recognize targets in a smaller range. The specific steps are as follows:

[0051] S301, randomly select the coordinates (x, y) of the reference point for stitching the pictures, where x and y represent the coordinates of the reference point on the X and Y axes; randomly select four pictures;

[0052] S302, the four pictures are resized and scaled according to the reference points, and are placed at the upper left, upper right, lower left, and lower right positions of the large picture of the specified size;

[0053] S303, mapping the relationship to the image label according to the size transformation method of each image;

[0054] S304: splice the large images according to the specified horizontal and vertical coordinates, and process the coordinates of the detection frame that exceeds the boundary.

[0055] The backbone network uses three main functional units: Focus slicing, spatial pyramid pooling (SPP) and cross-stage partial network (CSP), and integrates the attention mechanism Squeeze-and-Excitation (SE) into the SPP module. Then compression operation, excitation operation and feature recalibration are performed.

[0056] exist Figure 3 In the example, the input data is divided into four parts through Focus slicing, and the resolution of each part becomes half of the input. Then, the four parts are connected in the channel dimension using the Concat operation, and the number of channels in the output feature map becomes four times that of the input, and the resolution becomes half of the input.

[0057] The cross-stage local network (CSP) is a set of cross-stage residual units that divide the input into two branches, perform convolutions on each branch, and then concatenate the two branches. The present invention designs two CSP structures, including CSP1_X and CSP2_X. CSP1_X is used in the backbone network and has a residual unit, which reduces the number of parameters and calculations of the model and better processes edge information and high-frequency features.

[0058] The main function of the SPP module is to fuse local and global features. Figure 4 In the , 1×1, 5×5, 9×9 and 13×13 maximum pooling methods are used to concatenate the feature maps after pooling operations at different scales, and fuse the information from different scales together. This effectively expands the receptive field of the backbone features and significantly separates the most important context features compared to simply using k×k maximum pooling, where k represents the height and width of the pooling window.

[0059] The algorithm structure of the attention mechanism SE is as follows Figure 5 As shown in Figure 2. The role of SE is to enhance important features and suppress general features. trIt can be regarded as a standard convolution operator. Each layer of the input feature map E is convolved with a 2D convolution kernel to finally obtain C feature maps of size H×W. Each feature map represents a channel number, forming a feature map U of size C×H×W.

[0060] F tr ={a 1 , …, a t , …, a C} represents the set of convolution kernels, a t represents the convolution kernel of the tth channel of the feature map U, a t Contains C′ 2D convolution kernels, where 1≤t≤C, and C′ represents the number of channels of the input feature map E.

[0061]

[0062] Where U t represents the feature map of the output of the tth channel, * represents convolution, and E represents the input feature map. represents the sth convolution kernel corresponding to the tth channel, E s Represents the feature map of the s-th channel in the feature map E.

[0063] The compression operation performs global average pooling on the obtained C×H×W features to obtain a feature vector z of size C×1×1, z=[z 1 , …z t , …z c ], let each channel be represented by a numerical value. sq It is a function of global average pooling on the channel.

[0064]

[0065] In the formula, z t represents the tth element of z, H and W represent the height and width of the feature map respectively, and U t (i, j) represents the value of the feature map of the tth channel at the i-th row and j-th column.

[0066] The excitation operation uses a fully connected neural network to perform nonlinear transformation on the C×1×1 feature map after the compression operation. The 1×1×C vector is reduced to 1×1×C / r through a fully connected layer, where r refers to the compression ratio, and then it is increased back to 1×1×C through a second fully connected layer. ex It stands for "activation function". It is a nonlinear function used to generate the attention weights of the channels.

[0067] w t =F ex (z t ,w 1 ,w2 )=σ(w 2 δ(w 1 z t ));

[0068] In the formula, σ and δ represent the two activation functions of Sigmoid and ReLU respectively, and w 1 、w 2 represents the weights of two different fully connected layers, w t Represents the weight of the feature map.

[0069] F scale Represents the feature weighting function, which weights the attention weight obtained above to the features of each channel. Each feature map is multiplied by the corresponding weight to obtain the final feature map

[0070]

[0071] The CSP2_X structure is used in Neck. The difference between CSP2_X and CSP1_X is that the residual unit is replaced by multiple convolution modules to enhance the network feature extraction capability. Figure 2 As can be seen in the figure, Neck interpolates the feature map by upsampling on the left side, making the scale of the feature map larger, so as to facilitate the fusion of the feature map from Backbone, and to perform upward fusion of features, so that the feature map continues to grow; Neck continues to downsample on the right side, firstly to obtain feature maps of different scales, and secondly to make the shallow graphic features (mainly including color, texture, edge and other information) and deep semantic features better integrated, rather than just a simple Concat. The role of the Neck layer is to combine the shallow graphic features with the deep semantic features to obtain more complete features.

[0072] The path aggregation network (PANet) is replaced by a weighted bidirectional feature pyramid network (Bidirectional Feature Pyramid Network, BiFPN) in the neck. BiFPN allows information in the feature network layer to flow and merge in both top-down and bottom-up directions. The BiFPN structure is as follows Figure 6 As shown, P m The resolution of the representative feature map is 1 / 2 of the input image size m times, where m represents the level of the feature map. Backbone outputs 3 feature maps and feeds them into BiFPN, with resolutions of P 3 , P 4 , P 5 , P 5 The feature map P is obtained by downsampling 6, P 7 .

[0073] The prediction network corresponds to the Detect module network. Figure 2 The Head layer in is the Detect module. The network structure of the Detect module consists of three 1*1 convolutions, corresponding to three detection feature layers.

[0074] The model contains three loss functions in total: classification loss, localization loss, and confidence loss.

[0075] Classification loss is used to measure the difference between the predicted category probability and the true category. It is calculated using Binary Cross Entropy Loss (BCE). The loss function is:

[0076]

[0077] Among them, Loss cls is the classification loss, is the predicted class probability and q is the true class label.

[0078] The positioning loss is used to calculate the difference between the predicted box and the real box. It is calculated using the CIoU (Complete Intersection over Union) loss. On the basis of considering the intersection over union (IoU) of the predicted box and the real box, the center point distance of the box and the aspect ratio of the box are also considered to improve the accuracy of regression. The loss function is:

[0079]

[0080] Among them, IoU(B p ,B h ) represents the intersection-over-union ratio between the predicted box and the true box, B p is the predicted bounding box, B h is the true bounding box, p represents the predicted value, h represents the true value, ρ(b p ,b h ) represents the Euclidean distance between the center points of the predicted box and the true box, b p and b h represents the center point of the predicted box and the real box, e represents the length of the diagonal of the minimum closure box surrounding the predicted box and the real box, v represents the similarity of the aspect ratio of the predicted box and the real box, d p Indicates the aspect ratio of the prediction box, d hRepresents the aspect ratio of the real box, α is used to balance the weight factor of IoU and aspect ratio loss, Loss box is the positioning loss.

[0081] Confidence loss is used to measure the network's confidence in whether the predicted box contains the target, and binary cross entropy loss is used to calculate the confidence loss. The loss function is:

[0082]

[0083] Among them, Loss obj is the confidence loss, is the predicted confidence, and f is the true confidence label (usually 1 for containing the target and 0 for not containing the target).

[0084] The total loss function is:

[0085] Loss = λ box ·Loss box +λ cls ·Loss cls +λ obj ·Loss obj ;

[0086] Among them, λ box , cls , obj The weight parameters corresponding to the positioning loss, classification loss and confidence loss are used to balance the contribution of the three to the total loss. box , cls , obj Set them to 0.05, 0.5, and 1.0 respectively.

[0087] S4, detection model training;

[0088] In step S4, model training includes training parameters and evaluation indicators. After a large number of experiments, the number of iterations is set to 300, the batch size is set to 4, the parameter file of the model is modified, and then training is performed.

[0089] The evaluation indicators include true positive (TP), false positive (FP), and false negative (FN) to calculate recall, accuracy, average precision (AP), and mean average precision (mAP) to verify the performance of the image detection model.

[0090] Calculate the recall rate Recall represents the ratio of true positives to the sum of true positives and false negatives, as shown in the following formula:

[0091]

[0092] Precision represents the ratio of true positives to all positives, as shown in the following formula:

[0093]

[0094] The average precision AP is as follows:

[0095]

[0096] Among them, P(R) represents the PR curve, that is, the values ​​of precision and recall obtained at different IoU thresholds, and is a curve obtained on a two-dimensional coordinate with the recall rate as the horizontal axis and the precision rate as the vertical axis.

[0097] The mean average precision mAP represents the mean AP of all categories in the model, as shown in the following formula:

[0098]

[0099] Where n represents the number of all categories in the model; AP g represents the average precision of the g-th category.

[0100] S5. Surgical instrument detection: detecting the surgical instrument image according to the evaluation index.

[0101] The specific implementation process of this detection method is as follows:

[0102] (1) Prepare a data set and divide it into training set, test set, and validation set in a ratio of 8:1:1;

[0103] (2) Building a network model for surgical instrument detection;

[0104] (3) Use the divided data set to experiment with the network model, test the results of each training, and save the weights with the best effect;

[0105] (4) Using the best saved weight, detect the surgical instrument image acquired by the camera in real time to obtain the detection result.

[0106] S6. Surgical instrument tracking, using the DeepSort algorithm for surgical instrument tracking. The DeepSort algorithm tracks by combining spatial position information and appearance features. For each detected target, DeepSort extracts its appearance features and associates them with the target in the previous frame. In the DeepSort algorithm, Kalman filtering is used to predict and update the target state, and the Hungarian algorithm is used to process and match the detection results in each frame to ensure the consistency of tracking.

[0107] The DeepSort algorithm is used to track and output the bounding boxes, identifiers, and category information of all tracked targets in each frame. The bounding box of each target is drawn in the image, and the identifier of the target is marked on the box for visualization, and the tracking results of each frame are kept. For undetected targets, its trajectory is tracked according to the prediction of the Kalman filter until it is detected again or lost. If the target is not detected in multiple frames, it will be marked as lost and tracking will stop.

[0108] S7, surgical pathway planning, uses preoperative images (such as CT or MRI, etc.) to reconstruct a 3D model of the patient's anatomical structure, marking key organs, lesions, blood vessels, etc. with different colors.

[0109] Rapidly-exploring Random Trees (RRT) are used to plan the path based on the three-dimensional workspace of the marked surgical area. Based on the position of the instrument obtained by surgical instrument tracking and the changes in the surgical area, the path is locally replanned when an obstacle is detected entering the current path.

[0110] Therefore, the present invention adopts the above-mentioned instrument tracking method for surgical navigation, which can improve the real-time, accuracy and robustness of surgical instrument detection, and plays an important role in the development of surgical robots.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An instrument tracking method for surgical navigation, characterized in that: The following steps are involved: S1, data collection and preprocessing, collects surgical instrument image data by crawling images of surgical instruments on web pages using Python and taking images from relevant surgical instrument books, stores them as unified local files in a structured manner, and preprocesses the data; S2, data labeling and data enhancement, use the image labeling tool LabelImg to label the data set obtained after preprocessing; use image processing software or image processing library to enhance the image data set; S3, detection model construction, the constructed model includes four parts: input end, backbone network, neck network and prediction network; S4, detection model training; S5. Surgical instrument testing; S6, surgical instrument tracking, multi-target tracking based on the position information and appearance features of target detection; S7. Surgical path planning: path planning is performed based on the real-time position data of the tracked instrument and the target area of ​​the surgery.

2. The instrument tracking method for surgical navigation according to claim 1, characterized in that: In step S1, preprocessing includes removing noise, removing abnormal data, removing duplicate data, and improving data quality and reliability.

3. The instrument tracking method for surgical navigation according to claim 1, characterized in that: In step S2, when labeling the data, the following method is used to determine whether the image meets the labeling standards: if the surgical instrument is displayed in no less than half of the image, all the displayed parts of the surgical instrument are framed by a minimum rectangular frame, and the type of the surgical instrument is labeled; if the surgical instrument is displayed in less than half of the image, it is not labeled.

4. The instrument tracking method for surgical navigation according to claim 3, characterized in that: The annotated content is saved in a txt document with the corresponding name of the image. The document contains the category name of the surgical instrument, the center point coordinates of the marked frame, and the length and width of the marked frame. The center point coordinates and length and width values ​​of the marked frame are normalized to obtain the image data set.

5. The instrument tracking method for surgical navigation according to claim 1, characterized in that: In step S3, the image data set obtained in step S2 is used as the input image of the input end, four images are selected from the input end, and randomly spliced ​​so that the model can recognize the target in a small range. The specific steps are as follows: S301, randomly select the coordinates of the image stitching reference points, and randomly select four images; S302, the four pictures are resized and scaled according to the reference points, and are placed at the upper left, upper right, lower left, and lower right positions of the large picture of the specified size of the model; S303, mapping the relationship to the image label according to the size transformation method of each image; S304: splice the large images according to the specified horizontal and vertical coordinates, and process the coordinates of the detection frame that exceeds the boundary.

6. The instrument tracking method for surgical navigation according to claim 5, characterized in that: The backbone network adopts three main functional units: Focus slicing, spatial pyramid pooling SPP and cross-stage local network CSP, and integrates the attention mechanism SE into the SPP module. The attention mechanism SE includes compression operation, excitation operation and feature recalibration. Focus slicing divides the input data into four parts, and the resolution of each part becomes half of the input; The cross-stage local network CSP is a set of cross-stage residual units. It divides the input into two branches, performs convolution on each branch, and then concats the two branches. Two CSP structures are designed, including CSP1_X and CSP2_X. CSP1_X is used in the backbone network and has a residual unit to reduce the number of parameters and calculations of the model and better process edge information and high-frequency features. The CSP2_X structure is used in the neck network. The difference between CSP2_X and CSP1_X is that the residual unit is replaced by multiple convolution modules to enhance the network feature extraction capability. The main function of the SPP module is to fuse local and global features. It uses the maximum pooling methods of 1×1, 5×5, 9×9 and 13×13 to splice the feature maps after pooling operations at different scales and fuse the information from different scales together. The compression operation performs global average pooling on the C×H×W features obtained by the SPP module to obtain a feature map of size C×1×1, where C represents the number of channels, H represents the height, and W represents the width; The excitation operation uses a fully connected neural network to perform nonlinear transformation on the feature map of size C×1×1 after the compression operation; Feature recalibration uses the result of the excitation operation as a weight and multiplies it by the input feature.

7. The instrument tracking method for surgical navigation according to claim 5, characterized in that: The neck network mainly realizes the fusion of shallow graphic features and deep semantic features, in which the path aggregation network is replaced by a weighted bidirectional feature pyramid network.

8. The instrument tracking method for surgical navigation according to claim 5, characterized in that: The prediction network corresponds to the Detect module network. The structure of the Detect module network consists of three 1*1 convolutions, corresponding to three detection feature layers.

9. The instrument tracking method for surgical navigation according to claim 1, characterized in that: In step S4, the model training includes training parameters, evaluation indicators, modifying the model parameter file and then performing training.

10. The instrument tracking method for surgical navigation according to claim 1, characterized in that: In step S5, the surgical instrument image acquired by the camera in real time is detected according to the training result of step S4, the bounding box coordinates and confidence of each detected target are output, and unreliable detection results are filtered out according to the confidence threshold; In step S6, the result outputted from step S5 is used as the input of the tracker, the appearance features are extracted and associated with the target in the previous frame to achieve tracking; In step S7, a fast expanding random tree algorithm is used to plan the surgical path by combining the real-time position data tracked, the target area of ​​the surgery, the operating characteristics of the surgical instruments and other factors.

Images