Method for generating 3D track of surgical instrument in surgical navigation

Abstract

The invention belongs to the technical field of surgical navigation, and particularly relates to a method for generating a 3D track of a surgical instrument in surgical navigation. According to the invention, the network structure of the depth prediction model is adjusted, and the features obtained by the bottom convolution layer and the features obtained by the high convolution layer are fused through the feature fusion module and the optimization module, so that the situation that the brightness features exceed the shape features and occupy the dominant position after multi-layer convolution is avoided, and the image quality is improved. And separating the high-brightness region from the low-brightness region to influence a depth prediction result. The technical problem that in the prior art, depth information prediction is not accurate, and consequently the generated 3D track is not accurate is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of surgical navigation, and in particular relates to a method for generating a 3D trajectory of a surgical instrument in surgical navigation. Background Art

[0002] With the continuous advancement of medical technology, the proportion of minimally invasive surgery in total surgical operations has also increased year by year. In today's minimally invasive surgery, it is first necessary to establish an operating hole at the site of surgery and implant surgical instruments. Then a doctor will operate the endoscope for detection, while another doctor will observe the diseased tissue in the endoscopic image and use special minimally invasive surgical instruments connected to the trocar to operate on the tissue. However, factors such as straight-rod instrument operation, narrow operating space, handheld lens field of view jitter, and unfriendly ergonomic design have increased the difficulty of endoscopic surgery. Therefore, in order to make the operation of minimally invasive surgery more standardized and reasonable, some researchers have begun to use auxiliary technical methods to monitor the surgical process. Using endoscopic images to track surgical instruments is one of the core parts that need to be achieved in minimally invasive surgery auxiliary technology. It can help surgeons evaluate the standardization of the surgical process and make the surgical process more precise and intelligent.

[0003] For 3D position tracking of surgical instruments, the existing mainstream methods use ultrasound and electromagnetics to directly physically locate metal surgical instruments. Although surgical instrument positioning can be achieved, in order to ensure high positioning accuracy, active sensors need to be connected to the tool tip, which will have many impacts on the implementation of the surgery. Therefore, although instrument tracking can be achieved by using a variety of hardware sensors, encoders or external optical systems, this method requires extensive hardware integration and is not conducive to integration in the operating room.

[0004] The existing Chinese invention patent application with application publication number CN118490357A and application publication date 2024.08.16 discloses a target positioning method, device and surgical navigation system for surgical navigation. The method obtains the coordinate information (i.e., 2D trajectory) and depth information of the target (i.e., surgical instrument) in the image through visual technology, and combines the 2D trajectory and the corresponding depth information to obtain the operation trajectory of the surgical instrument in 3D space.

[0005] The depth map is an image generated by taking the distance (depth) value from the image collector to each point in the scene as the pixel value. The methods for obtaining the scene depth map can be divided into two categories: passive ranging and active ranging. Active ranging generally sends light waves to the scene and calculates the distance by calculating the flight time TOF (time of flight) of the light; passive ranging receives light emitted or reflected from the scene, forms a distribution function of the scene light energy, and then restores the depth information of the scene based on it. The most commonly used method is similar to the human visual system, using two cameras separated by a certain distance to simultaneously acquire scene images to generate a depth map through parallax. However, a monocular camera has only one observation point, there is no parallax, and it is impossible to directly obtain the depth information in the image. Therefore, the depth measurement through the monocular camera (an endoscope) image requires the use of a deep learning network to estimate, so as to realize the 2D to 3D positioning and tracking of surgical instruments.

[0006] In terms of depth prediction of monocular images: David Eigen first proposed two-scale deep network depth prediction in 2014. This study was the first to directly use a neural network to regress depth. The network consists of two clusters of deep networks. Following the study of two-scale deep network depth prediction, David Eigen designed and implemented a convolutional neural network with three scales, which can be used for depth estimation, surface normal estimation, and semantic segmentation at the same time.

[0007] After Long J proposed the fully convolutional network in 2015 and He K proposed the classic residual neural network ResNet in 2016, Laina I used the fully convolutional network with residual learning for the first time in 2016 to predict the depth information of a single RGB image. As the network depth increases and the network structure becomes more complex, the final depth prediction Figure 1 Step by step, the details become richer and the predicted depth becomes more accurate.

[0008] However, the above two-scale convolutional network and full convolutional residual network will have huge deviations in depth prediction when encountering strong light. For areas with obvious lighting features, the predicted depth is significantly less than the actual depth (that is, the brighter area is predicted to be closer). The reasons are: first, neither the two-scale convolutional network nor the full convolutional residual network fuses the features obtained from the bottom convolutional layer with the features of the high-level convolutional layer. Therefore, after multiple layers of convolution, the brightness features surpass the shape features and dominate, resulting in separation of high-brightness areas and low-brightness areas; second, neither the two-scale convolutional network nor the full convolutional residual network performs loss calculations on the continuous changes in depth in the x and y directions, so the edges of real objects are largely ignored. This will have a greater impact on continuous depth prediction for videos with obvious brightness changes. Summary of the invention

[0009] The purpose of the present invention is to provide a method for generating a 3D trajectory of a surgical instrument in surgical navigation, so as to solve the technical problem in the prior art that inaccurate depth information prediction leads to inaccurate generated 3D trajectory.

[0010] In order to solve the above technical problems, the present invention provides a technical solution for a method for generating a 3D trajectory of a surgical instrument in surgical navigation: a method for generating a 3D trajectory of a surgical instrument in surgical navigation, the method comprising:

[0011] S1, obtaining image information of the surgical scene during the operation;

[0012] S2. Extracting depth information of the surgical instrument from the image information using a trained depth prediction model;

[0013] The depth prediction model includes an encoder, a decoder, a feature fusion module and an optimization module; the encoder is used to extract the features of the image information; the decoder is used to improve the resolution of the features output by the encoder; the feature fusion module is used to fuse the features extracted by different layers of the encoder; the optimization module is used to obtain the depth information according to the output of the decoder and the output of the feature fusion module;

[0014] S3. Obtain a 3D trajectory of the surgical instrument according to the depth information combined with the 2D coordinate information of the surgical instrument in a pixel coordinate system.

[0015] The beneficial effect of the above technical solution is that the technical solution of the method for generating 3D trajectories of surgical instruments in surgical navigation of the present invention belongs to an improved invention. The present invention adjusts the network structure of the depth prediction model, and fuses the features obtained by the bottom convolution layer and the features obtained by the high-level convolution layer through the feature fusion module and the optimization module, so as to avoid that after multi-layer convolution, the brightness feature surpasses the shape feature and occupies a dominant position, so that the high brightness area and the low brightness area are separated and affect the depth prediction result. The present invention solves the technical problem in the prior art that the depth information prediction is inaccurate, resulting in inaccurate generated 3D trajectory.

[0016] Furthermore, the encoder includes a convolution layer and at least one residual block, and the convolution layer and the residual block of the encoder are connected in sequence; each time a convolution layer or a residual block of the encoder is passed, the resolution of the output feature map decreases continuously; the feature fusion module performs fusion processing on the output features of each residual block;

[0017] The decoder includes a convolution layer and an upward projection deconvolution block. The number of the upward projection deconvolution blocks of the decoder is the same as the number of the residual blocks of the encoder. The upward projection deconvolution blocks of the decoder and the convolution layer are connected in sequence. Each time a convolution layer or an upward projection deconvolution block of the decoder is passed, the resolution of the output feature map is continuously improved.

[0018] Furthermore, the feature fusion module includes a convolution layer and an upward projection deconvolution block connected to the residual block of the encoder in a one-to-one correspondence; the feature maps output by the upward projection deconvolution block of the feature fusion module have the same size; the outputs of each upward projection deconvolution block are cascaded and input into the convolution layer.

[0019] Furthermore, the optimization module first cascades the output of the decoder and the output of the feature fusion module, and then obtains the depth information after processing through at least one convolutional layer.

[0020] Furthermore, the training process of the depth prediction model includes: firstly training the depth prediction model using a general data set, and then fine-tuning the depth prediction model trained by the general data set using a dedicated data set;

[0021] The general data set refers to a data set in which the data is not limited to the surgical scene; the dedicated data set refers to a data set in which the data only includes the surgical scene.

[0022] Furthermore, the loss function L of the depth prediction model is:

[0023]

[0024] L=L D +aL g

[0025] Among them, L D is the depth loss function, d i is the depth prediction value of the i-th pixel, is the true depth value of the i-th pixel, α is a positive constant; L g is the depth gradient loss function, e i is the L1 norm of the true depth value and predicted value of the i-th pixel, is the e at the i-th pixel i The gradient in the x direction, is the e at the i-th pixel i The gradient in the y direction; a is the weight coefficient of the depth gradient loss function, and n is the number of pixels of the input sample.

[0026] Furthermore, the 2D coordinate information is obtained according to the following method: extract N squares around the position coordinates of the surgical instrument at the current moment of the image information as candidate samples, N≥2; input the candidate samples into a trained binary classifier to obtain the score of each candidate sample, and use the position of the sample with the highest score as the predicted position of the surgical instrument at the next moment, and use the center point of the predicted position as the 2D coordinate of the surgical instrument.

[0027] Furthermore, the binary classifier includes at least two different binary classifiers, and the score of the candidate sample is obtained according to the following method: the normalized scores of the candidate sample output by each binary classifier are weighted added to obtain the score of the candidate sample; the normalized score is obtained after normalizing the scores of each candidate sample output by the binary classifier.

[0028] Furthermore, the binary classifier includes an SVM binary classifier and a CNN binary classifier; the last layer of the CNN binary classifier is a fully connected layer that uses a Logistic classification function to implement binary classification.

[0029] Furthermore, positive samples and negative samples are extracted from the image information at any time when the operation begins as training samples for the SVM binary classifier; the trained SVM binary classifier is used in the subsequent entire operation process;

[0030] Positive samples and negative samples are extracted from the image information at any time during the operation as training samples of the CNN binary classifier. The network parameters of the last layer of the CNN binary classifier are adjusted through the training samples of the CNN binary classifier, and the remaining network parameters are kept unchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A method flow chart of an embodiment of the method of the present invention;

[0032] Figure 2 A 2D trajectory generation flow chart of an embodiment of the method of the present invention;

[0033] Figure 3 A network structure diagram of a depth prediction model according to an embodiment of the method of the present invention;

[0034] Figure 4 The 2D motion trajectory of the center point of the bounding box of the tool tools0 in the method embodiment of the present invention;

[0035] Figure 5 The 2D motion trajectory of the center point of the boundary box of the tool tools1 in the method embodiment of the present invention;

[0036] Figure 6 The 3D motion trajectory of the center point of the boundary box of the tool tools2 in the method embodiment of the present invention;

[0037] Figure 7 The 3D motion trajectory of the center point of the boundary box of the tool tools3 in the method embodiment of the present invention;

[0038] Figure 8 The 2D motion trajectory of the center point of the boundary box of the tool tools4-1 in the embodiment of the method of the present invention;

[0039] Fig. 9 The 2D motion trajectory of the center point of the boundary box of the tool tools4-2 in the method embodiment of the present invention;

[0040] Fig.10 The 3D motion trajectory of the center point of the boundary box of tools 4-3 in the method embodiment of the present invention;

[0041] Fig.11 It is the 3D motion trajectory of the center point of the boundary box of the tool tools5 in the method embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention adjusts the network structure of the depth prediction model, and fuses the features obtained by the bottom convolution layer and the features obtained by the high-level convolution layer through the feature fusion module and the optimization module, so as to avoid that after multiple layers of convolution, the brightness feature surpasses the shape feature and takes the dominant position, so that the high brightness area and the low brightness area are separated and affect the depth prediction result. The present invention solves the technical problem that the depth information prediction in the prior art is inaccurate, resulting in inaccurate generated 3D trajectory.

[0043] Embodiment of a method for generating a 3D trajectory of a surgical instrument in surgical navigation:

[0044] A method for generating 3D trajectories of surgical instruments in surgical navigation, such as Figure 1 As shown, the method comprises the following steps:

[0045] Step 1: Realize 2D tracking of surgical instruments by using support vector machine detection model (i.e. SVM binary classifier) ​​and convolutional neural network detection model (i.e. CNN binary classifier).

[0046] Specifically, Figure 2 As shown in the figure, the combined support vector machine detection model and convolutional neural network detection model realizes the 2D visual tracking of surgical instruments.

[0047] (1) Support Vector Machine Detection Model: Support Vector Machine (SVM) is a statistical machine learning binary classification classifier proposed by Corinna Vapnik in 1995. It is suitable for small sample, high-dimensional nonlinear classification problems. Its core idea is to first map the samples in the original space to the high-dimensional space through linear or nonlinear changes, and then calculate the optimal classification hyperplane or hypersurface with the longest distance between the two types of samples in the high-dimensional space as the classification basis.

[0048] Let T = {(x1, y1), ..., (x i ,y i ),…,(x n ,y n)} is a set of data in the feature space, x i represents the eigenvector, y i Represents the class label. If the two sample sets are linearly separable, then there must be a hyperplane in the feature space that makes the geometric distance between the two sample sets the farthest. Let the hyperplane equation be: i +b=0.

[0049] The distance from each sample point to the hyperplane is defined as:

[0050] The optimal solution parameters of the hyperplane can be obtained by calculation:

[0051]

[0052] Among them, α is the Lagrangian factor, and the optimal classification hyperplane can be expressed as: w * ·x+b * =0

[0053] The classification decision function is: f(x) = sign(w * ·x+b * )

[0054] Use the polynomial kernel function instead of f(x)=sign(w * ·x+b * ) and we get: Bringing this result back to the above formula, we get the optimal hyperplane and classification decision function.

[0055] In endoscopic surgery videos, surgical instruments are rigid bodies with clear edges and main colors of black and silver, while the background outside the instruments is mainly flesh-colored tissue and dark red blood with no obvious edges. Therefore, this embodiment uses color features represented by HSV color histograms and directional features represented by directional gradient histograms (HOG) as manual features, and combines the two features into a support vector machine model for target detection.

[0056] Training the SVM model: Extract 500 positive samples (positive samples are squares containing surgical instruments) and 5,000 negative samples (negative samples are squares containing only surgical backgrounds without surgical instruments) from the first frame of the surgical video (i.e., images of surgical scenes taken by an endoscope during surgery), and normalize the image size. Extract the HSV color histogram features and directional gradient histogram features of the positive and negative samples and merge them to obtain the merged manual features of 5,500 samples. Use the linear kernel function to obtain the optimal classification hyperplane and classification decision function, and finally obtain the detection model, which is applied to all subsequent frames of the video (i.e., the entire subsequent surgical process), and the accuracy of judging whether the candidate sample is a positive sample is 98.5%.

[0057] (2) Convolutional Neural Network Model: The convolutional neural network (CNN) of this embodiment consists of three convolutional layers, two pooling layers, and three fully connected layers, where the last fully connected layer is also a binary classifier.

[0058] The image obtained by the input layer is 107*107 in size and has three RGB channels. The first convolution layer conv1 has 96 convolution kernels, 7*7 in size, 2 in stride, followed by a maximum pooling layer with a stride of 2; the second convolution layer conv2 has 256 convolution kernels, 5*5 in size, 2 in stride, followed by a maximum pooling layer with a stride of 2; the third convolution layer conv3 has 512 convolution kernels, 3*3 in size, and 1 in stride.

[0059] The convolutional layer conv3 is followed by two fully connected layers. The two fully connected layers have Dropout operations for random sampling to prevent overfitting, and finally convert the two-dimensional features output by conv3 into a one-dimensional vector. The last fully connected layer b6 uses the Logistic classification function to implement the role of a binary classifier. Each surgical video (i.e., each surgical process) has its own corresponding unique b6 layer, and the remaining hidden layers are the same. Training the CNN network: It is stipulated that the overlap rate with the predicted target bounding box is greater than 70% as a positive sample, and the overlap rate is less than 30% as a negative sample. Fifty positive samples and two hundred negative samples are selected in the selected frames for training, the learning rate is 0.03%, and the loss function weight decay coefficient is 0.05%.

[0060] After completing the training of the SVM binary classifier and the CNN binary classifier, 256 samples are sampled around the target position (i.e., surgical instrument) in each frame using Gaussian distribution as candidate samples for the next frame, denoted as x1, x2, …, x n The convolutional neural network model is used to score the 256 candidate samples. After normalizing all the scores, the score corresponding to each sample is the probability that the sample is a positive sample, denoted as p(x i ). At the same time, the support vector machine model is used to classify and predict the 256 candidate samples, and the probability of each sample being a positive sample is also obtained, which is recorded as g(x i ). The probabilities obtained by the two models are weighted and the highest score sample x is recorded. * Predict motion position for the next frame:

[0061] x * =argmax(αp(x i )+βg(x i )),i=1,2,...,256

[0062] Among them, α is the probability score weight of the convolutional neural network detection model, and β is the probability score weight of the support vector machine detection model.

[0063] Finally, the center point of the predicted position bounding box is taken as the 2D coordinate of the surgical instrument, and all center points are connected to generate the 2D motion trajectory of the surgical instrument.

[0064] Specifically, in this embodiment, when the score of the CNN binary classifier is too low, the CNN binary classifier is retrained based on the historical data during the current surgery.

[0065] In other implementations, only one binary classifier may be used, in which case the score output by the binary classifier is normalized to be the final score of the candidate sample.

[0066] In other implementations, only three binary classifiers may be used, and in this case, the score of the candidate sample is obtained by weighted addition of the normalized scores output by the three binary classifiers.

[0067] In other embodiments, the binary classifier may use other classifiers, such as a Bayesian classifier, a decision tree classifier, or other neural network classifiers.

[0068] Step 2: By using 3ds Max simulation software, RGB images with different lighting and camera angles in different scenes were created, and the depth map of the camera angle was rendered in each scene through rendering software, which was used as the training set for fine-tuning the depth map neural network.

[0069] By using surgical instrument obj files and tga material files, surgical scene simulation modeling was performed in 3ds Max software, creating a total of 16 different scenes, and generating 100 RGB images of camera perspectives by changing the lighting and camera position. Then, Vray renderer was used in each scene to render the scene depth, constructing a total of 100 RGB-depth image pairs for secondary training and fine-tuning of the network.

[0070] Step 3: Build a deep graph neural network (i.e., deep prediction model) suitable for surgical scenarios. The network consists of four parts: encoder, decoder, pyramid feature fusion module, and optimization module. Figure 3 As shown in the figure, the encoder is responsible for feature extraction, the decoder is responsible for improving the resolution, the multi-scale feature fusion module is responsible for cascading the features of different scales extracted by each layer of the encoder, and the optimization module is responsible for fine-tuning the depth and edge features.

[0071] Encoder: The encoder consists of a convolutional layer and four residual blocks. The resolution of the feature map decreases with each layer or block. The input is a 3-channel RGB image. The conv1 (convolutional layer) has 64 convolution kernels of size 7*7, a convolution step of 2, padding=3, and outputs a 64-channel image with a resolution of the original image. Figure 1 The features of the first residual block are doubled; block 1 (the first residual block) consists of multiple convolutional layers and pooling layers, the number of output feature channels is 256, and the resolution continues to drop by 1 / 2; block 2, 3, and 4 are similar to block 1. After each residual block, the number of output feature channels doubles and the resolution drops by 1 / 2. The encoder finally outputs a 2048-channel, and the resolution becomes the original Figure 1 / 32 features.

[0072] Decoder: The decoder consists of a convolutional layer and four upward projection deconvolution blocks. The resolution of the feature map increases with each block. The decoder input is a 2048-channel, original Figure 1 / 32 resolution feature image, conv2 has 1024 convolution kernels of size 1*1, convolution step size is 1, padding=0, output a 1024 channel, the resolution is still the original Figure 1 / 32 features; up1 consists of an unpooling layer and a convolutional layer, with an output feature channel number of 512 and a resolution doubled; up2, 3, and 4 are similar to up1. Each time an up block is passed, the output feature channel number is halved and the resolution is doubled. The decoder finally outputs a 64-channel, resolution of the original Figure 1 / 2 characteristics.

[0073] Pyramid feature fusion module: The feature fusion module first performs an upward projection operation on the output of the four residual blocks in the encoder, where up5 multiplies the output resolution of block1 by 2, and the number of output channels is reduced from 256 to 16; up6 multiplies the output resolution of block2 by 4, that is, two upward projection deconvolution operations are completed, and the number of output channels is reduced from 512 to 16; up7 and 8 are similar. The size of the feature maps after the four upward projections is 1 / 2 of the input original image, that is, the size of the output of conv1, and the number of channels is 16. Then the four feature results are cascaded and passed through a convolution layer conv3 with a convolution kernel size of 5*5, a step size of 1, and padding=2, and finally a resolution of the original image is obtained. Figure 1 Half, with 64 channels of feature output.

[0074] Optimization module: The optimization module first takes the 64-channel resolution as input Figure 1 Half the decoder output and the same 64-channel resolution as the input Figure 1The output of the half-pyramid feature fusion module is cascaded and merged, and then passes through two convolution layers conv4 and conv5 with a kernel size of 5*5, a step size of 1, and padding = 2 to obtain a resolution of the input Figure 1 Half, with 128 channels of feature output. Finally, conv6 is an output layer with only one convolution kernel of size 5*5, convolution with stride 1 and padding=2, and the final output has one channel and a depth prediction map with a resolution of 1 / 2 of the input.

[0075] In other embodiments, the network structures of the encoder, decoder, feature fusion module and optimization module may not be limited to those described above. For example, the encoder uses more residual blocks to extract deeper features, or the optimization module only uses one convolutional layer to obtain depth information.

[0076] Step 4: Set the network depth loss function and depth gradient loss function, and use the weighted sum of the depth loss function and the depth gradient loss function as the overall loss function.

[0077] When estimating depth, objects in the image tend to have a logarithmic log distribution. The reason is that objects near are large and objects far are small. Objects in the image are more concentrated in the closer (smaller depth) area, and objects in the distant area are smaller in the image. Therefore, for a unit of depth error, the penalty of the depth loss function at close distances should be greater than that at long distances. Therefore, the depth loss function is set to:

[0078] Among them, L D is the depth loss function, d i is the depth prediction value of the i-th pixel, is the true depth value of the ith pixel, n is the number of pixels, and α is a normal number to ensure that it is a positive number after logarithm.

[0079] However, only a single depth loss function will appear where the depth changes dramatically (such as the edge of the handpiece), and the predicted depth changes slowly, resulting in unclear edges. It is not sensitive to the edges of objects with sudden changes in depth. Therefore, a depth gradient loss function that is sensitive to the image depth as the (x, y) coordinates shift (up, down, left, and right) is added to solve this problem. The formula for the depth gradient loss function is:

[0080]

[0081] Among them, L g is the depth gradient loss function, e i is the L1 norm of the true depth value and predicted value of the i-th pixel, is the e at the i-th pixeli The gradient in the x direction, is the e at the i-th pixel i The gradient in the y direction. This loss function is sensitive to the movement of the edge in the x and y directions, so we use the weighted sum of the two loss functions as the overall loss function to train the network: L = L D +aL g , a is the weight coefficient of the depth gradient loss function.

[0082] Step 5: End-to-end training network. First, use the NYU-depth training set (i.e., general data set) to train the network. For the initial parameters, the encoder loads the weights of the four blocks of ResNet-50 pre-trained with the ImageNet data set. The decoder, feature fusion module, and optimization module parameters are randomly initialized according to the case of no pre-training. Then, fine-tune the model parameters using the created surgical scene simulation training set.

[0083] First, the network is trained using the NYU-depth training set, which mainly consists of a variety of indoor scenes (such as bedrooms, bathrooms, basements, etc.). For the initial parameters, the encoder loads the weights of the 4 blocks of ResNet-50 pre-trained by the ImageNet dataset, and the decoder, feature fusion module, and optimization module parameters are randomly initialized according to the case without pre-training. Experiments are conducted under the pytorch framework, and the learning rate is set to 0.01%. For the input image pairs, a certain angle of random rotation is first performed to ensure that the overfitting problem can be better avoided during training. After that, the training model in the first step is used as the starting point, and the model parameters are fine-tuned using the created surgical scene simulation training set fine-tune (i.e., a dedicated dataset). Because the dataset is significantly smaller than the NYU-depth training set, the learning rate is set to 0.1%, the batch size is batch size = 2, and a total of 5 rounds of training are performed. Although there is basically no continuity problem between different pictures in this dataset, and the similarity is not high, due to the increase in the learning rate, in order to avoid overfitting, a certain probability of rotation processing is still performed before input.

[0084] In other embodiments, if the number of samples in the dedicated dataset is large enough, the depth prediction model may be trained using only the dedicated dataset.

[0085] Step 6: After obtaining the (x, y) coordinates of the center of the surgical instrument's bounding box in the pixel coordinate system through 2D visual tracking of the surgical instrument in the first step, the surgical scene depth map neural network trained in the fifth step is used to predict the overall depth information of each frame of the surgical scene, thereby adding one-dimensional depth information to the predicted two-dimensional coordinates, generating (x, y, d) coordinates in the pixel + depth coordinate system, connecting all coordinates, and completing 3D tracking of the surgical instrument.

[0086] First, we use the implemented SVM+CNN method to predict the 2D position of surgical instruments to complete the bounding box prediction of the surgical instrument to be tracked in each frame. Assume that the coordinates of the upper left corner of the bounding box are (a, b), the width of the bounding box is w pixels, and the height is h pixels. The midpoint of the box is used as the coordinate of the surgical instrument, and its value is (a+h / 2, b+h / 2), which is recorded as (x, y) as the 2D coordinate of the surgical instrument in the pixel coordinate system. The coordinate x and y units are both pixel, and then the predicted depth value at the coordinate is read (due to the limitation of GPU video memory, when predicting, the 1920*1080 size frame needs to be reduced to 480*270 before entering the network; the 720*576 size frame needs to be reduced to 360*288; the 352*288 size frame can be directly input). Because the resolution of the network output depth prediction map is half of the input image, the depth values ​​at the output prediction maps (x / 8, y / 8), (x / 4, y / 4), and (x / 2, y / 2) need to be taken respectively in the three cases to form the final 3D coordinates (x, y, d) in the pixel + depth coordinate system. The coordinate d unit is a depth unit, with a value of 0 to 255, and has an exponential correspondence with the physical space distance. Because the internal and external parameters of the endoscope camera in this study are unknown, endoscopes with known parameters can be used in subsequent studies to convert the predicted position results into the real-world coordinate system. Figures 4 to 11 The 2D and 3D motion trajectories of the center points of the bounding boxes of different types of surgical instruments are represented by a green box. The first frame position of the 2D image is represented by a green box, and the first three frames in the 3D trajectory are connected by green lines. The last frame position of the 2D image is represented by a red box, and the last three frames in the 3D trajectory are connected by red lines. The intermediate trajectories are connected by yellow lines.

Claims

1. A method for generating a 3D trajectory of a surgical instrument in surgical navigation, characterized in that: The method includes: S1, obtaining image information of the surgical scene during the operation; S2. Extracting depth information of the surgical instrument from the image information using a trained depth prediction model; The depth prediction model includes an encoder, a decoder, a feature fusion module and an optimization module; the encoder is used to extract the features of the image information; the decoder is used to improve the resolution of the features output by the encoder; the feature fusion module is used to fuse the features extracted by different layers of the encoder; the optimization module is used to obtain the depth information according to the output of the decoder and the output of the feature fusion module; S3. Obtain a 3D trajectory of the surgical instrument according to the depth information combined with the 2D coordinate information of the surgical instrument in a pixel coordinate system.

2. The method for generating 3D trajectories of surgical instruments in surgical navigation according to claim 1, characterized in that: The encoder includes a convolution layer and at least one residual block, and the convolution layer and the residual block of the encoder are connected in sequence; each time a convolution layer or a residual block of the encoder is passed, the resolution of the output feature map is continuously reduced; the feature fusion module performs fusion processing on the output features of each residual block; The decoder includes a convolution layer and an upward projection deconvolution block. The number of the upward projection deconvolution blocks of the decoder is the same as the number of the residual blocks of the encoder. The upward projection deconvolution blocks of the decoder and the convolution layer are connected in sequence. Each time a convolution layer or an upward projection deconvolution block of the decoder is passed, the resolution of the output feature map is continuously improved.

3. The method for generating 3D trajectory of surgical instruments in surgical navigation according to claim 2, characterized in that: The feature fusion module includes a convolution layer and an upward projection deconvolution block connected one-to-one with the residual block of the encoder; The feature maps output by the upward projection deconvolution blocks of the feature fusion module have the same size; the outputs of each upward projection deconvolution block are cascaded and input into the convolution layer.

4. The method for generating 3D trajectory of surgical instruments in surgical navigation according to claim 1, characterized in that: The optimization module first cascades the output of the decoder and the output of the feature fusion module, and then obtains the depth information after processing through at least one convolutional layer.

5. The method for generating 3D trajectory of surgical instruments in surgical navigation according to claim 1, characterized in that: The training process of the depth prediction model includes: firstly, training the depth prediction model using a general data set, and then fine-tuning the depth prediction model trained with the general data set using a dedicated data set; The general data set refers to a data set in which the data is not limited to the surgical scene; the dedicated data set refers to a data set in which the data only includes the surgical scene.

6. The method for generating a 3D trajectory of a surgical instrument in surgical navigation according to any one of claims 1 to 5, characterized in that: The loss function L used to train the depth prediction model is: <h2 style=";text-align:left;direction:ltr">L=L<h2 style=";text-align:left;direction:ltr"> D <h2 style=";text-align:left;direction:ltr"> +aL<h2 style=";text-align:left;direction:ltr"> g Among them, L D is the depth loss function, d i is the depth prediction value of the i-th pixel, is the true depth value of the i-th pixel, α is a positive constant; L g is the depth gradient loss function, e i is the L1 norm of the true depth value and predicted value of the i-th pixel, is the e at the i-th pixel i The gradient in the x direction, is the e at the i-th pixel i The gradient in the y direction; a is the weight coefficient of the depth gradient loss function, and n is the number of pixels of the input sample.

7. The method for generating 3D trajectory of surgical instruments in surgical navigation according to claim 1, characterized in that: The 2D coordinate information is obtained in the following manner: extract N squares around the position coordinates of the surgical instrument at the current moment of the image information as candidate samples, N≥2; input the candidate samples into a trained binary classifier to obtain the score of each candidate sample, and use the position of the sample with the highest score as the predicted position of the surgical instrument at the next moment, and use the center point of the predicted position as the 2D coordinate of the surgical instrument.

8. The method for generating 3D trajectory of surgical instruments in surgical navigation according to claim 7, characterized in that: The binary classifier includes at least two different binary classifiers, and the score of the candidate sample is obtained according to the following method: the normalized scores of the candidate sample output by each binary classifier are weightedly added to obtain the score of the candidate sample; the normalized score is obtained by normalizing the scores of each candidate sample output by the binary classifier.

9. The method for generating 3D trajectory of surgical instruments in surgical navigation according to claim 8, characterized in that: The binary classifier includes an SVM binary classifier and a CNN binary classifier; the last layer of the CNN binary classifier is a fully connected layer that uses a Logistic classification function to implement binary classification.

10. The method for generating 3D trajectory of surgical instruments in surgical navigation according to claim 9, characterized in that: Extract positive samples and negative samples from the image information at any time when the operation begins as training samples for the SVM binary classifier; The trained SVM binary classifier was used in the entire subsequent surgical process; Positive samples and negative samples are extracted from the image information at any time during the operation as training samples of the CNN binary classifier. The network parameters of the last layer of the CNN binary classifier are adjusted through the training samples of the CNN binary classifier, and the remaining network parameters are kept unchanged.

Images