Method and system for tracking endoscopic surgical instrument and generating motion trail of endoscopic surgical instrument

Abstract

The invention relates to a method and system for tracking an endoscopic surgical instrument and generating a motion trail of the endoscopic surgical instrument, and belongs to the technical field of image processing. The method comprises the following steps: taking a target position of a surgical instrument in a previous frame video as a center, and sampling by adopting a set distribution form to obtain a plurality of candidate samples of a current frame; respectively inputting the candidate samples into a pre-trained SVM (Support Vector Machine) detection model based on manual features and a pre-trained CNN (Convolutional Neural Network) target detection model, and respectively obtaining positive sample scores after weighted summation of scores of two models of each candidate sample; and taking the candidate sample with the maximum positive sample score as the target position of the surgical instrument of the current frame, and connecting the tracked target positions of each frame to generate the motion trail of the surgical instrument. On the basis of using the trained SVM model classification prediction model, depth features are extracted in combination with neural network features, a background and a target object are effectively distinguished, and the problems that an endoscope image is complex in structure, fuzzy in boundary and serious in shielding condition are solved.

Description

Technical Field

[0001] The present invention relates to a method and system for tracking an endoscopic surgical instrument and generating its motion trajectory, belonging to the technical field of image processing; in particular, a method for tracking an endoscopic surgical instrument and generating its motion trajectory based on the fusion of manual features and depth features. Background Art

[0002] The development of minimally invasive surgery has improved the surgical process. Compared with traditional surgical procedures, minimally invasive surgery has the advantages of small wounds, quick recovery, and less pain. In traditional minimally invasive surgery, one doctor is required to operate an endoscope such as a laparoscope or thoracoscope for detection, while another doctor observes the endoscopic image and uses specialized surgical instruments to perform operations such as exploration, electrocoagulation, hemostasis, tissue separation, and suturing of the diseased tissue. However, factors such as the operation of straight-rod instruments, the jitter of the handheld lens, and the ergonomic design have increased the operating accuracy, learning curve, and learning time of minimally invasive surgery, and minor errors in the operation will have an adverse effect on the operation and even endanger the patient's life.

[0003] With the development of information technology, some computer-assisted technologies can be used to enhance the surgeon's ability to control surgical instruments during minimally invasive surgery. Among the many methods, the most basic and critical one is to understand the status of surgical instruments in real time through target tracking. Through target tracking, the position, direction and other parameters of surgical instruments can be obtained more accurately, and the movement trajectory of surgical instruments and the standardization of operations can be evaluated. These information indicating the status of surgical instruments can be placed on the top of the instrument close to the surgeon's visual center, thereby enhancing the safety of using these tools and greatly reducing the difficulty and risk of surgical operations.

[0004] At present, the detection methods of surgical instruments include electromagnetic method, optical tracking, robot kinematics, ultrasound technology and fluoroscopy. Most of these methods require additional instruments or experiments, which are costly and not very accurate. The image-based tracking method does not need to modify the surgical operating table and can directly provide position and motion information in the coordinate frame of the endoscopic image. At present, most of the image-based tracking methods simply extract manual features such as color and brightness for distinction, or use traditional target tracking algorithms to separate surgical instruments from the background. Although the color, texture and other characteristics of tissues and surgical instruments are obviously different, the lighting used in medical environments combined with the smooth tissue surface will cause large specular reflections and destroy the color and texture characteristics of metal instruments. In addition, the deformation and bleeding of tissues during surgery will block the instruments, which leads to the lack of pertinence of surgical instrument tracking using simple traditional features or unimproved models, and its effect will also be significantly reduced. Summary of the invention

[0005] The object of the present invention is to provide a method and system for tracking an endoscopic surgical instrument and generating its motion trajectory, so as to solve the problem that the existing endoscopic surgical instrument tracking effect is poor.

[0006] To achieve the above object, the solution of the present invention includes:

[0007] A technical solution of a method for tracking an endoscopic surgical instrument and generating its motion trajectory of the present invention comprises the following steps:

[0008] 1) Taking the target position of the surgical instrument in the previous frame of video as the center, a set distribution form is used to sample and obtain several candidate samples of the current frame;

[0009] 2) The candidate samples are respectively input into the pre-trained SVM detection model based on manual features and the pre-trained CNN target detection model, and the positive sample score of each candidate sample is obtained by weighted summation of the scores of the two models; the score is the probability of judging that the corresponding sample is a positive sample;

[0010] 3) The candidate sample with the largest positive sample score is used as the target position of the surgical instrument in the current frame, and the target positions tracked in each frame are connected to generate the motion trajectory of the surgical instrument.

[0011] The present invention uses a trained SVM model classification prediction model and combines neural network features to extract deep features, effectively distinguishing the background and the target object, and solving the problems of complex endoscopic image structure, blurred boundaries, and severe occlusion.

[0012] Furthermore, in step 1), the set distribution form is a multi-dimensional Gaussian distribution.

[0013] Furthermore, in step 2), a color histogram and a histogram of oriented gradients are used as the manual features; a number of positive samples containing surgical instruments and negative samples not containing surgical instruments are extracted from the first frame image, and the color histogram features and the histogram features of oriented gradients of all samples are extracted and merged for pre-training of the SVM detection model.

[0014] Furthermore, the CNN target detection model includes three convolutional layers, two fully connected layers and a binary classification layer in sequence; there is a maximum pooling layer after the first convolutional layer and the second convolutional layer respectively; the two output units of the binary classification layer correspond to the surgical instrument target and the background respectively.

[0015] Furthermore, the CNN target detection model has several binary classification layers during pre-training, corresponding to the same number of endoscopic surgery videos, and only the binary classification layer of the corresponding video is used in each training, and the layer before the binary classification layer is shared when using each video for training; after pre-training, the CNN target detection model replaces several binary classification layers with a single binary classification layer of a new test sequence.

[0016] Furthermore, the CNN target detection model also adopts two online update strategies for the network: long-term update and short-term update: long-term update or short-term update is selected according to the speed of change of the appearance of the surgical instrument as the target; the long-term update uses negative samples from a long period of time in history to update the network, and the short-term update uses negative samples collected recently to update the network.

[0017] In order to take into account both the robustness and adaptability requirements of target tracking, the present invention designs a model update strategy. Regularly use negative samples collected over a long period of time for updating, and when tracking failure is detected, use positive samples collected in a short period of time for updating;

[0018] Furthermore, long-term update or short-term update is selected according to the positive sample score reflecting the speed of change of the surgical instrument appearance. If the positive sample score is lower than a set value, a short-term update is performed, otherwise a long-term update is performed at a fixed time interval.

[0019] Furthermore, in the online update strategy, the sample with the lowest score among the positive samples is selected as a negative sample for network update.

[0020] Furthermore, a bounding box regression model is added to fine-tune the target box of the target position predicted in step 3).

[0021] A technical solution of a system for tracking an endoscopic surgical instrument and generating its motion trajectory of the present invention includes a processor, wherein the processor executes a computer program to implement the steps of the method for tracking an endoscopic surgical instrument and generating its motion trajectory.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention constructs surgical instrument detection models based on SVM and CNN respectively. The former extracts color histogram features and HOG features as the input of SVM classifier based on manual features. The latter constructs a multi-domain CNN model based on deep features and extracts domain-independent deep features. This method solves the problem of insufficient training data while completing surgical instrument detection.

[0024] 2. The present invention designs an endoscopic surgical instrument tracking framework and optimization method. In the instrument tracking process, in order to improve its tracking accuracy, two model update strategies, long-term update and short-term update, are designed, and two optimization methods, hard example mining and bounding box regression, are added.

[0025] 3. The present invention has experimentally shown that the average success rate and average accuracy of surgical instrument tracking of the CNN model combined with the SVM model after adding bounding box regression reached 0.740 and 0.731, respectively, with the best performance. In five challenging scenarios, namely occlusion, illumination change, rapid motion, size change, and deformation, the model also exhibited strong stability and robustness. The comparison results with the seven existing methods show that the model has a more obvious advantage in surgical instrument tracking and can generate more accurate motion trajectories. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flow chart of a method for tracking an endoscopic surgical instrument and generating its motion trajectory in a method embodiment;

[0027] Figure 2 It is a diagram of the HOG feature extraction process in the method embodiment;

[0028] Figure 3 It is a structural diagram of a CNN model in a method embodiment;

[0029] FIG4( a ) is a schematic diagram of negative sample effects generated after the endoscopic surgical instrument tracking framework is optimized through hard example mining in a method embodiment;

[0030] FIG4( b ) is a schematic diagram of the predicted position effect of the endoscopic surgical instrument tracking framework before and after adding the bounding box regression in the method embodiment;

[0031] Figure 5 Results of surgical instrument tracking in different scenarios;

[0032] Figure 6 It is the motion trajectory diagram of surgical instruments;

[0033] Figure 7(a) shows the accuracy calculation curve of all data in the model evaluation test results (CNN+BB+SVM, different surgical instruments);

[0034] Figure 7(b) is the success rate calculation curve of all data in the model evaluation test results (CNN+BB+SVM, different surgical instruments);

[0035] Figure 8(a) shows the classification accuracy measurement curve in the model evaluation test results (CNN+BB+SVM, different challenging scenarios);

[0036] Figure 8(b) shows the classification success rate measurement curve in the model evaluation test results (CNN+BB+SVM, different challenging scenarios);

[0037] Figure 9(a) shows the average accuracy measurement curve in the model evaluation test results (comparison of four models);

[0038] Figure 9(b) is the success rate calculation curve in the model evaluation test results (comparison of four models);

[0039] Figure 10(a) shows the occlusion scene accuracy measurement curve in the model evaluation test results (comparison of four models);

[0040] Figure 10(b) is the occlusion scene success rate calculation curve in the model evaluation test results (comparison of four models);

[0041] Figure 11(a) shows the accuracy calculation curve of the scene with illumination changes in the model evaluation test results (comparison of four models);

[0042] Figure 11(b) is the success rate calculation curve of the illumination change scene in the model evaluation test results (comparison of four models);

[0043] Figure 12(a) shows the fast motion scene accuracy measurement curve in the model evaluation test results (comparison of four models);

[0044] Figure 12(b) is the success rate calculation curve of the fast motion scene in the model evaluation test results (comparison of four models);

[0045] Figure 13(a) shows the accuracy calculation curve of the size change scene in the model evaluation test results (comparison of four models);

[0046] Figure 13(b) is the success rate calculation curve of the size change scenario in the model evaluation test results (comparison of four models);

[0047] Figure 14(a) shows the deformation scene accuracy measurement curve in the model evaluation test results (comparison of four models);

[0048] Figure 14(b) is the deformation scene success rate measurement curve in the model evaluation test results (comparison of four models). DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail in a clear and complete manner in conjunction with the accompanying drawings and embodiments.

[0050] In order to overcome the problem that the existing image-based tracking method of endoscopic surgical instruments is easy to lose the target and has poor effect due to the severe mirror reflection, blurred boundaries and severe occlusion in endoscopic surgical images, the present invention provides a method for tracking endoscopic surgical instruments and generating their motion trajectories based on the fusion of manual features and deep features.

[0051] The idea of ​​the present invention is to combine the deep features extracted by neural network with the support vector machine classification detection based on manual features to effectively distinguish the background and the target object, and solve the problems of complex structure, blurred boundaries and serious occlusion of endoscopic images. A SVM (Support Vector Machine) detection model based on manual features and a target detection model based on CNN (Convolutional Neural Network) are constructed respectively, and the image feature map of the first frame is used to optimize the network performance by back propagation training. The model uses the feature map output by the feature extraction model as input, gives a score for each candidate area, and the area with a high score is used as the prediction area.

[0052] In order to take into account both the robustness and adaptability requirements of target tracking, a model update strategy is designed. The positive samples collected over a long period of time are used for regular updates, and when tracking failure is detected, the positive samples collected over a short period of time are used for updates.

[0053] In order to predict the state of the target in each frame, N candidate regions are extracted around the object in the previous frame, and then their scores are obtained according to the target detection model. The maximum positive sample score is then found and used as the optimal target state. Finally, the surgical instrument positions tracked in each frame are connected to generate the motion trajectory.

[0054] The principle of the present invention is:

[0055] (1) The existing manual feature extraction of endoscopic images cannot solve the problems of complex structure, blurred boundaries, and severe occlusion. The present invention proposes a method of combining support vector machine classification with neural network to extract deep features to effectively distinguish between background and target objects.

[0056] (2) The present invention constructs surgical instrument detection models based on SVM and CNN, respectively. According to the screened features, the design captures category information and distinguishes objects similar to the background.

[0057] (3) The present invention designs an endoscopic surgical instrument tracking framework, extracts candidate samples around the surgical instrument in the previous frame, and uses the target detection model to predict the state of the target in each frame.

[0058] (4) In order to improve its tracking accuracy, two model update strategies, long-term update and short-term update, are designed, and two optimization methods, hard example mining and bounding box regression, are added.

[0059] The present invention effectively improves the detection accuracy and robustness for some challenging scenarios in endoscopic surgery (such as illumination changes, background occlusion, deformation, etc.), and provides a method and system for tracking endoscopic surgical instruments and generating their motion trajectories based on the fusion of manual features and deep features. This method meets the requirements of different scenarios and has a wide range of applications.

[0060] Method Example:

[0061] The method flow of the present embodiment is as follows: Figure 1 As shown, first, it is necessary to extract features and pre-train the SVM and CNN classification models to build a surgical instrument detection model based on SVM and CNN. The implementation steps are as follows:

[0062] 1. Feature extraction.

[0063] In an endoscopic surgery video, surgical instruments are mostly silvery white with clear edges, while the background is basically dark red tissue and blood with no obvious boundaries. Therefore, we choose to use color and gradient as manual features, represented by color histogram and histogram of oriented gradients (HOG) respectively. The two features are combined and input into the SVM model for target detection.

[0064] (1) Color histogram.

[0065] In the process of calculating the color histogram, the color model is first divided into small intervals with equal components or dimensions, and then the entire image is traversed to count the number of pixels in each color interval to obtain the color distribution ratio. Let H(k) be a component of the color histogram, then Where k represents a color interval, n k It is the statistic of the number of pixels corresponding to the color interval, N is the total number of pixels in the image, and M represents the number of color intervals.

[0066] In the process of constructing the color features of surgical images, we used RGB and HSV models to model the colors of the images respectively, then set k to 32, calculated the color histograms of the three channels respectively, and finally merged them into a one-dimensional vector as part of the manual features.

[0067] (2)HOG features.

[0068] When calculating the gradient of an image, a gradient extraction operator is needed. Here we use the first-order centrosymmetric operator of [-1, 0, 1] to calculate the gradient size in two directions using the following formula:

[0069] G h (x,y)=f(x+1,y)-f(x-1,y)

[0070] G v (x,y)=f(x,y+1)-f(x,y-1)

[0071] Among them G h (x,y) and G v (x, y) represents the gradient magnitude of a pixel in the horizontal and vertical directions respectively. From this, we can get the gradient direction θ(x, y) and the total gradient magnitude M(x, y):

[0072]

[0073] When calculating the gradient direction, if the obtained θ(x,y) is greater than 180°, it can be normalized to 0°~180°, that is:

[0074] θ′(x,y)=θ(x,y)-π.

[0075] like Figure 2 As shown in the figure, when extracting HOG features from surgical images, we set each detection window to contain 7×7 blocks with a block size of 16×16 pixels. There are 2×2 cells in each block, and the size of each cell is 8×8 pixels. For each cell, the direction (0°~180°) is evenly divided into 9 groups (9bins), that is, every 20° is a group. All pixels in each cell can be divided into 9 groups according to the angle, and a 9-dimensional directional gradient histogram is extracted. After calculating the directional gradient histogram of each cell, the histograms are connected and normalized. Here, the L1 norm normalization is used, and then the square root (L1-sqrt) normalization method is used:

[0076]

[0077] v is introduced to represent a vector that has not been normalized and contains all the histogram information in a given cell, and ε represents a small constant.

[0078] As other implementations, L1 norm normalization (L1), L2 norm normalization (L2), or L2 norm normalization followed by limiting the maximum value and performing renormalization using the L2 norm (L2-Hys) may also be used.

[0079] After normalization, the histogram of each block is obtained, and then the complete directional gradient histogram is obtained by concatenation, thus completing the HOG feature extraction of the surgical image and obtaining the HOG feature map. The dimension of each window HOG feature is 49×4×9=1764.

[0080] 2. A surgical instrument detection model based on SVM and CNN was constructed.

[0081] (1) SVM training method.

[0082] In order to train the SVM model, 500 positive samples and 5000 negative samples are extracted from the first frame of the picture, and the size is normalized to 107×107. The positive samples include surgical instruments, while the negative samples do not. The color histogram features and HOG features of the positive and negative samples are extracted and merged respectively to obtain the features of each sample, whose dimension is 5379. Then a linear kernel function is selected, a quadratic programming problem is constructed, and the optimal classification hyperplane and classification decision function are obtained. A model with an average accuracy of 98.6% can be obtained. The calculation process is as follows:

[0083] Let T = {(x1,y1),(x2,y2),...,(x n ,y n )} is a data set in the feature space, where x i represents the feature vector, y i ∈{1,-1} represents the class label. If the two classes of samples are linearly separable, then a hyperplane can be found to ensure that the data set is correctly divided and the geometric distance is maximized. The hyperplane equation is defined as: w·x i +b=0.

[0084] Since no sample points can appear in the interval area of ​​the hyperplane, (x i ,y i ) need to meet:

[0085] y i (w·x i +b)≥1,i=1,...,N.

[0086] Define the hyperplane to the sample point (x i ,y i ) is:

[0087] Then we can get the geometric distance:

[0088]

[0089] In order to find the optimal classification hyperplane, we need to maximize That is, minimize ||w||. At this time, the problem is transformed into a quadratic programming problem:

[0090]

[0091] Among them, α is the Lagrangian factor and C is the penalty parameter. Solving this quadratic programming problem yields the optimal solution:

[0092]

[0093] Next, calculate:

[0094] In this way, the optimal classification hyperplane and classification decision function can be obtained:

[0095] w * ·x+b * =0

[0096] f(x)=sign(w * ·x+b * ).

[0097] Then use the Gaussian kernel function for high-dimensional mapping. When predicting, you need to use the kernel function to multiply the 5000 support vectors obtained by training with the features of the data set. The computational complexity of this method is thousands of times greater than that of using a linear kernel function, and the training speed and prediction speed drop significantly. However, its classification accuracy has only increased by less than 1 percentage point. Therefore, this embodiment still uses a linear kernel function for dimensional mapping. The calculation formula is as follows:

[0098] When doing the inner product of a vector, using a kernel function instead of the normal inner product operation can implicitly transform a low-dimensional vector into a high-dimensional vector. Commonly used kernel functions are as follows:

[0099] Polynomial kernel function: K(x,y)=(γx T +r) d ,γ>0;

[0100] Gaussian kernel function: K(x,y)=exp(-γ||xy|| 2 ),γ>0;

[0101] Sigmoid kernel function: K(x,y) = tanh(γx T y+r) d ,γ>0;

[0102] Where d, r, and γ are the parameters of the kernel function. After replacing the inner product operation of with the kernel function, we get:

[0103] Put the result back into the formula Japanese style Get the optimal classification hyperplane and classification decision function after using the kernel function.

[0104] During the tracking process, the SVM model trained by the first frame image is used for prediction to obtain the probability that each candidate sample in each subsequent frame is a positive sample.

[0105] (2) CNN model structure.

[0106] In this embodiment, it is expected to build a multi-domain CNN to distinguish surgical instruments and backgrounds in surgical videos. Although the target and background are different in each type of video, they still have some common properties, such as illumination changes, motion blur, scale changes, etc. In order to extract common features, we separate domain-independent information from specific videos by combining a multi-domain learning framework. The model is pre-trained on ImageNet, which can make up for the problem of insufficient training data and ensure that the trained network can well extract the features of surgical videos.

[0107] The model includes three convolutional layers, two fully connected layers and one binary classification layer. The convolutional layers (conv1-conv3) come from the VGG-M network, and the sizes of the convolution kernels are 51×51, 11×11, and 3×3, respectively. There is a maximum pooling layer with a stride of 2 after the first and second groups of convolutions, which enables the model to extract features from a wider range and reduces the input size of the next layer, thereby reducing the amount of calculation and the number of parameters. In order to obtain the feature map in the convolutional layer conv3, this paper sets the input of the model to be a 107×107 picture.

[0108] The fully connected layers (fc4 and fc5) each have 512 output units, which convert the two-dimensional feature map output by the convolutional layer into a one-dimensional vector of size 512. At the same time, a Dropout layer is added to perform random sampling with a certain probability to solve the problem of overfitting the training data.

[0109] The binary classification layer (b6) is essentially a fully connected layer with Softmax cross entropy loss. It has only two output units and is responsible for distinguishing the target and background in each domain. There are k binary classification layers in pre-training, corresponding to k videos. Each training only uses the binary classification layer corresponding to the video, and the previous layers are shared. The network structure of the CNN model is as follows: Figure 3 shown.

[0110] (3) CNN training method.

[0111] The network is pre-trained using stochastic gradient descent (SGD). For K branches, the number of iterations is set to 100K, the learning rate of the convolutional layer is 0.0001, and the learning rate of the fully connected layer and the binary classification layer is 0.001. 50 positive samples and 200 negative samples are extracted from each frame as the training set, where the overlap rate between the positive samples and the true target box is greater than 0.7, and the overlap rate of the negative samples is 0.5. Each iteration uses a video to update the parameters until the network converges or reaches a predefined number of iterations. Through this training process, domain-independent information can be modeled in the five shared layers, from which a universal feature representation can be obtained.

[0112] At the beginning of the tracking task, when the model is initialized, the number of iterations of the fully connected layer and the binary classification layer is set to 30, the learning rate of the fully connected layer is 0.0001, and the learning rate of the binary classification layer is 0.001. The training set contains 500 positive samples and 5000 negative samples in the first frame image.

[0113] 3. Endoscopic surgical instrument tracking framework and optimization method.

[0114] In order to complete the target tracking, a motion model based on multi-dimensional Gaussian distribution is used to generate candidate samples. After completing the pre-training of the SVM model and the CNN model, the k binary classification layers in the CNN are replaced with a single binary classification layer of a new test sequence. Next, an appropriate tracking strategy is adopted, and the binary classification layer and the fully connected layer in the shared layer are retrained during the tracking process, and an optimization method is added to improve the accuracy of the model.

[0115] (1) Motion model based on multi-dimensional Gaussian distribution.

[0116] In order to generate candidate samples in each frame of video, this paper takes the target position of the previous frame of video as the center and samples N = 256 candidate samples in the form of multi-dimensional Gaussian distribution. That is, the candidate samples follow the Gaussian distribution:

[0117]

[0118] Where D is The dimension, mean is the target box in the previous frame Covariance∑ t for where r is the average of the width and height of the object box in the previous frame.

[0119] After unifying the size of candidate samples to 107×107, they are used as network inputs for calculation.

[0120] (2) CNN update strategy.

[0121] Design a model update strategy for target tracking, using two update methods: long-term update and short-term update. During the tracking process, the structure of the model remains unchanged, and the choice of the two update strategies depends on the speed of target appearance change. In both cases, negative samples collected in a short period of time are used, and old samples that are irrelevant to the current frame are eliminated. Long-term updates correspond to 100 historical samples, and the network is updated at a fixed time interval (set to update once every 8 frames in the program); short-term updates correspond to 20 samples, and updates are performed when the target score is less than 0.5.

[0122] For the online update of the model, the number of iterations of the fully connected layer is set to 10, and the learning rate is three times the learning rate when the model is initialized for fast fitting. 50 positive samples and 200 negative samples are extracted from the current frame image as the training set, where the overlap rate between the positive sample and the predicted target box is greater than 0.7, and the overlap rate of the negative sample is less than 0.3. Each batch contains 32 positive samples and 96 negative samples. The momentum and weight decay coefficient of the loss function are set to 0.9 and 0.0005 respectively.

[0123] (3)Tracking strategy.

[0124] Use the motion model to generate N = 256 candidate samples x around the target position in the previous frame 1 ,...,x N First, use the CNN model to score, and normalize the score to get the probability f(x) of each sample being a positive sample. i ). Then use the trained SVM model to classify and predict the candidate samples, and get the probability g(x) that each sample is a positive sample. i ). These two scores are weighted and the optimal target state x is determined by finding the candidate sample with the highest score. * As a result of the tracking,

[0125]

[0126] Among them, a and b are the weights of the scores in the CNN model and the SVM model respectively, a+b=1, here a=0.8, b=0.2.

[0127] (4) Trajectory generation method.

[0128] We draw the trajectory of the surgical instrument based on the prediction results. Connect the midpoint of the predicted target position in each frame, that is, As the motion trajectory; (x, y) is the coordinate of the lower left corner of the predicted target position, w, h are the width and height of the predicted target position respectively. For data with a small motion range and complex trajectory, the midpoint of the target position is connected every 10 frames as the trajectory.

[0129] (5)Optimization method.

[0130] For endoscopic surgical instrument tracking, it is necessary to ensure both good accuracy and fast speed. Computer hardware conditions, the use of parallel computing, and image resolution all have a significant impact on the performance of the tracker. It is necessary to improve the accuracy of the detector as much as possible while minimizing the speed drop while ensuring that these conditions remain unchanged. This embodiment adds two optimization methods: hard case mining and bounding box regression, in which hard case mining screens training samples to improve accuracy when updating the CNN model, and bounding box regression makes further adjustments to the initial predicted results.

[0131] Difficult Example Mining:

[0132] In each iteration, a set of training samples includes positive samples M + and negative samples Select the negative sample M of the previous iteration - The sample with the lowest score is Retrain the CNN network. As the network is updated, it becomes increasingly difficult to distinguish between positive and negative samples, and the accuracy of the model will also improve. Figure 4(a) shows the changes in negative samples after adding hard example mining. The red and blue boxes represent the positive and negative samples in each small block, respectively. It can be seen that as the number of iterations increases, negative samples become increasingly difficult to distinguish from positive samples.

[0133] Bounding Box Regression Model:

[0134] Add the bounding box regression model, an optimization method in RCNN. Its purpose is to fine-tune the initially predicted target box so that the fine-tuned bounding box is closer to the true bounding box. Given the first frame of the test sequence, train a simple linear regression model to use the conv3 features of samples near the target position to predict the precise target position.

[0135] To train the model, input N sets of data {(P i ,G i )} i＝1,....,N , where P = (P x ,P y ,P w ,P h ) represents the pixel coordinates of the upper left corner of a predicted target box and its width and height. Similarly, G = (G x ,G y ,G w ,G h ) represents the true target box. The goal of this paper is to train a linear model to map the predicted target box P to the true target box G.

[0136] Define two size-invariant position translation functions d x (P), d y (P), and two position-invariant size-change functions d w (P), d h (P). Through these four functions, an input target box P can be mapped to the real target box G:

[0137]

[0138] Each function d * (P) Perform a linear transformation on the features extracted by the convolutional layer conv3 in P, that is Where φ3(P) represents the features extracted by conv3, w * Represents the parameter vector of the model. We learn w through ridge regression * :

[0139]

[0140] Among them, the regression target t of the training data (P, G) * It can be defined as:

[0141] t x =(G x -P x ) / P w

[0142] t y =(G y -P y ) / P y

[0143] t w =log(G w / P w )

[0144] t h =log(G h / P h )

[0145] It can be seen that this is a standard regularized least squares problem that can be solved. Since the training process is time-consuming, this embodiment only trains it in the first frame. In subsequent frames, the linear model is used to adjust the preliminary predicted target position. As shown in Figure 4(b), the green box represents the true target position, and the blue box and red box represent the predicted target box position before and after adding the bounding box regression model.

[0146] In order to prove the effect of the scheme of this embodiment, two model evaluation indicators are calculated through experiments. The first is the success rate, that is, the average degree of overlap between the predicted target and the labeled target in all frames; the second is the accuracy, that is, the percentage of frames in which the Euclidean distance between the predicted target and the labeled target is within a given pixel threshold distance in the total number of video frames. Here, the threshold is set to 20 pixels.

[0147] Experiments were conducted using four models, namely CNN, CNN+BB (bounding box regression), CNN+SVM, and the final model CNN+SVM+BB, which represents the best implementation, and it was found that adding an SVM model or adding a bounding box regression optimization method performed better than a single CNN model on most data sets. Figure 5 The figure shows the tracking frame results in 4 frames of endoscopic surgery videos in 8 different scenes. The red frame is the tracking result under the best implementation mode of the present invention, and the green frame is the tracking result of other methods. Figure 6 The figure shows the motion trajectory of 8 surgical instruments. The yellow line shows the motion trajectory tracked by the best implementation of the present invention. The average success rate and accuracy of the CNN+SVM+BB model are 0.740 and 0.731 respectively, which is the best performance among the four models. The specific experimental and test results can be referred to: Figure 7(a) , 7(b) The accuracy calculation and success rate calculation curves of all data (CNN+BB+SVM) are shown respectively. Figure 8(a) , 8(b) The classification accuracy calculation and success rate calculation curves (CNN+BB+SVM) are shown respectively. Figure 9(a) , 9(b) The average accuracy calculation and success rate calculation curves shown respectively (comparison of four models) are as follows Figure 10(a) , 10(b) The occlusion accuracy calculation and success rate calculation curves shown respectively (comparison of four models), Figure 11(a) , 11(b) The accuracy and success rate of illumination change are shown in the following curves (comparison of four models). Figure 12(a) , 12(b) The accuracy and success rate of rapid motion are shown in the figure (comparison of four models). Figure 13(a) , 13(b) The accuracy calculation and success rate calculation curves of the size change (the enlargement or reduction of the target caused by the scaling of the image) are shown respectively (comparison of four models), such as Figure 14(a) , 14(b) The accuracy calculation and success rate calculation curves of deformation (deformation of the soft tissue itself under the endoscope and deformation caused by the surgical instrument pulling the soft tissue) are shown respectively (comparison of four models).

[0148] Experimental development and testing environment: Hardware: 2.40GHz Intel Core i5-6200U processor, 8GB memory. Software: Install opencv (3.3.1), sklearn (0.22.1), Tensorflow (1.14.0) and pytorch (1.0.1) software packages under python 3.6.4. Use sklearn framework to build SVM model, and use Tensorflow framework to build CNN model.

[0149] The technical contents not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.

[0150] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

[0151] System Example:

[0152] A system for tracking an endoscopic surgical instrument and generating its motion trajectory in this embodiment includes a processor and a memory, wherein a computer program is stored in the memory. The processor executes the computer program to implement a method for tracking an endoscopic surgical instrument and generating its motion trajectory as described in the method embodiment. The method is described clearly enough in the method embodiment and will not be repeated here.

Claims

1. A method for tracking and generating a motion trajectory of an endoscopic surgical instrument, characterized in that: The steps include: 1) Taking the target position of the surgical instrument in the previous frame of video as the center, a set distribution form is used to sample and obtain several candidate samples of the current frame; 2) The candidate samples are respectively input into the pre-trained SVM detection model based on manual features and the pre-trained CNN target detection model, and the positive sample score of each candidate sample is obtained by weighted summation of the scores of the two models; the score is the probability of judging that the corresponding sample is a positive sample; 3) The candidate sample with the largest positive sample score is used as the target position of the surgical instrument in the current frame, and the target positions tracked in each frame are connected to generate the motion trajectory of the surgical instrument.

2. The method for tracking an endoscopic surgical instrument and generating its motion trajectory according to claim 1, characterized in that: In step 1), the set distribution form is a multi-dimensional Gaussian distribution.

3. The method for tracking an endoscopic surgical instrument and generating its motion trajectory according to claim 1, characterized in that: In step 2), a color histogram and a directional gradient histogram are used as the manual features; a number of positive samples containing surgical instruments and negative samples not containing surgical instruments are extracted from the first frame image, and the color histogram features and directional gradient histogram features of all samples are extracted and merged for pre-training of the SVM detection model.

4. The method for tracking an endoscopic surgical instrument and generating its motion trajectory according to claim 1, characterized in that: The CNN target detection model includes three convolutional layers, two fully connected layers and a binary classification layer in sequence; there is a maximum pooling layer after the first convolutional layer and the second convolutional layer respectively; the two output units of the binary classification layer correspond to the surgical instrument target and the background respectively.

5. The method for tracking an endoscopic surgical instrument and generating its motion trajectory according to claim 4, characterized in that: The CNN target detection model has several binary classification layers during pre-training, corresponding to the same number of endoscopic surgery videos. Each training only uses the binary classification layer of the corresponding video, and the layer before the binary classification layer is shared when using each video for training. After pre-training, the CNN target detection model replaces several binary classification layers with a single binary classification layer of a new test sequence.

6. The method for tracking an endoscopic surgical instrument and generating its motion trajectory according to claim 5, characterized in that: The CNN target detection model also adopts two online update strategies for the network: long-term update and short-term update: long-term update or short-term update is selected according to the speed of change of the appearance of the surgical instrument as the target; the long-term update uses negative samples from a long period of time in history to update the network, and the short-term update uses negative samples collected recently to update the network.

7. The method for tracking an endoscopic surgical instrument and generating its motion trajectory according to claim 6, characterized in that: Long-term update or short-term update is selected according to the positive sample score reflecting the speed of change of the surgical instrument appearance. If the positive sample score is lower than the set value, a short-term update is performed, otherwise a long-term update is performed at a fixed time interval.

8. The method for tracking an endoscopic surgical instrument and generating its motion trajectory according to claim 7, characterized in that: In the online update strategy, the sample with the lowest score among the positive samples is selected as the negative sample for network update.

9. The method for tracking an endoscopic surgical instrument and generating its motion trajectory according to claim 4, characterized in that: Add a bounding box regression model to fine-tune the target box of the target location predicted in step 3).

10. A system for tracking and generating a motion trajectory of an endoscopic surgical instrument, comprising a processor, characterized in that: The processor executes the computer program to implement the steps of the method for tracking an endoscopic surgical instrument and generating its motion trajectory as described in any one of claims 1 to 9.

Images