A method and system for automatic tracking of target regions in surgical videos
By updating the tracking template in real time and introducing the confidence parameter PSR, the stability and accuracy issues of target area tracking in surgical videos were resolved, achieving efficient and real-time tracking results in complex surgical environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2022-01-25
- Publication Date
- 2026-05-26
AI Technical Summary
Existing surgical video target region tracking methods struggle to achieve stable and accurate real-time tracking when faced with complex medical images and changing surgical environments. In particular, traditional methods suffer from high computational cost and poor robustness in surgical external endoscopy, and methods based on Siamese neural networks cannot adapt to changes in target region features and large-scale occlusion.
A real-time update tracking template method is adopted. The confidence of the template is supervised by using the UpdateNetRes512 network combined with the confidence parameter PSR. The Siamese network is used to extract features for online updates and robust control, ensuring that the tracking template is associated with the current image and preventing tracking drift.
Stable tracking of the target region in surgical videos was achieved, improving tracking accuracy and robustness. It can maintain good accuracy and real-time performance under feature changes and large-scale occlusion, thus improving the overall tracking performance.
Smart Images
Figure CN116543007B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision for medical surgical videos, and more specifically, to a method and system for automatic tracking of target regions in surgical videos. Background Technology
[0002] Existing target tracking methods are mainly divided into generative target tracking, discriminative target tracking, and the latest target tracking techniques based on Siamese neural networks. Generative target tracking, including methods like MeanShift, is a kernel density estimation method that moves each point to a local maximum of the density function to predict the target. This method has low computational cost and can achieve real-time results when the target region is known. However, it also has certain drawbacks, such as the inability to update the template in real time, the window scale not changing with the size of the target object during tracking, poor tracking performance at high target speeds, and insensitivity to color changes. Discriminative methods mainly use correlation filtering, which involves modeling the target region of interest in the current frame and finding the region most similar to the model in the next frame—the predicted location. However, correlation filtering is still a relatively simple online update method and does not perform well in tracking complex backgrounds or blurred boundaries. Target tracking methods based on Siamese neural networks have been proposed in recent years, offering advantages such as accurate tracking and strong adaptability. However, direct application to medical images can lead to problems such as tracking drift.
[0003] With the widespread use of surgical endoscopes, surgical microscopes, and surgical exoscopes, target tracking methods based on medical images have become a research hotspot in academia. Traditional surgical area tracking methods mostly rely on manual marking, such as injecting contrast agents into patients before surgery. These agents emit a fluorescent-like light in areas with abundant capillaries, like tumors, thus helping doctors pinpoint the location of the surgical target. However, these methods also have certain problems: firstly, they are costly and not universally applicable to current clinical surgeries; secondly, the injection of these agents may cause certain side effects for patients.
[0004] Due to the limitations of manually labeled tracking methods, the academic community has proposed target tracking methods based on computer vision. Currently, most tracking methods for medical images are designed for surgical endoscopes and microscopes, using template matching and edge detection. However, the accuracy of these methods does not meet the requirements of medical surgery. In 2020, Zhang et al. proposed a template matching-based target recognition algorithm for medical endoscope images. This method compared template matching and edge detection algorithms, finding that template matching results are easily affected by changes in lighting. Therefore, combining these two algorithms solved the problem of insufficient lighting during endoscope use. However, the loss rate increases significantly when dealing with target areas with changing features, and the resolution of endoscopes is much lower than that of external endoscopes. Therefore, this method cannot achieve real-time performance when improving image quality. In 2018, Erica et al. proposed a new method using CNNs to track cells in medical images. By treating the segmentation problem as a tracking problem, they achieved cell state tracking. However, this method is only applicable to cell tracking algorithms with clear boundaries and is not suitable for external endoscope problems. Currently, there are not many algorithms applied to medical image tracking, therefore, there is an urgent need to develop an algorithm applicable to medical surgical image tracking.
[0005] Compared to surgical endoscopes and surgical microscopes, surgical exoscopes have a much larger range of camera movement and variation, which increases the difficulty of tracking the target area in surgery.
[0006] Because the target area captured during surgery is often poorly defined and densely vascularized, the medical video images presented by the camera are extremely complex, and the features (texture, color, etc.) of the target area are difficult to extract and distinguish. Furthermore, the surgical target area is highly likely to change its features during surgery due to bleeding, burning, or cutting. This dynamic change in the surgical environment may cause changes in the characteristics of the target area itself, leading to inaccurate feature recognition and affecting the accuracy of surgical video area tracking. The target area can also be frequently obscured by surgical instruments or the surgeon's hands during surgical operations, posing a technical challenge to stable intraoperative tracking of the target area.
[0007] While classic template matching methods can achieve real-time tracking due to their low computational cost, they fail when the target region features change relative to the template, making stable tracking impossible. Correlation filtering algorithms, although able to update in real-time to ensure tracking accuracy when target features change, are susceptible to interference from similar objects or large-scale occlusion, causing tracking drift. Recent target tracking methods based on SiamBAN networks input user-labeled tracking template images and each frame of the search image into the template branch and search branch of the SiamBAN network, respectively, to obtain feature maps. These feature maps are then regressed to the target tracking region through classification and regression modules, yielding the final tracking result. SiamBAN-based tracking algorithms are computationally fast and can maintain stable tracking even under large-scale occlusion. However, existing SiamBAN methods use static tracking templates, which cannot adapt to the constantly changing features of medical images. Therefore, surgical video target region tracking technology needs to develop more reliable methods to ensure the stability and accuracy of real-time tracking during surgery. Summary of the Invention
[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide an automatic tracking method and system for target areas in surgical videos.
[0009] According to one aspect of the present invention, an automatic tracking method for a target region in a surgical video is provided, comprising:
[0010] Real-time updates to the tracking template;
[0011] The confidence level parameter is used to monitor the reliability of the tracking template.
[0012] Features are extracted from the image of the tracking template and the search image respectively;
[0013] Based on the current tracking template, the extracted features are processed to obtain the tracking result.
[0014] Preferably, the real-time update of the tracking template includes: using a residual network with two convolutional layers and one activation function to fuse the input first frame template, cumulative template and prediction template to obtain the current tracking template.
[0015] Preferably, the UpdateNetRes512 network is selected, and a general function is learned through the network for updating, as shown in the formula:
[0016]
[0017] in, Represents the original template. It is a cumulative template. It is the calculated current frame template, and the current template is updated by combining these three factors;
[0018] First, deep features are extracted from the initial frame using a convolutional network. ;
[0019] Then use the accumulated template obtained from all previous frames to extract features. And predict the position of the next frame using it. ;
[0020] Based on these three inputs, an online update network is formed, which then outputs a newly accumulated template after two convolutional layers and a ReLU function. For the first frame and Set them all to .
[0021] Preferably, the loss function of the UpdateNetRes512 network is set to the Euclidean distance between the updated template and the real template of the next frame:
[0022]
[0023] Preferably, the confidence level is expressed as the Peak-to-Sidelobe Ratio (PSR), which is defined as follows:
[0024]
[0025] in and These are the standard deviation and mean of the side lobes, respectively.
[0026] Preferably, the PSR threshold is set such that when the PSR is within the threshold, the network stops updating the tracking template; otherwise, the update continues.
[0027] Preferably, the feature extraction of the template graph and the search graph is implemented using a Siamese network;
[0028] The Siamese network includes:
[0029] Template branch, whose input is an image that traces the template, is represented as The output is ;
[0030] The search branch takes the search image as its input, represented as... The output is .
[0031] Preferably, the backbone network of the Siamese network is ResNet50, the template branch and the search branch share the same parameters, and additional parameters are added to the ResNet50. Convolution reduces the number of output channels to 256; the template branch center is selected. Regional characteristics.
[0032] Preferably, the processing of the extracted features includes:
[0033] The feature maps obtained from the template branch and the search branch are subjected to correlation convolution using a method similar to depthwise convolution to obtain classification feature maps and bias coordinate maps; the pixels of the image of the tracking template are classified to determine whether they belong to the foreground or the background, thus obtaining classification feature maps;
[0034] The distance from the bounding box to the sampling point in the classification feature map of the search image is predicted to obtain the deviation coordinate map;
[0035] The foreground points in the classification feature map are sampled, and multiple predicted boxes are obtained based on their distance from the bounding box. The best predicted box is obtained by applying translation penalty and scale penalty, thus obtaining the tracking result.
[0036] According to a second aspect of the present invention, an automatic tracking system for a target region in surgical videos is provided, comprising:
[0037] A template update tracking module, which updates the tracking module in real time;
[0038] A confidence level determination module, which determines the reliability of the tracking template update through a confidence level parameter;
[0039] The feature extraction module uses a Siamese network to process the input image and extract feature maps from the tracking template image and the search image.
[0040] The regression module includes a classification module and a regression module. It receives feature maps of the tracking template image and the search image, and outputs the classification results of the foreground and background and the regression results of the distance from the bounding box to the foreground pixel, respectively.
[0041] Preferably, the video region automatic tracking system is trained using the OTB100 dataset, which incorporates medical surgical video data.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] This invention introduces an online tracking template update mechanism based on the existing SiamBAN target tracking method, ensuring that the tracking template is always associated with the current image tracking area, and maintaining stable tracking even when the features of the target area change;
[0044] This invention introduces a PSR confidence parameter to correct erroneous updates during template updates in cases of large-scale occlusion, thereby preventing tracking drift.
[0045] The automatic target region tracking method for surgical videos provided in this invention can avoid interference from feature changes and large-scale occlusions during tracking, compared with current methods, and maintains good robustness and accuracy at all times. Attached Figure Description
[0046] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0047] Figure 1 This is a schematic diagram of the structure of an automatic target region tracking method for surgical videos according to an embodiment of the present invention;
[0048] Figure 2 This is a flowchart illustrating the online update of the network tracking template in an embodiment of the present invention;
[0049] Figure 3 A flowchart illustrating the construction of the surgical video region target tracking dataset in this embodiment of the invention. Detailed Implementation
[0050] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.
[0051] To address the shortcomings of existing technologies, this invention solves the problem of feature changes by continuously updating the current tracking template. Furthermore, for the full occlusion problem, a PSR confidence parameter is introduced to control the template update speed. Therefore, an embodiment is provided: an automatic target region tracking method for surgical videos, comprising:
[0052] S1, Real-time updates of tracking templates;
[0053] S2, The confidence level parameter is used to monitor the reliability of the tracking template;
[0054] S3, extract features from the image of the tracking template and the search image respectively;
[0055] S4. Based on the current tracking template, process the extracted features to obtain the tracking result.
[0056] In this embodiment, the search image refers to each frame of the medical video. The initial template image refers to the image of the target area to be tracked, which the user manually marks in the first frame.
[0057] Based on the above embodiments, a preferred embodiment is provided, the specific process of which is as follows: Figure 1 As shown.
[0058] The automatic tracking method for target regions in surgical videos in this embodiment includes:
[0059] S100, template update network tracks templates in real time;
[0060] S200, the update speed and reliability of the tracking template are controlled by the confidence level PSR;
[0061] S300 uses the Siamban network to extract features from the image of the tracking template and the search image respectively;
[0062] S400 processes the extracted features to obtain classification feature maps and search feature maps;
[0063] S500 applies translation and scale penalties to the classification feature map and the search feature map. That is, it performs linear interpolation on the translation distance and scale change of the current prediction frame and the previous frame to obtain the tracking result.
[0064] To enable real-time replacement of the tracking template, the present invention provides a preferred embodiment that executes S100, as follows: Figure 3 The diagram shown is a flowchart of the online update network tracing template in this embodiment. Specifically, the template update part uses the UpdateNetRes512 network to update the template. A general function is learned through the network to update the template, and the formula is:
[0065]
[0066] in, Represents the original template. It is a cumulative template. It is the calculated template for the current frame, and the current template is updated by combining these three elements. Based on the cumulative template and the current template, the template can be updated according to the needs of the current frame, while the original template... It can obtain highly reliable information from the original image, thereby avoiding tracking drift, quickly recovering the original target, and improving robustness.
[0067] Furthermore, deep features are first extracted from the initial frame using a convolutional network. Then, use the accumulated template obtained from all previous frames to extract features. And predict the position of the next frame using it. Then, based on these three inputs, a new accumulated template for prediction is generated after passing through two convolutional layers and a ReLU function. For the first frame and Set them all to .
[0068] Its Loss function is set to the Euclidean distance between the updated template and the real template in the next frame:
[0069]
[0070] To track the confidence level of the current template and prevent large-area occlusion, this invention provides a preferred embodiment for executing S200. In this embodiment, the confidence level determination mainly uses the PSR (Peak-to-Sidelobe Ratio) to determine whether there is large-area occlusion in the current image. The region near the peak of the tracking target is defined as a sidelobe. The definition of PSR is as follows:
[0071]
[0072] in and These represent the standard deviation and mean of the sidelobes, respectively. Experiments have shown that when the PSR value is between 20 and 60, the target detection accuracy is relatively high. If the PSR drops to around 7, it indicates large-scale occlusion or detection failure. Therefore, a PSR threshold of 8 is set. When the PSR is around 8, the network stops updating the template; otherwise, it continues updating. This eliminates the impact of large-area, long-term occlusion on the template updating network.
[0073] This invention provides a preferred embodiment for performing S300 to extract features. The feature extraction part mainly consists of a Siamese network, whose backbone network is ResNet50. The feature extraction part consists of two identical networks, one called the template branch, whose input is an image tracking the template, represented as... Another branch, called the search branch, takes the search image as its input, and is represented as... These two branches share the same parameters, and to reduce computational cost, additional steps were added during the method implementation. Convolution reduces the output channels to 256, and selects to use only the template branch center. Regional features, the outputs of the two branches are respectively and .
[0074] Based on S300, this invention provides a preferred embodiment that executes S400, processing the extracted features to obtain foreground and background classification results, and regression distances from the four edges of the search box to pixels in the classification feature map. Foreground points in the classification feature map are sampled, and multiple predicted boxes are obtained based on their distances to the bounding boxes. The optimal predicted box is obtained by applying translation and scale penalties, thus obtaining the tracking result. Specifically, in this embodiment, the processing is divided into two parts: a classification module and a regression module. Both receive... and As input, the classification module outputs two channels, representing the foreground and background classifications respectively. The regression module outputs four channels, representing the distance from the bounding box to the sampling point in the classification feature map, thus obtaining a tracking box that can adapt to the target size.
[0075] Furthermore, the loss includes cross-entropy loss. IoU loss and jitter loss The coefficients are all 1, and their expressions are as follows:
[0076]
[0077]
[0078] Therefore, the final loss function is:
[0079]
[0080] Applying the above embodiments to practice, the target tracking network in this embodiment uses the optimized OTB100 dataset, which adds a medical surgical video dataset to the OTB dataset.
[0081] like Figure 3 As shown, the dataset is constructed in the following way:
[0082] 1) Manually select surgical videos provided by doctors, focusing on video segments with clearly defined surgical objectives, and label the start and end times of each segment. For each video, the labeling method includes the video title, index method, UniqueName (video category_number), and a segment list (recording the start and end times of each segment, organized as a JSON string).
[0083] 2) Use tools to automatically cut videos to obtain video segments with clearly defined surgical target areas. The tool used is the iMovie library, which reads the start and end times of each segment from the annotation information and cuts the video. For each obtained video segment, an index file is automatically populated, including the video segment's UniqueName (video category_number), segment storage address, etc.
[0084] 3) Use DarkLabel to process the obtained video clips, select each frame to obtain GroundTruth data, and store it in an XML file.
[0085] 4) Use a Python script to read the XML file, read the bounding box content in GroundTruth, and store it in the video number.txt file. Then, decompose the video segment into each frame image and the corresponding bounding box information.
[0086] 5) Name the processed dataset as a "Surgery" folder and put it into the OTB100 dataset along with the Groundtruth.txt file. This will serve as the final dataset for training and testing the network.
[0087] Through specific practical experiments, in a medical surgical video scenario, with an Intel Core i7-11700K CPU and a GeForce GTX 1660 SUPER (6GB) graphics card, this embodiment can run at a real-time speed exceeding 40fps. Furthermore, on test sets with feature changes, large-scale occlusion, and large-scale movement, the number of tracking frames with pixel errors within the set threshold accounts for more than 90% of all frames, demonstrating higher accuracy and robustness compared to other methods. The obtained data is as follows:
[0088] Table 1. Longitudinal comparison with the SiamBAN method
[0089]
[0090] Table 2. Horizontal comparison with other methods
[0091]
[0092] As shown in Table 1, the online update method (SiamBAN+UPN) improved the SiamBAN network by 5.41%, and the confidence parameter (SiamBAN+PSR) improved it by 2.30%. The method proposed in this embodiment achieves an overall improvement of 7.14% compared to the SiamBAN method. While the addition of the update network slightly reduces the overall speed, this has little impact on the application environment of this embodiment and is acceptable. Table 2 shows that this embodiment ranks highly in accuracy among methods from recent years and has a faster tracking speed compared to methods with equivalent accuracy. Although it is not the fastest compared to other methods, it still maintains real-time tracking (greater than 24fps).
[0093] Based on the same concept as the above embodiments, the present invention also provides another embodiment: an automatic target region tracking system for surgical videos, comprising:
[0094] A template update tracking module, which updates the tracking module in real time;
[0095] A confidence level determination module, which determines the reliability of the tracking template update through a confidence level parameter;
[0096] The feature extraction module uses a Siamese network to process the input image and extract feature maps from the tracking template image and the search image.
[0097] The regression module includes a classification module and a regression module. It accepts feature maps of the tracking template image and the search image, and outputs the classification results of the foreground and background and the regression results of the distance from the border to the pixel in the classification feature map, respectively.
[0098] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding modules, devices, units, etc. in the system. Those skilled in the art can implement the steps of the method by referring to the technical solution of the system. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.
[0099] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0100] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.
Claims
1. A method for automatic tracking of target regions in surgical videos, characterized in that, include: Real-time updates to the tracking template; The confidence level parameter is used to monitor the reliability of the tracking template. Features are extracted from the image of the tracking template and the search image respectively; Based on the current tracking template, the extracted features are processed to obtain the tracking result; The UpdateNetRes512 network is selected, and a general function is learned to update the tracking template; wherein, represents an original template feature map, is a cumulative template feature map, is a predicted current frame template feature map; The real-time update tracking template includes: extracting deep features from the initial frame as the original template feature map through a convolutional network ; extracting features using the accumulated template obtained from all previous frames and predicting the position of the next frame by it ; The , and are input into the UpdateNetRes512 network, and the predicted current frame template feature map is output after two convolution layers and a ReLU function is also a newly accumulated template feature map, i.e., the current tracking template; The first frame time, and are set to .
2. The surgical video target area automatic tracking method of claim 1, wherein, The real-time update tracking template includes: using a residual network with two convolutional layers and one activation function to fuse the input original template, cumulative template and prediction template to update the current tracking template.
3. The method of claim 1, wherein, The loss function of the UpdateNetRes512 network is set to the Euclidean distance between the updated current tracking template and the ground truth template in the next frame: 。 4. The method of claim 1, wherein, The confidence level is selected using the peak-to-sidelobe ratio (PSR), which is defined as follows: ； wherein and are the standard deviation and mean of the sidelobes, respectively; Set the PSR threshold. When the PSR is within the threshold, stop updating the network to update the tracking template; otherwise, continue updating.
5. The method of claim 1, wherein, The features are extracted from the tracking template and the search image respectively, and the Siamese network is used for implementation. The Siamese network includes: a template branch, which inputs an image of the current tracking template, denoted as , and outputs ; a search branch, having as input the search image, denoted as , and having as output .
6. The surgical video target area automatic tracking method of claim 5, wherein, The backbone network of the Siamese network is ResNet50, the template branch and the search branch share the same parameters, and the ResNet50 is increased convolution, reducing the output channels to 256; the template branch center selects region features.
7. The method of claim 6, wherein, The process of processing the extracted features based on the current tracking template to obtain the tracking result includes: The pixels of the image of the tracking template are classified to determine whether they belong to the foreground or the background, and a classification feature map is obtained. The distance from the bounding box to the sampling point in the classification feature map of the search image is predicted to obtain the deviation coordinate map; The foreground points in the classification feature map are sampled, and multiple prediction boxes are obtained based on their distance from the bounding box. Apply translation and scale penalties to obtain the best predicted bounding box as the tracking result.
8. A surgical video target region automatic tracking system for implementing the surgical video target region automatic tracking method of claim 1, characterized in that, include: A template update tracking module, which updates the tracking module in real time; A confidence level determination module, which determines the reliability of the tracking template update through a confidence level parameter; The feature extraction module uses a Siamese network to process the input image and extract feature maps from the tracking template image and the search image. The regression module includes a classification module and a regression module. It simultaneously receives the feature maps of the tracking template image and the search image, and outputs the classification feature maps of the foreground and background and the regression results of the distance from the bounding box to the sampling point in the classification feature map, respectively.
9. The surgical video target area automatic tracking system of claim 8, wherein, The automatic target region tracking system for surgical videos was trained using the OTB100 dataset, which incorporates medical surgical video data.