Keyframe-guided virtual marker dynamic tracking method and system

Abstract

The present invention provides a keyframe-guided virtual marker dynamic tracking method and system, which relates to the field of virtual marker tracking. In the present invention, firstly, the virtual marker marked by the doctor on the first color image frame is received; secondly, a query mechanism for an online keyframe pool is designed and initialized; again, multiple similar keyframes are queried from the keyframe to reduce the difference between similar keyframes and the current color image frame, simplifying the tracking task; then, the tracking area is obtained using a sparse matching method; finally, multiple frames of optical flow are fused in the tracking area to obtain the final tracking marker, so as to achieve pixel-level precision tracking of the two-dimensional virtual marker on the endoscopic image, and ensure the intuitiveness and accuracy of information transmission. The two-stage tracking method adopted reduces erroneous tracking. In addition, by dynamically updating the keyframe pool, the diversity of the keyframe pool is enriched, and the robustness and accuracy of the dynamic tracking of virtual markers are further improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual marker tracking, and in particular to a key frame-guided virtual marker dynamic tracking method and system. Background Art

[0002] Endoscopic image marker tracking plays a vital role in various medical procedures, such as surgical navigation, augmented reality, and scene understanding. By focusing on the movement, deformation, or feature changes of specific points, it enables precise positioning and monitoring of specific areas or features within the endoscope's field of view.

[0003] In related technologies, the tracking of virtual markers such as surgical paths is mostly done by marking tissues with physical markers such as near-infrared and fluorescent markers, or using preoperative images for surgical path planning and operation. However, most tissue tracking methods can only achieve regional-level accuracy and cannot meet the needs of pixel-level precision tracking in complex surgical scenarios. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the deficiencies in the prior art, the present invention provides a keyframe-guided virtual marker dynamic tracking method and system, which solves the technical problem that only regional-level accuracy can be achieved and the requirements for pixel-level precision tracking of complex surgical scenes cannot be met.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A keyframe-guided virtual marker dynamic tracking method, comprising:

[0009] Receiving a virtual mark marked by a doctor on a first color image frame, wherein the virtual mark is represented by a target point set at the pixel level;

[0010] Obtain the target area where the target point set on the first color image frame is located, and use the optical flow algorithm to predict the trajectory of each point, obtain the target point set and the target area where it is located on the subsequent multiple color image frames to initialize the key frame pool;

[0011] For the current color image frame, traverse the key frame pool, use a sparse matching method to query similar key frames, and obtain the best template based on the sparse matching results between the queried similar key frames and the current color image frame;

[0012] When the best template is not an empty set, obtaining a tracking area on the current color image frame based on the target area on each similar key frame in the best template;

[0013] Based on the target point set in each of the target areas, the motion trajectory of each point is predicted using the optical flow algorithm to obtain the target point set on the corresponding tracking area; the target point set on each of the tracking areas is merged to obtain the final target point set, which is used as the tracking mark on the current color image frame.

[0014] Preferably, after acquiring the target point set on the current color image frame, the virtual marker dynamic tracking method further includes:

[0015] When the current color image frame is determined to be a new key frame based on a preset condition, it is placed in the key frame pool; wherein the preset condition includes:

[0016] (1) Compared with the virtual marker, a complete tracking marker is obtained;

[0017] (2) obtaining a tracking area where the final target point set is located, wherein the center of the tracking area is always within a preset range of the current color image frame;

[0018] (3) if the similar key frame in the best template is greater than one, the distance from the last new key frame to the key frame pool exceeds a first preset number of color image frames;

[0019] If the similar keyframe in the best template is equal to one, more than a second preset number of color image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking mark to the virtual mark is consistent with the scale change of the image frames in which they are located;

[0020] (4) The maximum capacity of the keyframe pool is not exceeded.

[0021] Preferably, the initialization of the key frame pool specifically includes:

[0022] use represents a key frame, where j is the key frame index in the key frame pool; Represents RGB value, W、H Represents width and height respectively; Indicates the target area; , They represent the extracted Feature points and feature descriptions in; represents the target point set;

[0023] The 0th color image frame F 0 specifies a key frame K 0∈ K , and define the target point set on it as , Indicates s The coordinates of the pixels, n Indicates the number of pixels. represents the set of real numbers;

[0024] by The central area Generated by:

[0025]

[0026] Among them, the center point , for The arithmetic mean of the maximum and minimum values ​​of the horizontal coordinates of all pixels in , for The arithmetic mean of the maximum and minimum values ​​of the ordinates of all pixels in ; L is the side length of the target area, L Not less than The difference between the maximum and minimum values ​​of the horizontal and vertical coordinates of all pixels in;

[0027] Define the target area Store four vertices and the center point , expressed as ,in Depend on It can be seen that The four vertices of: upper left, upper right, lower left, lower right;

[0028] The optical flow algorithm is used to predict the next two color image frames. F 1 and F 2 Movement V 0→1 ∈ , V 0→2 ∈ , and calculate each pixel point on the virtual marker according to the optical flow result exist F 1 and F Trajectory on 2:

[0029]

[0030] in, Inter (∙) represents the sampling process, and the corresponding target point set is obtained and , and refer to formula (1) to obtain the target area and ;

[0031] Will F 1.F 2 corresponds to the key frame K 1 、K 2, with K 0 together form the initial key frame pool P ={ K 0, K 1, K 2}.

[0032] Preferably, obtain the current color image frame F i Feature points and feature descriptions;

[0033] Traverse each key frame in the key frame pool K j , using the sparse matching method to obtain F i and K j The matching points between

[0034] statistics K j The number of matching points in the upper target area n ji , and select n ji The three largest keyframes are used as similar keyframes K simi ={ K a ,K b ,K c} P ,

[0035] The best template is obtained using the following formula:

[0036]

[0037] in, K j Indicates the first keyframe in the keyframe pool. j keyframes, j The value of a、b or c;n ji express K j Target area on With the current color image frame F i The number of matching points between ; s is the first threshold; K simi Represents all similar keyframes found.

[0038] Preferably, the step of obtaining the tracking area on the current color image frame based on the target area on each similar key frame in the best template specifically includes:

[0039] based on K j ⊆ K best and F i The matching points between them are used to estimate the homography matrix using RANSAC H ji , H ji The accuracy of get, n in is the number of interior points;

[0040] when ≥λ , 0< λ<1 , l is the second threshold, then for the target area The four vertices on Perform homography transformation and get exist F i The corresponding position on :

[0041]

[0042] Otherwise, the vertex coordinates are transformed using the neighborhood similarity transformation method:

[0043] Extract from sparse matching results F i and K j A set of matches , x im ∈ and y jm ∈ is a two-dimensional column vector representing the position of the feature point;

[0044] Search target area vertices of N Neighbors , get the corresponding matching point , to get the point exist F i The corresponding position on :

[0045]

[0046] in , w t and and is inversely proportional to the distance between them;

[0047] Based on vertex coordinates , determine an initial region, expressed as:

[0048]

[0049] in, , Respectively exist F i The corresponding position on The horizontal and vertical coordinates of u min 、u max Respectively represent the maximum and minimum values ​​of the horizontal coordinates of the four vertices; v min 、v max Respectively represent the maximum and minimum values ​​of the ordinates of the four vertices;

[0050] Based on the coordinates of the four vertices of the initial area, determine the tracking area The center point ;

[0051]

[0052] in, 、 Respectively represent the tracking area The center point The horizontal and vertical coordinates of

[0053] Refer to formula (1) to obtain the tracking area on the current color image frame .

[0054] Preferably, based on the target point set in each target area, the motion trajectory of each point is predicted by using an optical flow algorithm to obtain the target point set in the corresponding tracking area; and the target point set in each tracking area is merged to obtain the final target point set; specifically including:

[0055] Target point set Each point in p j,s In the whole frame K j The coordinates on the target area Coordinates The mapping relationship is as follows:

[0056]

[0057] Optical flow in the target area and tracking area The optical flow prediction result is V j→i , the sparse matching result should be consistent with the optical flow prediction result, and the difference between the two is calculated using the following formula:

[0058]

[0059] like d ≤ c , c is the third threshold, then the tracking area The target point set on Each point in p ji,s Obtained through the following methods:

[0060]

[0061] otherwise, Not participating Calculation of

[0062] Merge the target point sets on each tracking area , get the final target point set , expressed as:

[0063]

[0064] in, l ji is the weight, and =1, l ji and n ji Directly proportional.

[0065] A keyframe-guided virtual marker dynamic tracking system, comprising:

[0066] A marking receiving module, used for receiving a virtual marking marked by a doctor on the first color image frame; wherein the virtual marking is represented by a target point set at the pixel level;

[0067] The key frame pool initialization module is used to obtain the target area where the target point set on the first color image frame is located, and use the optical flow algorithm to predict the trajectory of each point, obtain the target point set and the target area where it is located on the subsequent multiple color image frames, so as to initialize the key frame pool;

[0068] The best template acquisition module is used to traverse the key frame pool for the current color image frame, perform similar key frame query using a sparse matching method, and obtain the best template based on the sparse matching results between the queried similar key frames and the current color image frame;

[0069] A region tracking module, used for obtaining a tracking region on a current color image frame based on a target region on each similar key frame in the best template when the best template is not an empty set;

[0070] The marker tracking module is used to predict the motion trajectory of each point based on the target point set in each target area using the optical flow algorithm to obtain the target point set on the corresponding tracking area; merge the target point set on each tracking area to obtain the final target point set, and use it as the tracking marker on the current color image frame.

[0071] Preferably, after the marker tracking module obtains the target point set on the current color image frame, the virtual marker dynamic tracking system further includes:

[0072] The key frame pool updating module is used to determine that the current color image frame is a new key frame based on preset conditions and put it into the key frame pool; wherein the preset conditions include:

[0073] (1) Compared with the virtual marker, a complete tracking marker is obtained;

[0074] (2) obtaining a tracking area where the final target point set is located, wherein the center of the tracking area is always within a preset range of the current color image frame;

[0075] (3) if the similar key frame in the best template is greater than one, the distance from the last new key frame to the key frame pool exceeds a first preset number of color image frames;

[0076] If the similar keyframe in the best template is equal to one, more than a second preset number of color image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking mark to the virtual mark is consistent with the scale change of the image frames in which they are located;

[0077] (4) The maximum capacity of the keyframe pool is not exceeded.

[0078] A storage medium stores a computer program for key-frame-guided virtual marker dynamic tracking, wherein the computer program enables a computer to execute the virtual marker dynamic tracking method as described above.

[0079] An electronic device, comprising:

[0080] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including means for executing the virtual marker dynamic tracking method as described above.

[0081] (III) Beneficial effects

[0082] The present invention provides a keyframe-guided virtual marker dynamic tracking method and system. Compared with the prior art, it has the following beneficial effects:

[0083] In the present invention, firstly, the virtual mark marked by the doctor on the first color image frame is received; secondly, a query mechanism of an online key frame pool is designed and initialized; again, multiple similar key frames are queried from the key frames to reduce the difference between similar key frames and the current color image frame and simplify the tracking task; then, the tracking area is obtained by using a sparse matching method; finally, multiple frames of optical flow are fused in the tracking area to obtain the final tracking mark, so as to achieve pixel-level precision tracking of the two-dimensional virtual mark on the endoscopic image and ensure the intuitiveness and accuracy of information transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0085] Figure 1 A block diagram of a key frame guided virtual marker dynamic tracking method provided by an embodiment of the present invention;

[0086] Figure 2 A block diagram of another key frame guided virtual marker dynamic tracking method provided by an embodiment of the present invention;

[0087] Figure 3 A complete technical roadmap for a keyframe-guided virtual marker dynamic tracking method provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] The embodiments of the present application provide a keyframe-guided virtual marker dynamic tracking method and system to solve the technical problem that only regional-level accuracy can be achieved and the requirements for pixel-level precision tracking of complex surgical scenes cannot be met.

[0090] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0091] Actual surgical scenes are complex and changeable. To achieve accurate virtual marker tracking, the following conditions need to be met: 1) When the camera and human tissue move, the tracking marker needs to follow the changes in the tissue texture of the endoscopic image; 2) It can identify the situation where the tracking marker moves out of the endoscope field of view; 3) The tracking marker can be re-tracked when it reappears; 4) It can identify the situation where the marker is partially visible and perform local tracking.

[0092] The key frame-guided virtual marker dynamic tracking method provided by the embodiment of the present invention meets the above conditions and at least involves the following key points:

[0093] (1) Aiming at the characteristics of the flexible dynamic environment in the body, a keyframe-guided virtual marker dynamic tracking method is proposed, which is applicable to but not limited to minimally invasive endoscopic surgery scenarios. This method combines sparse features and dense optical flow to achieve incremental complex tracking.

[0094] (2) Enriching the diversity of tracking templates (i.e., key frames in the key frame pool): A key frame extraction method is proposed to determine the addition of key frames based on conditions such as image similarity, thereby enriching the diversity of tracking templates.

[0095] (3) Reducing the complexity of tracking: An online keyframe pool query mechanism is proposed, which uses feature similarity to identify the best reference template, reduces the complexity error rate of tracking, and can handle situations where the target is lost and reappears.

[0096] (4) Two-stage tracking improves accuracy: When tracking virtual markers, a two-stage matching process from coarse to fine is adopted. First, similar key frames are determined by traversing the key frame pool and sparse matching. The tracking area is first determined using the sparse matching results, and then the tracking area is tracked to reduce erroneous tracking.

[0097] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0098] Embodiment 1:

[0099] like Figure 1 As shown, an embodiment of the present invention provides a key frame guided virtual marker dynamic tracking method, comprising:

[0100] S1, receiving the virtual mark marked by the doctor on the first color image frame; wherein the virtual mark is represented by a target point set at the pixel level;

[0101] S2, obtaining the target area where the target point set on the first color image frame is located, and using the optical flow algorithm to predict the trajectory of each point, obtaining the target point set and the target area where they are located on the subsequent multiple color image frames to initialize the key frame pool;

[0102] S3, for the current color image frame, traverse the key frame pool, use the sparse matching method to query similar key frames, and obtain the best template based on the sparse matching results between the queried similar key frames and the current color image frame;

[0103] S4, when the best template is not an empty set, obtaining a tracking area on the current color image frame based on the target area on each similar key frame in the best template;

[0104] S5. Based on the target point set in each target area, the motion trajectory of each point is predicted using the optical flow algorithm to obtain the target point set on the corresponding tracking area; the target point set on each tracking area is merged to obtain the final target point set, and it is used as the tracking mark on the current color image frame.

[0105] The embodiment of the present invention realizes pixel-level precision tracking of two-dimensional virtual markers on endoscopic images, ensuring intuitiveness and accuracy of information transmission.

[0106] In an optional embodiment, if Figure 2 As shown, after acquiring the target point set on the current color image frame, the virtual marker dynamic tracking method further includes:

[0107] S6. When the current color image frame is determined to be a new key frame based on preset conditions, the frame is placed in a key frame pool; wherein the preset conditions include:

[0108] (1) Compared with the virtual marker, a complete tracking marker is obtained;

[0109] (2) obtaining a tracking area where the final target point set is located, wherein the center of the tracking area is always within a preset range of the current color image frame;

[0110] (3) if the similar key frame in the best template is greater than one, the time between placing the last new key frame into the key frame pool exceeds a first preset number of color image frames;

[0111] If the similar keyframe in the best template is equal to one, more than a second preset number of color image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking mark to the virtual mark is consistent with the scale change of the image frames in which they are located;

[0112] (4) The maximum capacity of the keyframe pool is not exceeded.

[0113] The embodiment of the present invention enriches the diversity of the key frame pool by dynamically updating the key frame pool, thereby further improving the robustness and accuracy of the virtual marker dynamic tracking.

[0114] like Figure 3 As shown, Figure 3 The complete technical route of the virtual marker dynamic tracking method is given. Figure 3 The following are the steps of the above solution:

[0115] In step S1 , a virtual mark marked by a doctor on a first color image frame is received.

[0116] It can be understood that in minimally invasive surgery, the virtual marker tracks the position of the dynamic positioning target and provides real-time feedback to the doctor about the surgical position. The doctor selects the template frame as the 0th color image frame F 0∈ , and use tools such as a mouse or stylus to mark virtual markers such as planned surgical trajectories on the display. W、H Respectively F 0 width and height.

[0117] This step receives the above surgical marks and represents them using a pixel-level target point set. ;in represents a virtual marker on frame 0, p 0,s ∈ Indicates s The coordinates of the pixels, n Indicates the number of pixels, Represents the set of real numbers.

[0118] Through the above setting, it is convenient to follow the movement of the endoscope to i frame F i ∈ , use the proposed tracking method to get the virtual marker on this frame , the tracking results Annotate to the current color image frame F i It can provide visual guidance for doctors.

[0119] In step S2, the target area where the target point set on the first color image frame is located is obtained, and the trajectory of each point is predicted using the optical flow algorithm to obtain the target point set and the target area where it is located on the subsequent multiple color image frames to initialize the key frame pool.

[0120] The core principle of the keyframe strategy is that it can identify and record key moments as reference points for error correction. Using keyframes not only reduces computational complexity, but also improves tracking efficiency and ensures the stability of long-term tracking.

[0121] Exemplarily, this step needs to determine the initial three key frames as the template for subsequent tracking. Accordingly, the specific steps for initializing the key frame pool are as follows:

[0122] It should be noted that in the embodiments of the present invention, represents a key frame, where j is the key frame index in the key frame pool; Represents RGB value, W、H Represents width and height respectively; Indicates the target area; , They represent the extracted Feature points and feature descriptions in; represents the target point set.

[0123] First, the 0th color image frame F 0 specifies a key frame K 0∈ K , and define the target point set on it as , Indicates s The coordinates of the pixels, n Indicates the number of pixels. Represents the set of real numbers.

[0124] Secondly, The central area Generated by:

[0125]

[0126] Among them, the center point , for The arithmetic mean of the maximum and minimum values ​​of the horizontal coordinates of all pixels in , for The arithmetic mean of the maximum and minimum values ​​of the ordinates of all pixels in ; L is the side length of the target area, L Not less than The difference between the maximum and minimum values ​​of the horizontal and vertical coordinates of all pixels in .

[0127] Again, using existing descriptor-based sparse matching methods such as SIFT, lightglue, etc. F 0 to extract feature points and descriptors from the entire image and locate them in the target area The internal feature points and descriptors are recorded as .

[0128] Next, define the target area Store four vertices and the center point , expressed as ,in Depend on It can be seen that The four vertices are: upper left, upper right, lower left, and lower right.

[0129] Then, the optical flow algorithm is used to predict the next two color image frames. F 1 and F 2 Movement V 0→1 ∈ , V 0→2 ∈ , and calculate each pixel point on the virtual marker according to the optical flow result exist F 1 and F Trajectory on 2:

[0130]

[0131] in, Inter (∙) represents the sampling process, and the corresponding target point set is obtained and , and refer to formula (1) to obtain the target area and .

[0132] Finally, F 1. F 2 corresponds to the key frame K 1 、K 2, with K 0 together form the initial key frame pool P ={ K 0, K 1, K 2}.

[0133] In step S3, for the current color image frame, the key frame pool is traversed, a similar key frame query is performed using a sparse matching method, and the best template is obtained based on the sparse matching results between the queried similar key frames and the current color image frame; specifically, the following steps are included:

[0134] Get the current color image frame F i Feature points and feature descriptions;

[0135] Traverse each key frame in the key frame pool K j , using the sparse matching method to obtain F i and K j The matching points between

[0136] statistics K j The number of matching points in the upper target area n ji , and select n ji The three largest keyframes are used as similar keyframes K simi ={ K a ,K b ,K c} P ,

[0137] The best template is obtained using the following formula:

[0138]

[0139] in, K j Indicates the first keyframe in the keyframe pool. j keyframes, j The value of a、b or c;n ji express K j Target area on With the current color image frame F i The number of matching points between ; s is the first threshold; K simi Represents all similar keyframes found.

[0140] It should be noted that if Figure 3 As shown, based on the number of similar keyframes in the best template card ( K best ), current color image frame F i It can be divided into two states:

[0141] (1) If card ( K best )>0, that is, when the best template is not an empty set, it means that there is an area in the current frame that is highly similar to the target area on the key frame, and there is a mark that needs to be tracked. At this time, the current frame is considered to be a normal frame or a candidate key frame, and then the next step is entered to track the target area and accurately locate the pixel position of the virtual marker point.

[0142] It can be understood that setting this condition can identify the situation where the virtual tag is lost and then reappears.

[0143] (2) If card ( K best )=0, that is, when the best template is an empty set, it means that the similarity between the current frame and all key frames is not high, the marker has moved out of the endoscope field of view, and there is no marker that needs to be tracked. The current frame is considered to be a frame that does not need to be tracked, and the next color image frame is read directly.

[0144] Then, the embodiment of the present invention adopts a two-stage tracking method. When tracking the virtual marker, a two-stage matching from coarse to fine is adopted. First, similar key frames are determined by traversing the key frame pool and sparse matching. The local area is first determined by using the matching results, and then the local area is tracked to reduce error tracking; see steps S4-S5 for details:

[0145] In step S4, when the best template is not an empty set, the tracking area on the current color image frame is obtained based on the target area on each similar key frame in the best template.

[0146] This step determines the similar area between the current color image frame and the best template frame. It should be noted that tracking is based on the area, not the entire image.

[0147] Accordingly, the tracking area on the current color image frame is obtained, including:

[0148] First, based on homography transformation or neighborhood similarity transformation, determine the target area The four vertices on exist F i The corresponding position on ,include:

[0149] based on K j ⊆ Kbest and F i The matching points between them are used to estimate the homography matrix using RANSAC H ji , H ji The accuracy of get, n in is the number of interior points;

[0150] when ≥λ , 0< λ<1 , l is the second threshold, then for the target area The four vertices on Perform homography transformation and get exist F i The corresponding position on :

[0151]

[0152] Otherwise, the vertex coordinates are transformed using the neighborhood similarity transformation method:

[0153] Extract from sparse matching results F i and K j A set of matches , x im ∈ and y jm ∈ is a two-dimensional column vector representing the position of the feature point;

[0154] Search target area vertices of N Neighbors , get the corresponding matching point , to get the point exist F i The corresponding position on :

[0155]

[0156] in , w t and and is inversely proportional to the distance between them.

[0157] For example, hereN You can take four.

[0158] Next, based on the vertex coordinates , determine an initial region, expressed as:

[0159]

[0160] in, , Respectively exist F i The corresponding position on The horizontal and vertical coordinates of u min 、u max Respectively represent the maximum and minimum values ​​of the horizontal coordinates of the four vertices; v min 、v max Respectively represent the maximum and minimum values ​​of the vertical coordinates of the four vertices.

[0161] Then, based on the coordinates of the four vertices of the initial area, the tracking area is determined. The center point ;

[0162]

[0163] in, 、 Respectively represent the tracking area The center point The horizontal and vertical coordinates of .

[0164] Finally, refer to formula (1) to obtain the tracking area on the current color image frame .

[0165] In step S5, based on the target point set in each target area, the motion trajectory of each point is predicted using the optical flow algorithm to obtain the target point set on the corresponding tracking area; the target point set on each tracking area is merged to obtain the final target point set, and it is used as the tracking mark on the current color image frame.

[0166] This step marks and tracks the target area and the tracking area; specifically includes:

[0167] Target point set Each point in p j,s In the whole frame K j The coordinates on the target area Coordinates The mapping relationship is as follows:

[0168]

[0169] Optical flow in the target area and tracking area The optical flow prediction result is V j→i , the sparse matching result should be consistent with the optical flow prediction result, and the difference between the two is calculated using the following formula:

[0170]

[0171] like d ≤ c , c is the third threshold, then the tracking area The target point set on Each point in p ji,s Obtained through the following methods:

[0172]

[0173] otherwise, Not participating Calculation.

[0174] Merge the target point sets on each tracking area , get the final target point set , expressed as:

[0175]

[0176] in, l ji is the weight, and =1, l ji and n ji Directly proportional.

[0177] It should be noted that in fact, the embodiment of the present invention can obtain relatively accurate results by selecting one or two key frames with the highest similarity. Whether each pixel in the image is on the image, the complete tracking mark or the partial tracking mark is overlaid on the current color image frame to provide real-time guidance for the doctor.

[0178] In step S6, based on the preset conditions, when the current color image frame is determined to be a new key frame, it is put into the key frame pool.

[0179] After each tracking is completed, the embodiment of the present invention further determines the current color image frameF i Whether it is a new keyframe, thereby enriching the keyframe pool. Adding a new keyframe must meet the following preset conditions:

[0180] (1) Compared with the virtual marker, a complete tracking marker is obtained; it is expressed as , a keyframe must contain a complete virtual marker to have sufficient information.

[0181] (2) Obtain the final target point set Tracking area , the center of the tracking area Always in the current color image frame F i within the preset range; expressed as ∈ ,in l is the fourth threshold. In order to eliminate redundancy, and Needs to be satisfied , or 1 is the fifth threshold, and or 1<1. This condition hopes that there is a certain difference between the target area and the key frame, and the target area on the key frame is as far away from the edge of the image as possible.

[0182] (3) If the number of similar key frames in the best template is greater than one, the time between placing the last new key frame into the key frame pool exceeds a first preset number of color image frames. This condition is intended to exclude frames with high similarity from being added to the key frame pool.

[0183] Exemplarily, the first preset number here is ten, and the corresponding expression is card ( K best )>1, i - f >10, of which f The index of the most recently added keyframe, 10 frames since the last keyframe was inserted.

[0184] If the similar keyframe in the best template is equal to one, more than a second preset number of color image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking mark to the virtual mark is consistent with the scale change of the image frame in which they are located. This condition can exclude some image frames that are not tracked accurately due to large scale changes from entering the keyframe pool.

[0185] Exemplarily, the second preset number here is four, which is correspondingly expressed as card ( K best )=1,i - f >4, and ,in or 2. or 3 are the sixth threshold and the seventh threshold respectively, and or 2<1< or 3. r The calculation formula is r =max{ h 0 ,h 1}, h 0 、h 1 respectively indicates F i and K j The homography matrix H ji The first two diagonal elements of h 0 、h 1 corresponds to the image scaling factor in the horizontal and vertical directions respectively.

[0186] (4) does not exceed the maximum capacity of the keyframe pool; expressed as card ( P )≤ n max , n max Keyframe Pool P The maximum capacity, for example, here n max Take 16. In order to prevent unnecessary calculations from being added to the repeated key frames, when a certain capacity is reached, it is considered that the key frames have saved the key information from each perspective.

[0187] Therefore, the embodiment of the present invention can also handle the situation of target loss and re-tracking, and achieve robust tracking.

[0188] Embodiment 2:

[0189] An embodiment of the present invention provides a keyframe-guided virtual marker dynamic tracking system, comprising:

[0190] A marking receiving module, used for receiving a virtual marking marked by a doctor on the first color image frame; wherein the virtual marking is represented by a target point set at the pixel level;

[0191] The key frame pool initialization module is used to obtain the target area where the target point set on the first color image frame is located, and use the optical flow algorithm to predict the trajectory of each point, obtain the target point set and the target area where it is located on the subsequent multiple color image frames, so as to initialize the key frame pool;

[0192] The best template acquisition module is used to traverse the key frame pool for the current color image frame, perform similar key frame query using a sparse matching method, and obtain the best template based on the sparse matching results between the queried similar key frames and the current color image frame;

[0193] A region tracking module, used for obtaining a tracking region on a current color image frame based on a target region on each similar key frame in the best template when the best template is not an empty set;

[0194] The marker tracking module is used to predict the motion trajectory of each point based on the target point set in each target area using the optical flow algorithm to obtain the target point set on the corresponding tracking area; merge the target point set on each tracking area to obtain the final target point set, and use it as the tracking marker on the current color image frame.

[0195] In an optional implementation, after the marker tracking module acquires the target point set on the current color image frame, the virtual marker dynamic tracking system further includes:

[0196] The key frame pool updating module is used to determine that the current color image frame is a new key frame based on preset conditions and put it into the key frame pool; wherein the preset conditions include:

[0197] (1) Compared with the virtual marker, a complete tracking marker is obtained;

[0198] (2) obtaining a tracking area where the final target point set is located, wherein the center of the tracking area is always within a preset range of the current color image frame;

[0199] (3) if the similar key frame in the best template is greater than one, the distance from the last new key frame to the key frame pool exceeds a first preset number of color image frames;

[0200] If the similar keyframe in the best template is equal to one, more than a second preset number of color image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking mark to the virtual mark is consistent with the scale change of the image frames in which they are located;

[0201] (4) The maximum capacity of the keyframe pool is not exceeded.

[0202] Embodiment 3:

[0203] An embodiment of the present invention provides a storage medium storing a computer program for key-frame-guided virtual marker dynamic tracking, wherein the computer program enables a computer to execute the virtual marker dynamic tracking method as described in Example 1.

[0204] Embodiment 4:

[0205] An embodiment of the present invention provides an electronic device, including:

[0206] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including means for executing the virtual marker dynamic tracking method as described above.

[0207] It can be understood that the key frame-guided virtual marker dynamic tracking system, storage medium and electronic device provided in the embodiments of the present invention correspond to the key frame-guided virtual marker dynamic tracking method provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the virtual marker dynamic tracking method, which will not be repeated here.

[0208] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0209] 1. In an embodiment of the present invention, firstly, a virtual mark marked by a doctor on the first color image frame is received; secondly, a query mechanism of an online key frame pool is designed and initialized; again, multiple similar key frames are queried from the key frames to reduce the difference between similar key frames and the current color image frame and simplify the tracking task; then, a sparse matching method is used to obtain the tracking area; finally, multiple frames of optical flow are fused in the tracking area to obtain the final tracking mark, so as to achieve pixel-level precision tracking of the two-dimensional virtual mark on the endoscopic image and ensure the intuitiveness and accuracy of information transmission.

[0210] 2. The embodiment of the present invention adopts a two-stage tracking method. When tracking virtual markers, a two-stage matching from coarse to fine is adopted. First, similar key frames are determined by traversing the key frame pool and sparse matching. The matching results are used to first determine the local area, and then the local area is tracked to reduce erroneous tracking.

[0211] 3. The embodiment of the present invention enriches the diversity of the key frame pool by dynamically updating the key frame pool, thereby further improving the robustness and accuracy of the virtual marker dynamic tracking.

[0212] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0213] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A keyframe-guided virtual marker dynamic tracking method, characterized in that: include: receiving a virtual mark annotated by a doctor on the first color image frame; The virtual marker is represented by a target point set at pixel level; Obtain the target area where the target point set on the first color image frame is located, and use the optical flow algorithm to predict the trajectory of each point, obtain the target point set and the target area where it is located on the subsequent multiple color image frames to initialize the key frame pool; For the current color image frame, traverse the key frame pool, use a sparse matching method to query similar key frames, and obtain the best template based on the sparse matching results between the queried similar key frames and the current color image frame; When the best template is not an empty set, obtaining a tracking area on the current color image frame based on the target area on each similar key frame in the best template; Based on the target point set in each of the target areas on the first color image frame, the motion trajectory of each point is predicted using the optical flow algorithm to obtain the target point set on the corresponding tracking area on the current color image frame; the target point set on each of the tracking areas is merged to obtain the final target point set, and it is used as the tracking mark on the current color image frame.

2. The virtual marker dynamic tracking method according to claim 1, characterized in that: After acquiring the target point set on the current color image frame, the virtual marker dynamic tracking method further includes: When the current color image frame is determined to be a new key frame based on a preset condition, it is placed in the key frame pool; wherein the preset condition includes: (1) Compared with the virtual marker on the first color image frame, a complete tracking marker is obtained on the current color image frame; (2) obtaining a tracking area where the final target point set is located, wherein the center of the tracking area is always within a preset range of the current color image frame; (3) if the similar key frame in the best template is greater than one, the time between placing the last new key frame into the key frame pool exceeds a first preset number of color image frames; If the similar keyframe in the best template is equal to one, more than a second preset number of color image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking mark to the virtual mark is consistent with the scale change of the image frames in which they are located; (4) The maximum capacity of the keyframe pool is not exceeded.

3. The virtual marker dynamic tracking method according to claim 1 or 2, characterized in that: The initialization key frame pool specifically includes: use represents a key frame, where j is the key frame index in the key frame pool; Represents RGB value, W.H. Represents width and height respectively; Indicates the target area; , They represent the extracted Feature points and feature descriptions in; represents the target point set; The 0th color image frame F 0 specifies a key frame K 0∈ K , and define the target point set on it as , Indicates s The coordinates of the pixels, n Indicates the number of pixels. represents the set of real numbers; by The central area Generated by: Among them, the center point , for The arithmetic mean of the maximum and minimum values ​​of the horizontal coordinates of all pixels in , for The arithmetic mean of the maximum and minimum values ​​of the ordinates of all pixels in ; L is the side length of the target area, L Not less than The difference between the maximum and minimum values ​​of the horizontal and vertical coordinates of all pixels in; Define the target area Store four vertices and the center point , expressed as ,in Depend on It can be seen that The four vertices of: upper left, upper right, lower left, lower right; The optical flow algorithm is used to predict the next two color image frames. F 1 and F 2 Movement V 0→1 ∈ , V 0→2 ∈ , and calculate each pixel point on the virtual marker according to the optical flow result exist F 1 and F Trajectory on 2: in, Inter (∙) represents the sampling process, and the corresponding target point set is obtained and , and refer to formula (1) to obtain the target area and ; Will F 1. F 2 corresponds to the key frame K 1 、K 2, with K 0 together form the initial key frame pool P ={ K 0, K 1, K 2}.

4. The virtual marker dynamic tracking method according to claim 3, characterized in that: The method of traversing the key frame pool for the current color image frame, performing a similar key frame query using a sparse matching method, and obtaining an optimal template based on a sparse matching result between the queried similar key frame and the current color image frame specifically includes: Get the current color image frame F i Feature points and feature descriptions; Traverse each key frame in the key frame pool K j , using the sparse matching method to obtain F i and K j The matching points between statistics K j The number of matching points in the upper target area n ji , and select n ji The three largest keyframes are used as similar keyframes K simi ={ K a ,K b ,K c } P , The best template is obtained using the following formula: in, K j Indicates the first keyframe in the keyframe pool. j keyframes, j The value of a, b or c;n ji express K j Target area on With the current color image frame F i The number of matching points between ; σ is the first threshold; K simi Represents all similar keyframes found.

5. The virtual marker dynamic tracking method according to claim 4, characterized in that: The step of obtaining a tracking area on the current color image frame based on the target area on each similar key frame in the best template specifically includes: based on K j ⊆ K best and F i The matching points between them are used to estimate the homography matrix using RANSAC H ji , H ji The accuracy of get, n in is the number of interior points; when ≥λ , 0< λ<1 , λ is the second threshold, then for the target area The four vertices on Perform homography transformation and get exist F i The corresponding position on : Otherwise, the vertex coordinates are transformed using the neighborhood similarity transformation method: Extract from sparse matching results F i and K j A set of matches , x im ∈ and y jm ∈ is a two-dimensional column vector representing the position of the feature point; Search target area vertices of N Neighbors , get the corresponding matching point , to get the point exist F i The corresponding position on : in , w t and and is inversely proportional to the distance between them; Based on vertex coordinates , determine an initial region, expressed as: in, , Respectively exist F i The corresponding position on The horizontal and vertical coordinates of u min 、u max Respectively represent the maximum and minimum values ​​of the horizontal coordinates of the four vertices; v min 、v max Respectively represent the maximum and minimum values ​​of the ordinates of the four vertices; Based on the coordinates of the four vertices of the initial area, determine the tracking area The center point ; in, 、 Respectively represent the tracking area The center point The horizontal and vertical coordinates of Refer to formula (1) to obtain the tracking area on the current color image frame .

6. The virtual marker dynamic tracking method according to claim 5, characterized in that: The method comprises: based on the target point set in each target area on the first color image frame, using the optical flow algorithm to predict the motion trajectory of each point, obtaining the target point set in the corresponding tracking area on the current color image frame; merging the target point set in each tracking area to obtain the final target point set; specifically comprising: Target point set Each point in p j,s In the whole frame K j The coordinates on the target area Coordinates The mapping relationship is as follows: Optical flow in the target area and tracking area The optical flow prediction result is V j→i , the sparse matching result should be consistent with the optical flow prediction result, and the difference between the two is calculated using the following formula: like d ≤ γ , γ is the third threshold, then the tracking area The target point set on Each point in p ji,s Obtained through the following methods: otherwise, Not participating Calculation of Merge the target point sets on each tracking area , get the final target point set , expressed as: in, λ ji is the weight, and =1, λ ji and n ji Directly proportional.

7. A keyframe-guided virtual marker dynamic tracking system, characterized in that: include: A marking receiving module, used for receiving a virtual marking marked by a doctor on the first color image frame; The virtual marker is represented by a target point set at pixel level; The key frame pool initialization module is used to obtain the target area where the target point set on the first color image frame is located, and use the optical flow algorithm to predict the trajectory of each point, obtain the target point set and the target area where it is located on the subsequent multiple color image frames, so as to initialize the key frame pool; The best template acquisition module is used to traverse the key frame pool for the current color image frame, perform similar key frame query using a sparse matching method, and obtain the best template based on the sparse matching results between the queried similar key frames and the current color image frame; A region tracking module, used for obtaining a tracking region on a current color image frame based on a target region on each similar key frame in the best template when the best template is not an empty set; The marker tracking module is used to predict the motion trajectory of each point based on the target point set in each target area on the first color image frame using the optical flow algorithm to obtain the target point set on the corresponding tracking area on the current color image frame; merge the target point set on each tracking area to obtain the final target point set, and use it as the tracking marker on the current color image frame.

8. The virtual marker dynamic tracking system according to claim 7, characterized in that: After the marker tracking module obtains the target point set on the current color image frame, the virtual marker dynamic tracking system further includes: The key frame pool updating module is used to determine that the current color image frame is a new key frame based on preset conditions and put it into the key frame pool; wherein the preset conditions include: (1) Compared with the virtual marker on the first color image frame, a complete tracking marker is obtained on the current color image frame; (2) obtaining a tracking area where the final target point set is located, wherein the center of the tracking area is always within a preset range of the current color image frame; (3) if the similar key frame in the best template is greater than one, the time between placing the last new key frame into the key frame pool exceeds a first preset number of color image frames; If the similar keyframe in the best template is equal to one, more than a second preset number of color image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking mark to the virtual mark is consistent with the scale change of the image frames in which they are located; (4) The maximum capacity of the keyframe pool is not exceeded.

9. A storage medium, characterized in that: It stores a computer program for key-frame-guided virtual marker dynamic tracking, wherein the computer program enables a computer to execute the virtual marker dynamic tracking method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include programs for executing the virtual marker dynamic tracking method as described in any one of claims 1 to 6.

Images