Endoscope branch positioning method, device and equipment based on airway detection tracking

Abstract

The invention provides an endoscope branch positioning method, device and equipment based on airway detection tracking, and the method comprises the steps: obtaining bronchial images at all moments, and carrying out the airway detection of the bronchial images, and obtaining the airway hierarchical relation of the bronchial images; determining a first matching tracking result based on the airway features of the bronchial image at each moment in the past and the airway features of the bronchial image; under the condition that the airway characteristics of the bronchial image are not matched with the airway characteristics of the bronchial image at each moment in the past, determining a second matching tracking result based on a tracheal tree branch map and an airway hierarchical relationship; based on the first matching tracking result and the second matching tracking result, a target matching tracking result is determined, in the matching tracking process, a new detection result in each frame of image is matched with all historical tracking results, the error accumulation problem occurring in an incremental positioning method is avoided, automatic recovery is achieved after errors occur, and the accuracy of positioning is improved. And the accuracy and reliability of endoscope branch positioning are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an endoscope branch positioning method, device and equipment based on airway detection and tracking. Background Art

[0002] Vision-based endoscope positioning methods are usually divided into two categories: coordinate-level positioning and branch positioning.

[0003] Although coordinate-level positioning can obtain more specific location information, the millimeter-level error of the algorithm will cause the positioning effect to become worse and worse as the endoscope goes deeper because the diameter of the tracheal branches with higher grades gradually decreases. Since the algorithms that are accurate to coordinates usually adopt an incremental strategy, the error will gradually accumulate and increase over time. Specifically in terms of direction, the shortcomings of existing coordinate-level positioning methods also include: 1) The SFM (Structure From Motion) method has a strong theoretical basis, but the endoscope image is different from the open scene image and often lacks feature points. When there are too few feature points, PnP (Perspective-n-Point) will not be able to solve and lose the positioning. 2) Most 2D / 3D registrations are based on iterative optimization algorithms, which means that it takes a long time to calculate each frame of positioning, and the posture of the endoscope changes very quickly in actual examinations, resulting in positioning loss. The remaining methods are based on deep learning retrieval. For each user, it is necessary to collect their virtual endoscope images to train the retrieval network, which requires a lot of computing power and storage space. It also takes a long time to prepare for training before surgery, which is difficult to apply in practice.

[0004] Existing branch-level positioning methods are based on too many assumptions (such as stable camera motion, the airway always appearing in the center of the field of view, the bifurcated structure of the airway, uniform lighting conditions, etc.), based on simple airway detection and tracking strategies (traditional detection and tracking algorithms, only detecting the next level of airway), and need to combine control signals to obtain the approximate direction of camera movement. Therefore, they have only been verified in simulation environments or small data, and are difficult to be applied in practice. Summary of the invention

[0005] The present invention provides an endoscope branch positioning method, device and equipment based on airway detection and tracking, so as to solve the defects of endoscope positioning based on coordinate-level positioning and branch positioning in the prior art.

[0006] The present invention provides an endoscope branch positioning method based on airway detection and tracking, comprising:

[0007] Acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images;

[0008] Determining a first matching pursuit result based on the airway features of the bronchial images at each moment in the past and the airway features of the bronchial images;

[0009] In a case where the airway features of the bronchial image do not match the airway features of the bronchial images at each past moment, determining a second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship;

[0010] Performing feature matching between the key frames of the bronchial images at the past moments and the bronchial image, and updating the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image if the feature matching is successful, and updating the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship if the airway features of the bronchial image and the airway features of the bronchial images at the past moments do not match;

[0011] A target matching pursuit result is determined based on the first matching pursuit result and the second matching pursuit result.

[0012] According to an endoscope branch positioning method based on airway detection and tracking provided by the present invention, the airway features include airway position features and airway appearance features;

[0013] Correspondingly, the determining of the first matching tracking result based on the airway features of the bronchial images at each moment in the past and the airway features of the bronchial images includes:

[0014] The first matching tracking result is determined based on the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial image.

[0015] According to an endoscope branch positioning method based on airway detection and tracking provided by the present invention, the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial images, determine the first matching tracking result, including:

[0016] The first matching tracking result is determined based on the IoU distance between the airway position features of the bronchial images at the past moments and the airway position features of the bronchial image, and the similarity between the airway appearance features of the bronchial images at the past moments and the airway appearance features of the bronchial image.

[0017] According to an endoscope branch positioning method based on airway detection and tracking provided by the present invention, the airway appearance feature is based on the output of the airway appearance network;

[0018] The airway appearance network is trained based on sample bronchial images and labeled bronchial classification results.

[0019] According to an endoscope branch positioning method based on airway detection and tracking provided by the present invention, the airway features of the bronchial images at various moments in the past and the airway features of the bronchial images are used to determine the first matching tracking result, and the method also includes:

[0020] The position of the airway of the bronchial image at each past moment at the current moment is predicted based on the Kalman filter algorithm, the predicted frame position of the airway in the bronchial image is obtained, and the predicted frame position is added to the camera motion compensation.

[0021] According to an endoscope branch positioning method based on airway detection and tracking provided by the present invention, the adding of the predicted frame position to the camera motion compensation comprises:

[0022] Calculating an image transformation relationship between the bronchial images of adjacent frames, and determining a minimization problem based on the image transformation relationship;

[0023] The minimization problem is solved to obtain image transformation parameters, and the image transformation parameters and the predicted box position are added to the camera motion compensation.

[0024] According to an endoscope branch positioning method based on airway detection and tracking provided by the present invention, the feature matching adopts any one of a fast feature point extraction and description algorithm, a size-invariant feature conversion algorithm, and a dense feature point matching algorithm based on deep learning.

[0025] The present invention also provides an endoscope branch positioning device based on airway detection and tracking, comprising:

[0026] An airway detection unit, used to obtain bronchial images at each moment, and perform airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images;

[0027] A first matching pursuit unit, configured to determine a first matching pursuit result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images;

[0028] A second matching pursuit unit is used to determine a second matching pursuit result based on a tracheal tree branch map and the airway hierarchical relationship when the airway feature of the bronchial image does not match the airway feature of the bronchial image at each past moment;

[0029] an updating unit, configured to perform feature matching between the key frames of the bronchial images at the past moments and the bronchial images, and to update the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images if the feature matching is successful, and to update the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship if the airway features of the bronchial images and the airway features of the bronchial images at the past moments do not match;

[0030] The target matching tracking result determining unit is configured to determine a target matching tracking result based on the first matching tracking result and the second matching tracking result.

[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an endoscope branch positioning method based on airway detection and tracking as described in any one of the above is implemented.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the endoscope branch positioning method based on airway detection and tracking as described in any one of the above is implemented.

[0033] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-described methods for endoscopic branch positioning based on airway detection and tracking.

[0034] The endoscope branch positioning method, device and equipment based on airway detection and tracking provided by the present invention obtain bronchial images at various times, perform airway detection on the bronchial images, obtain the airway hierarchical relationship of the bronchial images, determine the first matching tracking result based on the airway features of the bronchial images at various times in the past and the airway features of the bronchial images, and in the case where the airway features of the bronchial images do not match the airway features of the bronchial images at various times in the past, determine the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship, and finally, determine the second matching tracking result based on the first matching tracking result and the second matching tracking result. Second, matching tracking results are used to determine the target matching tracking results. During the matching tracking process, the new detection results in each frame of the image are matched with all historical tracking results, thereby avoiding the error accumulation problem that may occur in the incremental positioning method, and the error can be automatically recovered after an error occurs, thereby improving the accuracy and reliability of endoscope-based branch positioning; moreover, multi-level airways can be observed for each frame of the image detection, and the forward and backward detection of the endoscope is embedded in the process of airway tracking and branch labeling. Therefore, no additional camera motion information input is required in the minimum system, which broadens the application scope of endoscope-based branch positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is one of the flow diagrams of the endoscope branch positioning method based on airway detection and tracking provided by the present invention;

[0037] Figure 2 This is the second flow chart of the endoscope branch positioning method based on airway detection and tracking provided by the present invention;

[0038] Figure 3 It is a structural schematic diagram of an endoscope branch positioning device based on airway detection and tracking provided by the present invention;

[0039] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In the related art, vision-based endoscope positioning methods are usually divided into two categories: coordinate-level positioning and branch positioning.

[0042] Most studies focus on coordinate-level positioning, that is, obtaining the coordinates of the endoscope in the bronchial coordinate system. Specific research directions include:

[0043] 1) SFM (Structure From Motion) method, first extract the feature points of the endoscope image, match the feature points one by one for two adjacent frames, and solve PnP (Perspective-n-Point) based on this, so as to estimate the camera pose. In addition, the camera pose estimation results can be corrected and optimized by Bundle Adjustment and RANSAC (RANdom SAmple Consensus) methods.

[0044] 2) 2D / 3D registration: The most common method in related technologies is to register the 2D image captured by the endoscope to the preoperative reconstructed model, so as to obtain the position of the camera in the model. To perform 2D / 3D registration, a virtual endoscope must be designed, that is, a virtual camera captures a virtual image in the reconstructed model, and the image registration is completed by comparing the virtual image with the real image.

[0045] There are also studies aimed at airway-level positioning of the endoscope, that is, only to find out which branch of the tracheal tree the endoscope is in. Past studies include:

[0046] 1) Through the depth estimation of the endoscope image, the depth peak point is regarded as the airway hole. Combined with the control signals such as the forward and backward movement of the endoscope, the number of layers of the endoscope in the airway is obtained (not yet accurate to the branch).

[0047] 2) The airway center is extracted through the multi-scale MSER (MSER-Maximally Stable Extremal Regions) method, and the airways are clustered to obtain the next level of airways that can be observed in each frame of the image. The detected tracheal center is matched to the center line of the tracheal tree reconstructed according to CT before surgery, so as to determine which branch of the airway the endoscope is in.

[0048] 3) Through airway detection in multiple images, the hierarchy is determined according to the overlapping relationship of the airways detected between frames and mapped to the preoperative tracheal tree.

[0049] In summary, an ideal endoscope visual positioning system needs to meet the requirements of robustness, real-time performance, continuous positioning, and occupy as little computing power and storage space as possible, while existing methods have defects in different aspects.

[0050] Based on the above problems, the present invention provides an endoscope branch positioning method based on airway detection and tracking. Figure 1 It is one of the flow charts of the endoscope branch positioning method based on airway detection and tracking provided by the present invention, Figure 2 FIG. 2 is a flow chart of the endoscope branch positioning method based on airway detection and tracking provided by the present invention. Figure 1 , Figure 2 As shown, the method includes:

[0051] Step 110, acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images.

[0052] Specifically, bronchial images at various moments may be acquired, and the bronchial images here may be acquired based on an endoscope.

[0053] After acquiring the bronchial images at each time, airway detection may be performed on the bronchial images to obtain the airway hierarchical relationship of the bronchial images.

[0054] Here, airway detection of bronchial images can use a target detection method based on deep learning or traditional computer vision. Here, the deep learning target detection network can be a Faster R-CNN network, or a YOLO network series, etc. The traditional computer vision method can be MSER (MSER-Maximally Stable Extremal Regions), etc. The embodiment of the present invention does not make specific limitations on this.

[0055] The airway hierarchical relationship here reflects the connection relationship between the airways at the hierarchical level in the bronchial image.

[0056] Here, the method for determining the hierarchical relationship of the airways can use the yolov7 target detection network to detect a target detection frame, the target detection frame selects all levels of airways in the bronchial image, and determines the hierarchical relationship of the airways inside the bronchial image through the inclusion relationship between the detection frames.

[0057] It is understandable that, unlike previous studies, in the airway detection process, the embodiment of the present invention not only estimates the position of the next-level airway, but also detects the airways that can be observed at all levels, and clusters the detected airways according to the position and inclusion relationship of the detection frame to obtain the local tracheal tree of each frame of the endoscopic image. For example, when the detection frame a includes the detection frame b, the detection frame b is recorded as the next-level branch of the detection frame a.

[0058] Here, the method for determining the inclusion relationship between airways can be to select the airway with the highest branching level in the image with a target detection box, or to detect the central position of the airway with the highest branching level, and determine the hierarchical relationship of the airways inside the image through methods such as the position, appearance or high-dimensional features of the detected airways.

[0059] Because the endoscope can always observe the next branch in advance before entering the next branch, and the endoscope can always observe the current branch when it retracts to the previous branch, based on such a detection strategy, the forward and backward movements of the endoscope no longer require additional input of endoscope motion or control signals, but are naturally included in the airway detection and tracking process. In addition, the forward and backward directions of the airway can be estimated by image optical flow detection, and the forward and backward directions estimated by the detection and tracking process can be verified. Methods such as robotic arm control and operator gesture recognition can also be used as algorithm inputs to increase the reliability of motion direction estimation.

[0060] It is understandable that multiple levels of airways can be observed for each frame of image detection, and the detection of the endoscope's forward and backward movement is embedded in the process of airway tracking and branch annotation, so no additional camera motion information input is required in the minimum system. Supplementary visual modules: optical flow calculation; multimodal information: robot control signals, operator posture recognition can improve the accuracy and reliability of the algorithm.

[0061] Step 120 , determining a first matching tracking result based on the airway features of the bronchial images at each moment in the past and the airway features of the bronchial images.

[0062] Specifically, bronchial images at various moments in the past may be acquired, where the bronchial images at various moments in the past refer to bronchial images at past moments corresponding to each moment, for example, may be bronchial images of the first 300 frames of the current moment, and the embodiment of the present invention does not specifically limit this.

[0063] The first matching tracking result can be determined based on the airway features of the bronchial images at various moments in the past and the airway features of the bronchial images. The airway features here can be obtained by performing airway detection on the bronchial images, and the airway features can include airway position features and airway appearance features.

[0064] For example, the airway features of the bronchial images at various moments in the past and the airway features of the bronchial images can be compared, that is, the association between the airways detected in the past and the airways detected at this moment can be determined, that is, which airways detected at each moment belong to the same airway. Through the association between the airways detected in the past and the airways detected at this moment, it is determined to which branch of the airway tree the airway detected at this moment can be mapped with the airways associated with the past detections.

[0065] Step 130, when the airway features of the bronchial image do not match the airway features of the bronchial images at each time in the past, determining a second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship;

[0066] Step 140, feature matching is performed between the key frames of the bronchial images at the past moments and the bronchial image. If the feature matching is successful, the first matching tracking result is updated based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image. If the airway features of the bronchial image do not match the airway features of the bronchial images at the past moments, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchical relationship.

[0067] Step 150: determine a target matching pursuit result based on the first matching pursuit result and the second matching pursuit result.

[0068] Specifically, when the airway features of the bronchial image do not match the airway features of the bronchial images at each time in the past, the second matching tracking result can be determined based on the tracheal tree branch map and the airway hierarchical relationship. The tracheal tree branch map here is obtained by skeletonizing the preoperative CT (Computed Tomography) image.

[0069] Among them, when the number of airways detected in a certain layer is less than the number of airways in the tracheal tree branch map, the correspondence between the detected airways and the bronchial tree branches in the tracheal tree branch map is expressed in the form of a probability function, and this is used as the basis for judging the mapping relationship. The factors included in the probability function include: airway angle, airway depth relative to the camera, and the intersection position of the airway with the previous level, etc., which are not specifically limited in the embodiment of the present invention.

[0070] In order to improve the positioning stability and reduce the positioning failure caused by matching errors, a loop detection step can be added after the annotation completion process. That is, the key frames of the bronchial images at various moments in the past are feature matched with the bronchial images. When the feature matching is successful, the first matching tracking result is updated based on the airway features of the bronchial images at various moments in the past and the airway features of the bronchial images. When the airway features of the bronchial images do not match the airway features of the bronchial images at various moments in the past, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchy relationship, thereby restoring the correct positioning. Here, the feature matching adopts any one of ORB (Oriented FAST and RotatedBRIEF, fast feature point extraction and description algorithm), SIFT (Scale-invariant feature transform, size invariant feature transformation algorithm) and dense feature point matching algorithm based on deep learning, and the embodiment of the present invention does not specifically limit this.

[0071] Here, the dense feature point matching algorithm based on deep learning may be LoFTR (Local Feature TRansformer) or the like, and the embodiment of the present invention does not specifically limit this.

[0072] Finally, a target matching tracking result can be determined based on the first matching tracking result and the second matching tracking result. The target matching tracking result here reflects the matching relationship between the airway of the bronchial image at each moment and the airway of the bronchial image at each moment in the past.

[0073] The method provided by the embodiment of the present invention obtains bronchial images at various moments, performs airway detection on the bronchial images, obtains the airway hierarchical relationship of the bronchial images, determines a first matching tracking result based on the airway features of the bronchial images at various moments in the past and the airway features of the bronchial images, and determines a second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship when the airway features of the bronchial images do not match the airway features of the bronchial images at various moments in the past. Finally, the target matching tracking result is determined based on the first matching tracking result and the second matching tracking result. During the matching tracking process, the new detection result in each frame of the image matches all the historical tracking results, thereby avoiding the error accumulation problem that may occur in the incremental positioning method, and can automatically recover after an error occurs, thereby improving the accuracy and reliability of endoscope-based branch positioning; and, in addition, multi-level airways can be observed for each frame of the image detection, and the forward and backward detection of the endoscope is embedded in the process of airway tracking and branch labeling, so that no additional camera motion information input is required in the minimum system, thereby broadening the application scope of endoscope-based branch positioning.

[0074] The advantages of the method provided by the embodiment of the present invention are:

[0075] 1. The main algorithm is based on a deep learning method for real-time operation, with fast calculation speed;

[0076] 2. Compared with the existing hierarchical positioning method, multi-level airway detection and tracking embeds the forward and backward movements of the endoscope; supplementary visual modules such as optical flow calculation can be added to further improve positioning reliability; a large number of unrealistic assumptions are reduced, and it can cope with complex situations such as the airway at the edge of the field of view or not observed, uneven lighting, and multi-branch airways, meeting actual use requirements.

[0077] 3. Information from other modalities, such as robot control signals and operator hand posture recognition, can be used as auxiliary inputs to improve the reliability of the algorithm in estimating camera motion.

[0078] 4. Perform airway detection on the bronchial image to obtain the airway hierarchy relationship of the bronchial image. Therefore, there is no obvious difference in the error between the low-level airway and the high-level airway.

[0079] Based on the above embodiment, the airway features include airway position features and airway appearance features;

[0080] Correspondingly, step 120 includes:

[0081] Step 121, determining the first matching tracking result based on the airway position features and airway appearance features of the bronchial images at each moment in the past, and the airway position features and airway appearance features of the bronchial images.

[0082] Specifically, the airway features may include airway position features and airway appearance features, wherein the airway position features reflect feature information at the airway position level, and the airway appearance features reflect feature information at the airway appearance level.

[0083] Correspondingly, the first matching tracking result may be determined based on the airway position features and airway appearance features of the bronchial images at various moments in the past, and the airway position features and airway appearance features of the bronchial images.

[0084] For example, the airway position features of the bronchial images at various past moments may be compared with the airway position features of the bronchial images, and the airway appearance features of the bronchial images at various past moments may be compared with the airway appearance features of the bronchial images, thereby determining the first matching tracking result.

[0085] Based on the above embodiment, step 121 includes:

[0086] Step 1211, based on the IoU distance between the airway position features of the bronchial images at the past moments and the airway position features of the bronchial images, and the similarity between the airway appearance features of the bronchial images at the past moments and the airway appearance features of the bronchial images, determine the first matching tracking result.

[0087] Specifically, the first matching tracking result can be determined based on the IoU distance between the airway position features of the bronchial images at each past moment and the airway position features of the bronchial images, and the similarity between the airway appearance features of the bronchial images at each past moment and the airway appearance features of the bronchial images.

[0088] For detection boxes A and B, their IoU represents the ratio of the intersection to the union of the two detection boxes, that is:

[0089]

[0090] The formula for IoU distance is:

[0091]

[0092] Here, the similarity between the airway appearance features of the bronchial images at various moments in the past and the airway appearance features of the bronchial images can be calculated using methods such as Re-ID, cosine similarity, and Pearson correlation coefficient, and the embodiment of the present invention does not specifically limit this.

[0093] Based on the above embodiment, the airway appearance feature is based on the output of the airway appearance network;

[0094] The airway appearance network is trained based on sample bronchial images and labeled bronchial classification results.

[0095] Specifically, the airway appearance feature is based on the output of the airway appearance network. The airway appearance network can be a Re-ID network, or HRNet, or various deep neural networks such as Transformer and ResNet, and the embodiment of the present invention does not make specific limitations on this.

[0096] The airway appearance network here is trained based on sample bronchial images and labeled bronchial classification results.

[0097] That is, sample bronchial images and labeled bronchial classification results can be collected in advance, and an initial airway appearance network can also be pre-built.

[0098] After obtaining the initial airway appearance network, the pre-collected sample bronchial images and labeled bronchial classification results can be used to train the initial airway appearance network:

[0099] First, the sample bronchial image is input into the initial airway appearance network, which obtains and outputs the bronchial classification prediction result.

[0100] After obtaining the bronchial classification prediction result, the bronchial classification prediction result can be compared with the labeled bronchial classification result, and the loss function value can be calculated according to the degree of difference between the two. The parameters of the initial airway appearance network are iterated based on the loss function value, and the initial airway appearance network after the parameter iteration is recorded as the airway appearance network.

[0101] It can be understood that the greater the difference between the bronchial classification prediction result and the labeled bronchial classification result, the greater the loss function value; the smaller the difference between the bronchial classification prediction result and the labeled bronchial classification result, the smaller the loss function value.

[0102] The airway appearance feature is a high-dimensional vector (descriptor) generated by the deep neural network based on the input image. For each frame of detection results, the part within the detection frame is cropped and input into the Re-ID network to obtain the feature descriptor of the target. For two detection frame images, the cosine distance of the corresponding descriptors can be used to calculate the similarity of the contents of the two detection frames.

[0103] In the training of the airway appearance network, the problem is converted into a classification problem, aiming to train a network to classify the same target into one category. During the test, the last fully connected layer of the network is removed, and only the convolution part is retained, and the output is the descriptor high-dimensional vector. In the embodiment of the present invention, the appearance airway network specifically adopts the ResNet50 network, and the network weights trained on the pedestrian Re-ID dataset are fine-tuned using manually annotated sample bronchial images.

[0104] Based on the above embodiment, step 120, before step 120, further includes:

[0105] Step 121, predicting the position of the airway of the bronchial image at each past moment at the current moment based on the Kalman filter algorithm, obtaining the predicted frame position of the airway in the bronchial image, and adding the predicted frame position to the camera motion compensation.

[0106] Specifically, the position of the airway of the bronchial image at each moment in the past at the current moment can be predicted based on the Kalman filter algorithm, the predicted frame position of the airway in the bronchial image can be obtained, and the predicted frame position can be added to the camera motion compensation.

[0107] In addition, the confidence can also be calculated in the airway detection stage. The detection frame with low confidence may contain the result of detection error or may contain the obscured target. Therefore, in the embodiment of the present invention, not only the detection frame with high confidence is retained, but the detection result is filtered once with very weak NMS (Non-Maximum Suppression), and the remaining detection frames are divided into two groups of high confidence and low confidence.

[0108] Then, the high-confidence detections are matched with the tracking records of the previous frame. The similarity criteria used for matching are IoU distance and Re-ID descriptor similarity, and the matching method is the Hungarian algorithm.

[0109] Then, the high-confidence detections and low-confidence detections that have not been successfully matched are matched with all past tracking records. The similarity calculation method used for matching is IoU distance, and the matching method is Hungarian algorithm.

[0110] Then update the Kalman filter parameters based on all matching detections.

[0111] Finally, if the confidence of the remaining unmatched detection is higher than the set threshold, the detection is added as a new tracking record.

[0112] Based on the above embodiment, the step of adding the predicted frame position to the camera motion compensation in step 121 includes:

[0113] Step 1211, calculating the image transformation relationship between the bronchial images of adjacent frames, and determining a minimization problem based on the image transformation relationship;

[0114] Step 1212, solving the minimization problem to obtain image transformation parameters, and adding the image transformation parameters and the predicted box position to the camera motion compensation.

[0115] Specifically, the rapid movement of the endoscope will also cause the airway position in the image to move rapidly. The important assumption of matching the detection frame on the time sequence in target tracking is that the position intersection of the target is large enough. The movement of the camera will seriously affect the correctness of this assumption. In order to compensate for the camera motion, the ECC (Enhanced Correlation Coefficient Maximization) method is used to approximately calculate the image transformation relationship between the bronchial images of adjacent frames, and based on the image transformation relationship, the minimization problem is determined, that is, the solution to the minimization problem is:

[0116]

[0117] Among them, ‖·‖ represents the Euclidean norm, p is the image transformation parameter, i r and i w (p) are the endoscopic image and the transformed endoscopic image, respectively. and is the grayscale normalized i r and i w (p).

[0118] After solving the minimization problem, the image transformation parameters are obtained, and then the image transformation parameters and the predicted box position are added to the camera motion compensation.

[0119] The image transformation relationship may include any one of Euclidean transformation, affine transformation and homography transformation.

[0120] Here, the image transformation may be a Euclidean Transformation for example, and an affine transformation or a homographic transformation may also be used.

[0121] For a point on an image:

[0122]

[0123] The Euclidean transformation matrix is:

[0124]

[0125] in, is the rotation matrix. is the translation vector.

[0126] From P to P ′ The coordinate transformation is:

[0127]

[0128] Additionally, camera motion supplements alternatives:

[0129] The image transformation can be a homography transformation, then for a point on the image:

[0130]

[0131] The homography transformation matrix is:

[0132]

[0133] P is transformed to P through homography ′ After that, its image coordinates are:

[0134]

[0135]

[0136] At this time, the calculation of H needs to find the matching point pairs of the two frames of images through the feature matching method, and solve it by minimizing the reprojection error through the least squares method:

[0137]

[0138] in,

[0139] In addition, as the endoscope moves, new airway targets will continue to appear in the bronchial image. Due to the violent movement of the camera and poor lighting, detection errors will also occur in the bronchial image. Therefore, it is necessary to match the new detection targets with the branches on the tracheal tree branch map and correct the wrong detection tracking.

[0140] For a new frame of bronchial image, the airway matching tracking algorithm is used to obtain a possibly incomplete tracheal annotation result. The annotation completion process is as follows:

[0141] a) If the number of annotated detection and tracking is greater than 1, the camera roll angle of the current frame is estimated based on the center position of these detection frames and the position in the past frame, i.e., the camera roll angle; if the number of detection and tracking is less than or equal to 1, the camera roll angle of the current frame needs to be calculated based on feature point matching and camera intrinsic parameters. At this time, the camera roll angle is calculated by the image rotation component in camera motion compensation (the rotation matrix A is used for rotation and translation transformation, and the rotation component needs to be further decomposed for homography transformation).

[0142] b) Sort the annotated tracking detection frames from high to low according to the number of tracked frames and the detection confidence, and then begin to annotate and correct the detection frames associated with them in order.

[0143] c) If there are unlabeled detection frames in the next level, match them to the tracheal tree map based on the center position of each detection frame at the next level. If the detected next-level branches are less than the number of next-level branches in the map, calculate the probability of observing each airway based on the angle between the current branch and each branch at the next level. In the embodiment of the present invention, the smaller the angle with the current branch, the greater the probability of the next-level branch being observed. Then make a cross-section of the next-level branch on the tracheal tree branch map, rotate each intersection according to the current camera roll angle, and obtain the position of each branch in the current field of view (two-dimensional local tracheal tree map). After normalizing the local tracheal tree map and the center coordinates of the current frame detection frame, the Hungarian algorithm matches the detection frame and the local map according to the Euclidean distance metric to obtain the branch annotations of each detection frame.

[0144] d) If there are unlabeled detection boxes at the same level, match them to the bronchial branch map at the previous level as in c).

[0145] e) If there are unlabeled detection boxes in the previous level, since the previous level branch is unique, the previous level branch number is directly assigned to the unlabeled detection box.

[0146] Here, the camera roll angle can be calculated based on the SFM method using only feature matching and camera intrinsic parameters. For a robot-controlled endoscope, the camera roll angle can be referenced by the control signal input; for a handheld endoscope, the camera roll angle can be estimated based on the operator's gesture recognition.

[0147] The following is a description of the endoscope branch positioning device based on airway detection and tracking provided by the present invention. The endoscope branch positioning device based on airway detection and tracking described below and the endoscope branch positioning method based on airway detection and tracking described above can be referenced to each other.

[0148] Based on any of the above embodiments, the present invention provides an endoscope branch positioning device based on airway detection and tracking, Figure 3 FIG. 1 is a schematic diagram of the structure of an endoscope branch positioning device based on airway detection and tracking provided by the present invention. Figure 3 As shown, the device comprises:

[0149] An airway detection unit 310 is used to obtain a bronchial image at each moment, and perform airway detection on the bronchial image to obtain an airway hierarchical relationship of the bronchial image;

[0150] A first matching pursuit unit 320, configured to determine a first matching pursuit result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images;

[0151] A second matching pursuit unit 330 is used to determine a second matching pursuit result based on a tracheal tree branch map and the airway hierarchical relationship when the airway feature of the bronchial image does not match the airway feature of the bronchial image at each past moment;

[0152] An updating unit 340 is used to perform feature matching between the key frames of the bronchial images at the past moments and the bronchial images, and if the feature matching is successful, update the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images, and if the airway features of the bronchial images and the airway features of the bronchial images at the past moments do not match, update the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship;

[0153] The target matching tracking result determining unit 350 is configured to determine a target matching tracking result based on the first matching tracking result and the second matching tracking result.

[0154] The device provided by the embodiment of the present invention obtains bronchial images at various moments, performs airway detection on the bronchial images, obtains the airway hierarchical relationship of the bronchial images, determines a first matching tracking result based on the airway features of the bronchial images at various moments in the past and the airway features of the bronchial images, and determines a second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship when the airway features of the bronchial images do not match the airway features of the bronchial images at various moments in the past. Finally, the target matching tracking result is determined based on the first matching tracking result and the second matching tracking result. During the matching tracking process, the new detection result in each frame of the image matches all the historical tracking results, thereby avoiding the error accumulation problem that may occur in the incremental positioning method, and can automatically recover after an error occurs, thereby improving the accuracy and reliability of endoscope-based branch positioning; and, in addition, multi-level airways can be observed for each frame of the image detection, and the forward and backward detection of the endoscope is embedded in the process of airway tracking and branch labeling, so that no additional camera motion information input is required in the minimum system, thereby broadening the application scope of endoscope-based branch positioning.

[0155] Based on any of the above embodiments, the airway characteristics include airway position characteristics and airway appearance characteristics;

[0156] Correspondingly, the first matching tracking unit 320 specifically includes:

[0157] The first matching tracking subunit is used to determine the first matching tracking result based on the airway position characteristics and airway appearance characteristics of the bronchial images at each past moment, and the airway position characteristics and airway appearance characteristics of the bronchial image.

[0158] Based on any of the foregoing embodiments, the first matching tracking subunit is specifically configured to:

[0159] The first matching tracking result is determined based on the IoU distance between the airway position features of the bronchial images at the past moments and the airway position features of the bronchial image, and the similarity between the airway appearance features of the bronchial images at the past moments and the airway appearance features of the bronchial image.

[0160] Based on any of the above embodiments, the airway appearance feature is based on the output of the airway appearance network;

[0161] The airway appearance network is trained based on sample bronchial images and labeled bronchial classification results.

[0162] Based on any of the above embodiments, it further includes a camera motion compensation unit, and the camera motion compensation unit specifically includes:

[0163] A prediction unit is used to predict the position of the airway of the bronchial image at each past moment at the current moment based on a Kalman filter algorithm, obtain the predicted frame position of the airway in the bronchial image, and add the predicted frame position to the camera motion compensation.

[0164] Based on any of the above embodiments, the prediction unit is specifically used for:

[0165] Calculating an image transformation relationship between the bronchial images of adjacent frames, and determining a minimization problem based on the image transformation relationship;

[0166] The minimization problem is solved to obtain image transformation parameters, and the image transformation parameters and the predicted box position are added to the camera motion compensation.

[0167] Based on any of the above embodiments, the feature matching adopts any one of a fast feature point extraction and description algorithm, a size-invariant feature conversion algorithm, and a dense feature point matching algorithm based on deep learning.

[0168] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute an endoscopic branch positioning method based on airway detection and tracking, the method comprising: acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; determining a first matching tracking result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images; and determining a second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship when the airway features of the bronchial images and the airway features of the bronchial images at each past moment do not match. Results; feature matching is performed between the key frames of the bronchial images at the past moments and the bronchial images, and when the feature matching is successful, the first matching tracking result is updated based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images, and when the airway features of the bronchial images and the airway features of the bronchial images at the past moments do not match, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchy relationship; based on the first matching tracking result and the second matching tracking result, the target matching tracking result is determined.

[0169] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0170] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the endoscopic branch positioning method based on airway detection and tracking provided by the above-mentioned methods, the method including: acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images, determining a first matching tracking result; when the airway features of the bronchial image and the airway features of the bronchial images at each past moment do not match When the airway features of the bronchial images at the past moments do not match, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchical relationship; the key frames of the bronchial images at the past moments are feature matched with the bronchial images, and when the feature matching is successful, the first matching tracking result is updated based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images; when the airway features of the bronchial images do not match the airway features of the bronchial images at the past moments, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchical relationship; and the target matching tracking result is determined based on the first matching tracking result and the second matching tracking result.

[0171] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the endoscopic branch positioning method based on airway detection and tracking provided by the above-mentioned methods, the method comprising: acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images, determining a first matching tracking result; in the case where the airway features of the bronchial image and the airway features of the bronchial images at each past moment do not match, based on the tracheal tree branch map and the airway hierarchy relationship, determine the second matching tracking result; perform feature matching on the key frames of the bronchial images at the past moments and the bronchial image, and if the feature matching is successful, re-update the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image, and if the airway features of the bronchial image and the airway features of the bronchial images at the past moments do not match, update the second matching tracking result based on the tracheal tree branch map and the airway hierarchy relationship; determine the target matching tracking result based on the first matching tracking result and the second matching tracking result.

[0172] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An endoscope branch positioning method based on airway detection and tracking, characterized in that: include: Acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; Determining a first matching pursuit result based on the airway features of the bronchial images at each moment in the past and the airway features of the bronchial images; In a case where the airway features of the bronchial image do not match the airway features of the bronchial images at each past moment, determining a second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship; Performing feature matching between the key frames of the bronchial images at the past moments and the bronchial image, and updating the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image if the feature matching is successful, and updating the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship if the airway features of the bronchial image and the airway features of the bronchial images at the past moments do not match; A target matching pursuit result is determined based on the first matching pursuit result and the second matching pursuit result.

2. The endoscope branch positioning method based on airway detection and tracking according to claim 1 is characterized in that: The airway characteristics include airway position characteristics and airway appearance characteristics; Correspondingly, the determining of the first matching tracking result based on the airway features of the bronchial images at each moment in the past and the airway features of the bronchial images includes: The first matching tracking result is determined based on the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial image.

3. The endoscope branch positioning method based on airway detection and tracking according to claim 2 is characterized in that: The determining the first matching tracking result based on the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial images, comprises: The first matching tracking result is determined based on the IoU distance between the airway position features of the bronchial images at the past moments and the airway position features of the bronchial image, and the similarity between the airway appearance features of the bronchial images at the past moments and the airway appearance features of the bronchial image.

4. The endoscope branch positioning method based on airway detection and tracking according to claim 2 is characterized in that: The airway appearance feature is based on the output of the airway appearance network; The airway appearance network is trained based on sample bronchial images and labeled bronchial classification results.

5. The endoscope branch positioning method based on airway detection and tracking according to claim 1 is characterized in that: The airway features of the bronchial images at various moments in the past and the airway features of the bronchial images are used to determine the first matching tracking result, and the method also includes: The position of the airway of the bronchial image at each past moment at the current moment is predicted based on the Kalman filter algorithm, the predicted frame position of the airway in the bronchial image is obtained, and the predicted frame position is added to the camera motion compensation.

6. The endoscope branch positioning method based on airway detection and tracking according to claim 5 is characterized in that: The adding the predicted frame position to the camera motion compensation comprises: Calculating an image transformation relationship between the bronchial images of adjacent frames, and determining a minimization problem based on the image transformation relationship; The minimization problem is solved to obtain image transformation parameters, and the image transformation parameters and the predicted box position are added to the camera motion compensation.

7. The endoscope branch positioning method based on airway detection and tracking according to any one of claims 1 to 6, characterized in that: The feature matching adopts any one of a fast feature point extraction and description algorithm, a size-invariant feature conversion algorithm, and a dense feature point matching algorithm based on deep learning.

8. An endoscope branch positioning device based on airway detection and tracking, characterized in that: include: An airway detection unit, used to obtain bronchial images at each moment, and perform airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; a first matching pursuit unit, configured to determine a first matching pursuit result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images; A second matching pursuit unit is used to determine a second matching pursuit result based on a tracheal tree branch map and the airway hierarchical relationship when the airway feature of the bronchial image does not match the airway feature of the bronchial image at each past moment; an updating unit, configured to perform feature matching between the key frames of the bronchial images at the past moments and the bronchial images, and to update the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images if the feature matching is successful, and to update the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship if the airway features of the bronchial images and the airway features of the bronchial images at the past moments do not match; The target matching tracking result determining unit is configured to determine a target matching tracking result based on the first matching tracking result and the second matching tracking result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the endoscope branch positioning method based on airway detection and tracking as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the endoscope branch positioning method based on airway detection and tracking as described in any one of claims 1 to 7 is implemented.

Images