Endoscope branch positioning method and apparatus based on airway detection and tracking, and device

Abstract

Provided in the present application are an endoscope branch positioning method and apparatus based on airway detection and tracking, and a device. The method comprises: acquiring a bronchial image at each moment, and performing airway detection on the bronchial image, so as to obtain an airway hierarchy relationship of the bronchial images; determining a first matching pursuit result on the basis of an airway feature of a bronchial image at each moment in the past and an airway feature of the bronchial image; when the airway feature of the bronchial image does not match the airway feature of the bronchial image at each moment in the past, determining a second matching pursuit result on the basis of a tracheal-tree branch map and the airway hierarchy relationship; and determining a target matching pursuit result on the basis of the first matching pursuit result and the second matching pursuit result, wherein during matching pursuit, a new detection result in each image frame matches all historical pursuit results, and automatic recovery is implemented after an error occurs.

Description

Endoscope branch positioning method, device and equipment based on airway detection and tracking

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese invention patent application number 202311039499.2, filed on August 17, 2023, entitled “Diversion communication method, device, equipment and readable storage medium”, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to the field of artificial intelligence technology, and in particular to an endoscope branch positioning method, device and equipment based on airway detection and tracking. Background Art

[0004] Vision-based endoscope positioning methods are generally divided into two categories: coordinate-level positioning and branch-level positioning.

[0005] Although coordinate-level localization can provide more specific location information, due to the gradually decreasing diameter of tracheal branches at higher levels, the algorithm's millimeter-level errors can lead to increasingly poor localization results as the endoscope penetrates deeper. Since algorithms that achieve accurate coordinates typically employ incremental strategies, errors can also accumulate over time. Specifically, existing coordinate-level localization methods also suffer from the following shortcomings: 1) While the SFM (Structure From Motion) method has a strong theoretical foundation, endoscopic images, unlike open-field images, often lack feature points. When feature points are too few, the PnP (Perspective-n-Point) algorithm cannot be solved, resulting in localization loss. 2) Most 2D / 3D registration methods rely on iterative optimization algorithms, which means that obtaining localization for each frame requires a long computational time. The pose of the endoscope changes rapidly during actual examinations, leading to localization loss. Other methods rely on deep learning retrieval, requiring the acquisition of virtual endoscopic images for each user to train the retrieval network. This consumes significant computing power and storage space, and requires extensive preoperative training and preparation, making it difficult to implement in practice.

[0006] Existing branch-level localization methods are based on too many assumptions (such as stable camera motion, the airway always appearing in the center of the field of view, the bifurcated structure of the airway, uniform lighting conditions, etc.), and are based on simple airway detection and tracking strategies (traditional detection and tracking algorithms, only detecting the next level of airways). They need to combine control signals to obtain the approximate direction of camera movement. Therefore, they have only been verified in simulation environments or with small data sets, making them difficult to apply in practice.

[0007] Summary of the Invention

[0008] The present application provides an endoscope branch positioning method, device and equipment based on airway detection and tracking, which is used to solve the defects of endoscope positioning based on coordinate-level positioning and branch positioning in the prior art.

[0009] The present application provides an endoscope branch positioning method based on airway detection and tracking, comprising:

[0010] Acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain an airway hierarchical relationship of the bronchial images;

[0011] determining a first matching pursuit result based on airway features of bronchial images at various past moments and the airway features of the bronchial images;

[0012] In a case where the airway features of the bronchial image do not match the airway features of the bronchial images at each past moment, determining a second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship;

[0013] performing feature matching on the key frames of the bronchial images at the past moments with the bronchial image, and if the feature matching is successful, updating the first matching pursuit result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image; and updating the second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship if the airway features of the bronchial image do not match the airway features of the bronchial images at the past moments;

[0014] A target matching pursuit result is determined based on the first matching pursuit result and the second matching pursuit result.

[0015] According to an endoscope branch positioning method based on airway detection and tracking provided by the present application, the airway features include airway position features and airway appearance features;

[0016] Correspondingly, determining the first matching pursuit result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images includes:

[0017] The first matching pursuit result is determined based on the airway position features and airway appearance features of the bronchial images at each past moment and the airway position features and airway appearance features of the bronchial image.

[0018] According to an endoscope branch positioning method based on airway detection and tracking provided by the present application, determining the first matching tracking result based on the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial images, includes:

[0019] The first matching tracking result is determined based on the intersection relationship between the airway position features of the bronchial images at each past moment and the airway position features of the bronchial image, and the similarity between the airway appearance features of the bronchial images at each past moment and the airway appearance features of the bronchial image.

[0020] According to an endoscope branch positioning method based on airway detection and tracking provided by the present application, the airway appearance feature is based on the output of the airway appearance network;

[0021] The airway appearance network is trained based on sample bronchial images and labeled bronchial classification results.

[0022] According to an endoscope branch positioning method based on airway detection and tracking provided by the present application, the airway features of bronchial images at various past moments and the airway features of the bronchial images are used to determine a first matching tracking result, and the method also includes:

[0023] The position of the airway in the bronchial image at each past moment at the current moment is predicted based on the model, the predicted frame position of the airway in the bronchial image is obtained, and the predicted frame position is added to the camera motion compensation.

[0024] According to an endoscope branch positioning method based on airway detection and tracking provided by the present application, the adding of the predicted frame position to the camera motion compensation includes:

[0025] calculating an image transformation relationship between the bronchial images of adjacent frames, and determining a minimization problem based on the image transformation relationship;

[0026] Solve the minimization problem to obtain image transformation parameters, and add the image transformation parameters and the predicted box position to the camera motion compensation.

[0027] According to an endoscope branch positioning method based on airway detection and tracking provided in the present application, the feature matching adopts any one of a fast feature point extraction and description algorithm, a size-invariant feature conversion algorithm, and a dense feature point matching algorithm based on deep learning.

[0028] The present application also provides an endoscope branch positioning device based on airway detection and tracking, comprising:

[0029] an airway detection unit, configured to acquire bronchial images at each moment, and perform airway detection on the bronchial images to obtain an airway hierarchical relationship of the bronchial images;

[0030] a first matching pursuit unit, configured to determine a first matching pursuit result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images;

[0031] a second matching pursuit unit, configured to determine a second matching pursuit result based on a tracheal tree branch map and the airway hierarchical relationship when the airway features of the bronchial image do not match the airway features of the bronchial images at each past moment;

[0032] an updating unit, configured to perform feature matching between the key frames of the bronchial images at the past moments and the bronchial image, and, if the feature matching is successful, re-update the first matching pursuit result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image; and, if the airway features of the bronchial image do not match the airway features of the bronchial images at the past moments, update the second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship;

[0033] The target matching pursuit result determining unit is configured to determine a target matching pursuit result based on the first matching pursuit result and the second matching pursuit result.

[0034] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the endoscope branch positioning method based on airway detection and tracking as described above is implemented.

[0035] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for endoscopic branch positioning based on airway detection and tracking as described above is implemented.

[0036] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described methods for endoscopic branch positioning based on airway detection and tracking.

[0037] The present application provides an endoscope branch positioning method, device and apparatus based on airway detection and tracking, which obtains bronchial images at each moment, performs airway detection on the bronchial images, obtains the airway hierarchical relationship of the bronchial images, and determines a first matching tracking result based on the airway features of the bronchial images at each moment in the past and the airway features of the bronchial images. In the case that the airway features of the bronchial images do not match the airway features of the bronchial images at each moment in the past, a second matching tracking result is determined based on the tracheal tree branch map and the airway hierarchical relationship. Finally, based on the first matching tracking result and the second matching tracking result, the airway hierarchical relationship is determined. Second, matching tracking results are used to determine the target matching tracking results. During the matching tracking process, the new detection results in each frame of the image are matched with all historical tracking results, thereby avoiding the error accumulation problem that may occur in the incremental positioning method, and the system can automatically recover after an error occurs, thereby improving the accuracy and reliability of endoscope-based branch positioning; moreover, multi-level airways can be observed in each frame of the image detection, and the forward and backward detection of the endoscope is embedded in the process of airway tracking and branch labeling. Therefore, no additional camera motion information input is required in the minimum system, which broadens the application scope of endoscope-based branch positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] FIG1 is a flow chart of an endoscope branch positioning method based on airway detection and tracking provided by the present application;

[0040] FIG2 is a second flow chart of the endoscope branch positioning method based on airway detection and tracking provided by the present application;

[0041] FIG3 is a schematic structural diagram of an endoscope branch positioning device based on airway detection and tracking provided by the present application;

[0042] FIG4 is a schematic structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0044] In related technologies, vision-based endoscope positioning methods are generally divided into two categories: coordinate-level positioning and branch positioning.

[0045] Most research focuses on coordinate-level positioning, that is, obtaining the coordinates of the endoscope in the bronchial coordinate system. Specific research directions include:

[0046] 1) The SFM (Structure From Motion) method first extracts feature points from the endoscopic image. For two adjacent frames, these feature points are matched one by one and used to calculate the Perspective-n-Point (PnP) equation to estimate the camera pose. Furthermore, the camera pose estimation result can be corrected and optimized using the Bundle Adjustment and RANdom Sampling Consensus (RANSAC) methods.

[0047] 2D / 3D registration: The most common method in related technologies aims to register the 2D image captured by the endoscope to the preoperatively reconstructed model, thereby determining the camera's position within the model. 2D / 3D registration requires the design of a virtual endoscope, where a virtual camera captures a virtual image within the reconstructed model. Image registration is then performed by comparing the virtual image with the real image.

[0048] There are also studies aimed at airway-level positioning of the endoscope, that is, only determining which branch of the tracheal tree the endoscope is in. Past studies include:

[0049] 1) By estimating the depth of the endoscopic image, the depth peak point is regarded as the airway hole. Combined with the control signals such as the forward and backward movement of the endoscope, the number of layers of the endoscope in the airway is calculated (not yet accurate to the branch).

[0050] 2) The airway centers are extracted and clustered using the multiscale MSER (Maximally Stable Extremal Regions) method to obtain the next-level airway that can be observed in each frame. The detected tracheal centers are matched to the centerline of the tracheal tree reconstructed preoperatively based on CT, thereby determining which branch of the airway the endoscope is located.

[0051] 3) Through airway detection in multiple images, the hierarchy is determined based on the overlapping relationship of the airways detected between frames and mapped to the preoperative tracheal tree.

[0052] In summary, an ideal endoscope visual positioning system needs to meet the requirements of robustness, real-time performance, continuous positioning, and take up as little computing power and storage space as possible. However, existing methods have defects in different aspects.

[0053] Based on the above problems, the present application provides an endoscope branch positioning method based on airway detection and tracking. FIG1 is a flow chart of the endoscope branch positioning method based on airway detection and tracking provided by the present application. FIG2 is a flow chart of the endoscope branch positioning method based on airway detection and tracking provided by the present application. As shown in FIG1 and FIG2, the method includes:

[0054] Step 110 : Acquire bronchial images at each moment, and perform airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images.

[0055] Specifically, bronchial images at various moments may be acquired, and the bronchial images herein may be acquired based on an endoscope.

[0056] After acquiring the bronchial images at each time, airway detection may be performed on the bronchial images to obtain the airway hierarchical relationship of the bronchial images.

[0057] Here, airway detection in bronchial images can be performed using a target detection method based on deep learning or traditional computer vision. Here, the deep learning target detection network can be a Faster R-CNN network or a YOLO network series, etc. The traditional computer vision method can be MSER (MSER-Maximally Stable Extremal Regions) and the like. The embodiments of the present application do not specifically limit this.

[0058] The airway hierarchical relationship here reflects the connection relationship between the airways at the hierarchical level in the bronchial image.

[0059] Here, the method for determining the hierarchical relationship of the airways can use the yolov7 target detection network to detect the target detection frame, the target detection frame selects all levels of airways in the bronchial image, and determines the hierarchical relationship of the airways inside the bronchial image through the inclusion relationship between the detection frames.

[0060] It is understandable that, unlike previous research, during airway detection, the present embodiment not only estimates the location of the next-level airway, but also detects all observable airways at all levels. The detected airways are clustered based on the location and inclusion relationship of the detection frames, resulting in a local tracheal tree for each endoscopic image frame. For example, if detection frame a contains detection frame b, detection frame b is recorded as the next-level branch of detection frame a.

[0061] Here, the method for determining the inclusion relationship between airways can be to select the airway with the highest branching level in the image with a target detection box, or to detect the center position of the airway with the highest branching level, and determine the hierarchical relationship of the airways within the image through methods such as the position, appearance or high-dimensional features of the detected airways.

[0062] Because the endoscope can always observe the next branch in advance before entering the next branch, and can always observe the current branch when retracting to the previous branch, based on this detection strategy, the forward and backward movements of the endoscope no longer require additional endoscope motion or control signal input, but are naturally included in the airway detection and tracking process. In addition, the forward and backward directions of the airway can be estimated by image optical flow detection, and the forward and backward directions estimated by the detection and tracking process can be verified. Methods such as robotic arm control and operator gesture recognition can also be used as algorithm input to increase the reliability of motion direction estimation.

[0063] It's understandable that multiple airways can be observed in each image frame. By embedding the endoscope's forward and backward movement detection into the airway tracking and branch annotation process, the minimal system requires no additional camera motion input. Supplementing the vision module with optical flow calculation and multimodal information such as robot control signals and operator posture recognition can improve the algorithm's accuracy and reliability.

[0064] Step 120 : Determine a first matching pursuit result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images.

[0065] Specifically, bronchial images at various past moments can be obtained. Here, bronchial images at various past moments refer to bronchial images at past moments corresponding to each moment. For example, they can be bronchial images of the first 300 frames of the current moment. This embodiment of the present application does not specifically limit this.

[0066] The first matching tracking result can be determined based on the airway features of the bronchial images at various moments in the past and the airway features of the bronchial images. The airway features here can be obtained by performing airway detection on the bronchial images, and the airway features can include airway position features and airway appearance features.

[0067] For example, the airway features of bronchial images at various past moments can be compared with the airway features of the current bronchial image to determine the association between previously detected airways and the airways detected at the current moment, i.e., which airways detected at each moment belong to the same airway. Based on the association between previously detected airways and the airways detected at the current moment, it is determined to which branch of the airway tree the airway detected at the current moment can be mapped with the airways associated with the past detections.

[0068] Step 130: if the airway features of the bronchial image do not match the airway features of the bronchial images at each past moment, determine a second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship;

[0069] Step 140: Feature matching is performed between the key frames of the bronchial images at the past moments and the bronchial image. If the feature matching is successful, the first matching pursuit result is updated based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image. If the airway features of the bronchial image do not match the airway features of the bronchial images at the past moments, the second matching pursuit result is updated based on the tracheal tree branch map and the airway hierarchical relationship.

[0070] Step 150: Determine a target matching pursuit result based on the first matching pursuit result and the second matching pursuit result.

[0071] Specifically, if the airway features of the bronchial image do not match those of bronchial images at previous moments, a second matching pursuit result can be determined based on the tracheal tree branch map and the airway hierarchical relationship. The tracheal tree branch map is obtained by skeletonizing the preoperative CT (Computed Tomography) image.

[0072] When the number of airways detected at a certain level is less than the number of airways in the tracheal tree branch map, the correspondence between the detected airways and the bronchial tree branches in the tracheal tree branch map is expressed as a probability function, which serves as the basis for determining the mapping relationship. Factors included in the probability function include: airway angle, airway depth relative to the camera, and the intersection position of the airway with the previous level, which are not specifically limited in this embodiment of the application.

[0073] In order to improve the positioning stability and reduce the positioning failure caused by matching errors, a loop detection step can be added after the annotation completion process. That is, the key frames of the bronchial images at various past moments are feature matched with the bronchial images. If the feature matching is successful, the first matching tracking result is updated based on the airway features of the bronchial images at various past moments and the airway features of the bronchial images. If the airway features of the bronchial images do not match the airway features of the bronchial images at various past moments, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchical relationship, thereby restoring the correct positioning. Here, the feature matching adopts any one of ORB (Oriented FAST and Rotated BRIEF, fast feature point extraction and description algorithm), SIFT (Scale-invariant feature transform, size invariant feature conversion algorithm) and dense feature point matching algorithm based on deep learning, and the embodiments of the present application do not make specific restrictions on this.

[0074] Here, the dense feature point matching algorithm based on deep learning can be LoFTR (Local Feature TRansformer) or the like, and this embodiment of the present application does not specifically limit this.

[0075] Finally, a target matching pursuit result can be determined based on the first matching pursuit result and the second matching pursuit result. The target matching pursuit result here reflects the matching relationship between the airway of the bronchial image at each moment and the airway of the bronchial image at each moment in the past.

[0076] The method provided in an embodiment of the present application obtains bronchial images at each moment and performs airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images. A first matching tracking result is determined based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images. If the airway features of the bronchial images do not match the airway features of the bronchial images at each past moment, a second matching tracking result is determined based on the tracheal tree branch map and the airway hierarchical relationship. Finally, a target matching tracking result is determined based on the first matching tracking result and the second matching tracking result. During the matching tracking process, the new detection result in each frame of the image is matched with all historical tracking results, thereby avoiding the error accumulation problem that may occur in incremental positioning methods and automatically recovering after an error occurs, thereby improving the accuracy and reliability of endoscope-based branch positioning. In addition, multiple levels of airways can be observed in each frame of the image detection, and the detection of the forward and backward movement of the endoscope is embedded in the airway tracking and branch labeling process. Therefore, in the minimum system, no additional camera motion information input is required, which expands the application scope of endoscope-based branch positioning.

[0077] The advantages of the method provided in the embodiment of the present application are:

[0078] 1. The main algorithm is based on a deep learning method for real-time operation, with fast computing speed;

[0079] 2. Compared with existing hierarchical positioning methods, multi-level airway detection and tracking embeds the forward and backward movements of the endoscope; supplementary visual modules such as optical flow calculation can be added to further improve positioning reliability; it reduces a large number of unrealistic assumptions and can cope with complex situations such as airways at the edge of the field of view or not observed, uneven lighting, and multi-branch airways, meeting actual usage requirements.

[0080] 3. Information from other modalities, such as robot control signals and operator hand posture recognition, can be used as auxiliary inputs to improve the reliability of the algorithm's camera motion estimation.

[0081] 4. Perform airway detection on the bronchial image to obtain the airway hierarchy relationship of the bronchial image. Therefore, there is no obvious difference in the error between the low-level airways and the high-level airways.

[0082] Based on the above embodiment, the airway characteristics include airway position characteristics and airway appearance characteristics;

[0083] Correspondingly, step 120 includes:

[0084] Step 121 : Determine the first matching tracking result based on the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial images.

[0085] Specifically, the airway features may include airway position features and airway appearance features. The airway position features herein reflect feature information at the airway position level, and the airway appearance features herein reflect feature information at the airway appearance level.

[0086] Correspondingly, the first matching tracking result may be determined based on the airway position features and airway appearance features of the bronchial images at various past moments, and the airway position features and airway appearance features of the bronchial images.

[0087] For example, the airway position features of the bronchial images at various past moments can be compared with the airway position features of the bronchial images, and the airway appearance features of the bronchial images at various past moments can be compared with the airway appearance features of the bronchial images to determine the first matching tracking result.

[0088] Based on the above embodiment, step 121 includes:

[0089] Step 1211: Determine the first matching pursuit result based on the intersection-over-union relationship between the airway position features of the bronchial images at each past moment and the airway position features of the bronchial image, and the similarity between the airway appearance features of the bronchial images at each past moment and the airway appearance features of the bronchial image, wherein the intersection-over-union relationship refers to the IoU distance between the airway position features of the bronchial images at each past moment and the airway position features of the bronchial image.

[0090] Specifically, the first matching tracking result can be determined based on the IoU distance between the airway position features of the bronchial image at each past moment and the airway position features of the bronchial image, and the similarity between the airway appearance features of the bronchial image at each past moment and the airway appearance features of the bronchial image.

[0091] For detection boxes A and B, the IoU between the two represents the ratio of the intersection to the union of the two detection boxes, that is:

[0092] The formula for IoU distance is:

[0093] Here, the similarity between the airway appearance features of the bronchial images at each past moment and the airway appearance features of the bronchial images can be calculated using methods such as Re-ID, cosine similarity, Pearson correlation coefficient, etc., and this embodiment of the present application does not specifically limit this.

[0094] Based on the above embodiment, the airway appearance feature is based on the output of the airway appearance network;

[0095] The airway appearance network is trained based on sample bronchial images and labeled bronchial classification results.

[0096] Specifically, the airway appearance feature is based on the output of the airway appearance network. The airway appearance network can be a Re-ID network, HRNet, or various deep neural networks such as Transformer and ResNet. The embodiments of this application do not make specific limitations on this.

[0097] The airway appearance network here is trained based on sample bronchial images and labeled bronchial classification results.

[0098] That is, sample bronchial images and labeled bronchial classification results can be collected in advance, and an initial airway appearance network can also be pre-built.

[0099] After obtaining the initial airway appearance network, you can use the pre-collected sample bronchial images and labeled bronchial classification results to train the initial airway appearance network:

[0100] First, the sample bronchial image is input into the initial airway appearance network, which obtains and outputs the bronchial classification prediction result.

[0101] After obtaining the bronchial classification prediction result, the bronchial classification prediction result can be compared with the labeled bronchial classification result, and the loss function value can be calculated according to the degree of difference between the two. The parameters of the initial airway appearance network are iterated based on the loss function value, and the initial airway appearance network after the parameter iteration is completed is recorded as the airway appearance network.

[0102] It can be understood that the greater the difference between the bronchial classification prediction result and the labeled bronchial classification result, the greater the loss function value; the smaller the difference between the bronchial classification prediction result and the labeled bronchial classification result, the smaller the loss function value.

[0103] Airway appearance features are high-dimensional vectors (descriptors) generated by a deep neural network based on the input image. For each detection result, the portion within the detection frame is cropped and fed into the Re-ID network to obtain the feature descriptor for that object. For two detection frame images, the cosine distance of the corresponding descriptors can be used to calculate the similarity between the two detection frame contents.

[0104] During airway appearance network training, the problem is transformed into a classification problem, aiming to train a network to classify the same object into the same category. During testing, the final fully connected layer of the network is removed, retaining only the convolutional layer, and the output is the high-dimensional descriptor vector. In the embodiment of this application, the appearance airway network specifically uses the ResNet50 network, and manually annotated sample bronchial images are used to fine-tune the network weights trained on the pedestrian Re-ID dataset.

[0105] Based on the above embodiment, step 120 may further include:

[0106] Step 121, based on the model, predict the position of the airway of the bronchial image at the current moment, obtain the predicted box position of the airway in the bronchial image, and add the predicted box position to the camera motion compensation, wherein the model refers to the Kalman filter algorithm.

[0107] Specifically, the position of the airway in the bronchial image at each past moment at the current moment can be predicted based on the Kalman filter algorithm, the predicted frame position of the airway in the bronchial image can be obtained, and the predicted frame position can be added to the camera motion compensation.

[0108] In addition, the airway detection stage can also calculate confidence. Detection frames with low confidence may contain detection errors or occluded objects. Therefore, in this embodiment, not only do we retain the detection frames with high confidence, but we also filter the detection results with a very weak NMS (Non-Maximum Suppression) and divide the remaining detection frames into two groups: high confidence and low confidence.

[0109] High-confidence detections are then matched with the tracking records of the previous frame. The similarity criteria used for matching are Intersection over Union (IoU) distance and Re-ID descriptor similarity, and the matching method is the Hungarian algorithm.

[0110] The unmatched high-confidence detections and low-confidence detections are then matched against all past tracking records. The similarity calculation method used for matching is the Intersection over Union (IoU) distance, and the matching method is the Hungarian algorithm.

[0111] Then, based on all matching detections, the Kalman filter parameters are updated.

[0112] Finally, if the confidence of the remaining unmatched detection is higher than the set threshold, the detection is added as a new tracking record.

[0113] Based on the above embodiment, the step 121 of adding the predicted frame position to the camera motion compensation includes:

[0114] Step 1211, calculating an image transformation relationship between the bronchial images of adjacent frames, and determining a minimization problem based on the image transformation relationship;

[0115] Step 1212: Solve the minimization problem to obtain image transformation parameters, and add the image transformation parameters and the predicted box position to the camera motion compensation.

[0116] Specifically, the rapid movement of the endoscope also causes the airway position in the image to shift rapidly. A key assumption in target tracking for matching detection frames over time is that the target positions have a sufficiently large intersection. Camera motion can seriously affect the accuracy of this assumption. To compensate for camera motion, the Enhanced Correlation Coefficient Maximization (ECC) method is used to approximately calculate the image transformation relationship between bronchial images in adjacent frames. Based on this image transformation relationship, a minimization problem is determined, i.e., the solution to the minimization problem is:

[0117] Among them, ‖·‖ represents the Euclidean norm, p is the image transformation parameter, i r and i w (p) are the endoscopic image and the transformed endoscopic image, respectively. and is the grayscale normalized i r and i w (p).

[0118] After solving the minimization problem, the image transformation parameters are obtained, and then the image transformation parameters and the predicted box position are added to the camera motion compensation.

[0119] The image transformation relationship may include any one of Euclidean transformation, affine transformation, and homography transformation.

[0120] Here, the image transformation may be a Euclidean transformation as an example, and an affine transformation or a homographic transformation may also be used.

[0121] For a point on an image:

[0122] The Euclidean transformation matrix is:

[0123] in, is the rotation matrix. is the translation vector.

[0124] The coordinate transformation from P to P' is:

[0125] Additionally, camera motion supplements alternatives:

[0126] The image transformation can be a homography transformation, then for a point on the image:

[0127] The homography transformation matrix is:

[0128] After P is transformed to P′ through homography, its image coordinates are:

[0129] At this time, the calculation of H requires finding the matching point pairs of the two frames of images through the feature matching method and solving it by minimizing the reprojection error through the least squares method:

[0130] in,

[0131] Furthermore, as the endoscope moves, new airway targets constantly appear in the bronchial image. Due to dramatic camera motion and poor lighting, detection errors can also occur in bronchial images. Therefore, it is necessary to match newly detected targets with branches on the tracheal tree branch map and correct any erroneous detections and tracking.

[0132] For a new bronchial image, the airway matching tracking algorithm is used to obtain a possibly incomplete tracheal annotation result. The annotation completion process is as follows:

[0133] a) If the number of annotated detections and tracks is greater than one, the camera roll angle of the current frame is estimated based on the center positions of these detection frames and their positions in previous frames, i.e., the camera roll angle. If the number of detections and tracks is less than or equal to one, the camera roll angle of the current frame is calculated based on feature point matching and camera intrinsic parameters. In this case, the camera roll angle is calculated using the image rotation component in camera motion compensation (the rotation matrix A is used for rotational-translational transformation, and the rotation component needs to be further decomposed for homography transformation).

[0134] b) Sort the annotated tracking detection frames from high to low according to the number of tracking frames and the detection confidence, and then begin to annotate and correct the detection frames associated with them in order.

[0135] c) If there are unmarked detection frames at the next level, match them to the tracheal tree map based on the center position of each detection frame at the next level. If the number of detected next-level branches is less than the number of next-level branches in the map, the probability of observing each airway is calculated based on the angle between the current branch and each branch at the next level. In the embodiment of the present application, the smaller the angle with the current branch, the greater the probability of the next-level branch being observed. Then, make a cross-section of the next-level branch on the tracheal tree branch map, rotate each intersection according to the current camera roll angle, and obtain the position of each branch in the current field of view (two-dimensional local tracheal tree map). After normalizing the local tracheal tree map and the center coordinates of the current frame detection frame, match the detection frame and the local map according to the Euclidean distance metric using the Hungarian algorithm to obtain the branch annotations of each detection frame.

[0136] d) If there are unlabeled detection boxes at the same level, match them to the bronchial branch map of the previous level as in c).

[0137] e) If there are unlabeled detection frames in the previous level, since the previous level branch is unique, the previous level branch number is directly assigned to the unlabeled detection frame.

[0138] Here, the camera roll angle can be calculated based on the SFM method using only feature matching and camera intrinsic parameters. For robot-controlled endoscopes, the camera roll angle can be referenced by the control signal input; for handheld endoscopes, the camera roll angle can be estimated based on the operator's gesture recognition.

[0139] The following describes the endoscope branch positioning device based on airway detection and tracking provided in the present application. The endoscope branch positioning device based on airway detection and tracking described below and the endoscope branch positioning method based on airway detection and tracking described above can be referenced to each other.

[0140] Based on any of the above embodiments, the present application provides an endoscope branch positioning device based on airway detection and tracking. FIG3 is a structural schematic diagram of the endoscope branch positioning device based on airway detection and tracking provided by the present application. As shown in FIG3, the device includes:

[0141] An airway detection unit 310 is configured to acquire bronchial images at various moments and perform airway detection on the bronchial images to obtain an airway hierarchical relationship of the bronchial images;

[0142] a first matching pursuit unit 320, configured to determine a first matching pursuit result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images;

[0143] A second matching pursuit unit 330 is configured to determine a second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship when the airway features of the bronchial image do not match the airway features of the bronchial images at each past moment;

[0144] an updating unit 340 configured to perform feature matching between the key frames of the bronchial images at the past moments and the bronchial image, and, if the feature matching is successful, to update the first matching pursuit result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image; and, if the airway features of the bronchial image do not match the airway features of the bronchial images at the past moments, to update the second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship;

[0145] The target matching pursuit result determining unit 350 is configured to determine a target matching pursuit result based on the first matching pursuit result and the second matching pursuit result.

[0146] The apparatus provided in an embodiment of the present application acquires bronchial images at each moment and performs airway detection on the bronchial images to obtain an airway hierarchical relationship in the bronchial images. A first matching tracking result is determined based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images. If the airway features of the bronchial images do not match the airway features of the bronchial images at each past moment, a second matching tracking result is determined based on the tracheal tree branch map and the airway hierarchical relationship. Finally, a target matching tracking result is determined based on the first matching tracking result and the second matching tracking result. During the matching tracking process, the new detection result in each frame of the image is matched with all historical tracking results, thereby avoiding the error accumulation problem that may occur in incremental positioning methods and automatically recovering after an error occurs, thereby improving the accuracy and reliability of endoscope-based branch positioning. In addition, multi-level airways can be observed in each frame of the image detection, and the detection of the forward and backward movement of the endoscope is embedded in the airway tracking and branch labeling process. Therefore, in the minimum system, no additional camera motion information input is required, which expands the application scope of endoscope-based branch positioning.

[0147] Based on any of the above embodiments, the airway characteristics include airway position characteristics and airway appearance characteristics;

[0148] Correspondingly, the first matching tracking unit 320 specifically includes:

[0149] The first matching pursuit subunit is configured to determine the first matching pursuit result based on the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial images.

[0150] Based on any of the above embodiments, the first matching pursuit subunit is specifically configured to:

[0151] The first matching tracking result is determined based on the IoU distance between the airway position features of the bronchial images at the past moments and the airway position features of the bronchial image, and the similarity between the airway appearance features of the bronchial images at the past moments and the airway appearance features of the bronchial image.

[0152] Based on any of the above embodiments, the airway appearance feature is based on the output of the airway appearance network;

[0153] The airway appearance network is trained based on sample bronchial images and labeled bronchial classification results.

[0154] Based on any of the above embodiments, it further includes a camera motion compensation unit, and the camera motion compensation unit specifically includes:

[0155] A prediction unit is used to predict the position of the airway of the bronchial image at each past moment at the current moment based on a Kalman filter algorithm, obtain the predicted frame position of the airway in the bronchial image, and add the predicted frame position to the camera motion compensation.

[0156] Based on any of the above embodiments, the prediction unit is specifically configured to:

[0157] calculating an image transformation relationship between the bronchial images of adjacent frames, and determining a minimization problem based on the image transformation relationship;

[0158] Solve the minimization problem to obtain image transformation parameters, and add the image transformation parameters and the predicted box position to the camera motion compensation.

[0159] Based on any of the above embodiments, the feature matching adopts any one of a fast feature point extraction and description algorithm, a size-invariant feature conversion algorithm, and a dense feature point matching algorithm based on deep learning.

[0160] FIG4 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG4 , the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute an endoscopic branch positioning method based on airway detection and tracking, the method comprising: acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; determining a first matching tracking result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images; and determining a second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship when the airway features of the bronchial images do not match the airway features of the bronchial images at each past moment. Results; feature matching is performed on the key frames of the bronchial images at the past moments with the bronchial images, and when the feature matching is successful, the first matching tracking result is updated based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images, and when the airway features of the bronchial image and the airway features of the bronchial images at the past moments do not match, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchical relationship; based on the first matching tracking result and the second matching tracking result, the target matching tracking result is determined.

[0161] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0162] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the endoscopic branch positioning method based on airway detection and tracking provided by the above methods, the method including: acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images, determining a first matching tracking result; when the airway features of the bronchial image and the airway features of the bronchial images at each past moment do not match When the airway features of the bronchial images at the past moments do not match, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchical relationship; the key frames of the bronchial images at the past moments are feature matched with the bronchial images, and when the feature matching is successful, the first matching tracking result is updated based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images, and when the airway features of the bronchial images do not match the airway features of the bronchial images at the past moments, the second matching tracking result is updated based on the tracheal tree branch map and the airway hierarchical relationship; the target matching tracking result is determined based on the first matching tracking result and the second matching tracking result.

[0163] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the endoscopic branch positioning method based on airway detection and tracking provided by the above-mentioned methods, the method comprising: acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; determining a first matching tracking result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images; and determining a first matching tracking result based on the tracheal tree branch map when the airway features of the bronchial images and the airway features of the bronchial images at each past moment do not match. and the airway hierarchical relationship, determine the second matching tracking result; perform feature matching on the key frames of the bronchial images at the past moments and the bronchial images, and if the feature matching is successful, update the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images, and if the airway features of the bronchial images and the airway features of the bronchial images at the past moments do not match, update the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship; determine the target matching tracking result based on the first matching tracking result and the second matching tracking result.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0165] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An endoscope branch positioning method based on airway detection and tracking, comprising: Acquiring bronchial images at each moment, and performing airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; Determining a first matching pursuit result based on the airway features of the bronchial images at each moment in the past and the airway features of the bronchial images; In a case where the airway features of the bronchial image do not match the airway features of the bronchial images at each past moment, determining a second matching pursuit result based on the tracheal tree branch map and the airway hierarchical relationship; Performing feature matching between the key frames of the bronchial images at the past moments and the bronchial image, and updating the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial image if the feature matching is successful, and updating the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship if the airway features of the bronchial image and the airway features of the bronchial images at the past moments do not match; A target matching pursuit result is determined based on the first matching pursuit result and the second matching pursuit result.

2. The endoscope branch positioning method based on airway detection and tracking according to claim 1, wherein: The airway characteristics include airway position characteristics and airway appearance characteristics; Correspondingly, the determining of the first matching tracking result based on the airway features of the bronchial images at each moment in the past and the airway features of the bronchial images includes: The first matching tracking result is determined based on the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial image.

3. The endoscope branch positioning method based on airway detection and tracking according to claim 2, wherein: The determining the first matching tracking result based on the airway position features and airway appearance features of the bronchial images at each past moment, and the airway position features and airway appearance features of the bronchial images, comprises: Based on the airway position features of the bronchial images at various moments in the past and the intersection relationship between the airway position features of the bronchial images, and the bronchial images at various moments in the past The first matching tracking result is determined based on the similarity between the airway appearance features of the bronchial image and the airway appearance features of the bronchial image.

4. The endoscope branch positioning method based on airway detection and tracking according to claim 2, wherein: The airway appearance feature is based on the output of the airway appearance network; The airway appearance network is trained based on sample bronchial images and labeled bronchial classification results.

5. The endoscope branch positioning method based on airway detection and tracking according to claim 1, wherein: The airway features of the bronchial images at various moments in the past and the airway features of the bronchial images are used to determine the first matching tracking result, and the method also includes: The position of the airway of the bronchial image at each past moment at the current moment is predicted based on the model, the predicted frame position of the airway in the bronchial image is obtained, and the predicted frame position is added to the camera motion compensation.

6. The endoscope branch positioning method based on airway detection and tracking according to claim 5, wherein: The adding the predicted frame position to the camera motion compensation comprises: Calculating an image transformation relationship between the bronchial images of adjacent frames, and determining a minimization problem based on the image transformation relationship; The minimization problem is solved to obtain image transformation parameters, and the image transformation parameters and the predicted box position are added to the camera motion compensation.

7. The endoscope branch positioning method based on airway detection and tracking according to any one of claims 1 to 6, wherein: The feature matching adopts any one of a fast feature point extraction and description algorithm, a size-invariant feature conversion algorithm, and a dense feature point matching algorithm based on deep learning.

8. An endoscope branch positioning device based on airway detection and tracking, comprising: An airway detection unit, used to obtain bronchial images at each moment, and perform airway detection on the bronchial images to obtain the airway hierarchical relationship of the bronchial images; a first matching pursuit unit, configured to determine a first matching pursuit result based on the airway features of the bronchial images at each past moment and the airway features of the bronchial images; A second matching pursuit unit is used to determine a second matching pursuit result based on a tracheal tree branch map and the airway hierarchical relationship when the airway feature of the bronchial image does not match the airway feature of the bronchial image at each past moment; an updating unit, configured to perform feature matching between the key frames of the bronchial images at the past moments and the bronchial images, and to update the first matching tracking result based on the airway features of the bronchial images at the past moments and the airway features of the bronchial images if the feature matching is successful, and to update the second matching tracking result based on the tracheal tree branch map and the airway hierarchical relationship if the airway features of the bronchial images and the airway features of the bronchial images at the past moments do not match; The target matching tracking result determining unit is configured to determine a target matching tracking result based on the first matching tracking result and the second matching tracking result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the endoscope branch positioning method based on airway detection and tracking as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the endoscope branch positioning method based on airway detection and tracking as described in any one of claims 1 to 7 is implemented.

Images