Endoscopic navigation positioning method and device

Abstract

The present specification relates to the technical field of endoscopy, and specifically discloses an endoscope navigation positioning method and device, wherein the method comprises: acquiring an endoscope image of a target tissue organ; the endoscope image comprises a first image collected by a monocular camera arranged in the endoscope before the endoscope performs a preset action and a second image collected after the preset action is performed; based on visual SLAM, pose data of an endoscope catheter and SLAM map data corresponding to the target tissue organ are determined using the first image and the second image; three-dimensional model data of the target tissue organ is acquired; a coordinate system of the SLAM map data is registered with a coordinate system of the three-dimensional model data, so as to generate virtual endoscope navigation data based on the pose data of the endoscope catheter. The above scheme can improve the accuracy of endoscope navigation positioning.

Description

Technical Field

[0001] This specification relates to the field of endoscopy technology, and in particular to an endoscopy navigation and positioning method and device. Background Technology

[0002] The bronchial anatomy is complex and the luminal surfaces are similar, therefore bronchoscopy requires a navigation and positioning system. Bronchoscopy navigation and positioning systems provide the position of the bronchoscope in CT images. Common methods include CT video-based systems, systems based on additional electromagnetic sensors, or hybrid systems. CT video-based bronchoscopy tracking estimates the bronchoscope camera pose by minimizing the difference between the bronchoscope image (RB) and the virtual bronchoscope image (VB). Electromagnetic sensor-based bronchoscopy tracking navigates using the output of electromagnetic sensors attached to the camera tip or guidewire. This requires preoperative manual-eye calibration to convert the sensor's 3D output from sensor coordinates to the actual camera coordinates, or other registration methods to convert the sensor's 3D output from sensor coordinates to CT image coordinates.

[0003] However, CT video-based bronchoscopic tracking relies heavily on the similarity between RB and VB images, lacking local texture information (bifurcation or folding), leading to significant tracking errors or tracking failures. Furthermore, it requires substantial computation time to generate virtual and matching images. Electromagnetic sensor-based bronchoscopic tracking has several drawbacks: the positional data from electromagnetic sensors may fluctuate; the sensors may be affected by metal surgical instruments in the operating room; and the bronchoscopes using electromagnetic sensors are relatively large and cannot be inserted into the terminal bronchus.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This specification provides an endoscopic navigation and positioning method and apparatus to address the issues of low accuracy and speed in existing bronchoscopic positioning methods.

[0006] This specification provides an endoscopic navigation and positioning method, including:

[0007] Acquire endoscopic images of target tissues and organs; the endoscopic images include a first image captured by a monocular camera installed in the endoscope before the endoscope performs a preset action and a second image captured after the preset action is performed;

[0008] Visual SLAM is used to determine the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue or organ using the first and second images.

[0009] Acquire the three-dimensional model data of the target tissue or organ; register the coordinate system of the SLAM map data with the coordinate system of the three-dimensional model data to generate virtual endoscopic navigation data based on the pose data of the endoscopic catheter.

[0010] In one embodiment, the preset actions include rotational actions and translational actions.

[0011] In one embodiment, visual SLAM is used to determine the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue or organ using the first image and the second image, including:

[0012] Visual SLAM coordinates are initialized using the first image and the second image;

[0013] The endoscopic catheter is tracked and located using visual SLAM, and the target tissue or organ is reconstructed to obtain the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue or organ.

[0014] In one embodiment, the coordinate system of the SLAM map data is registered with the coordinate system of the 3D model data to generate virtual endoscopic navigation data based on the pose data of the endoscopic catheter, including:

[0015] The coordinate system of the SLAM map data is registered with the coordinate system of the 3D model data to obtain the transformation matrix;

[0016] The pose data is transformed into the coordinate system corresponding to the three-dimensional model data using the transformation matrix to generate virtual endoscope navigation data.

[0017] In one embodiment, visual SLAM coordinate initialization is performed using the first image and the second image, including:

[0018] Feature extraction is performed on the first image to obtain the first feature point; feature extraction is performed on the second image to obtain the second feature point;

[0019] The first feature point and the second feature point are matched to obtain a matching point pair;

[0020] Based on the matching point pairs, a rotation matrix and a unit translation vector corresponding to the monocular camera are generated; the displacement vector is calculated based on the displacement length data in the preset action and the unit translation vector.

[0021] The map points corresponding to the matching point pairs are generated using the rotation matrix and the displacement vector.

[0022] When the number of map points exceeds the preset number, the visual SLAM coordinate initialization is considered successful.

[0023] In one embodiment, feature extraction is performed on the first image to obtain first feature points, including:

[0024] The first image is downsampled to generate N pyramid images of different scales; where N is an integer greater than 2.

[0025] Feature points are extracted from the N-layer pyramid images at different scales; the extracted feature points are then filtered using a quadtree to obtain the first feature points that are evenly distributed in the image.

[0026] Calculate the gray-scale centroid of the first feature point, and calculate the orientation angle of the first feature point based on the position of the gray-scale centroid and the position of the first feature point;

[0027] Calculate the feature descriptor of the first feature point to represent the image information around the first feature point in binary form.

[0028] In one embodiment, matching the first feature point and the second feature point to obtain a matching point pair includes:

[0029] Take a first feature point A in the first image, and select a preset area centered on the first feature point A as the specified range for searching matching features in the second image;

[0030] Calculate the Hamming distance between the feature descriptor of the first feature point A and the feature descriptors of the second feature points within the specified range in the second image; sort the calculated Hamming distances in descending order to obtain the shortest Hamming distance and the second shortest Hamming distance;

[0031] When the ratio between the shortest Hamming distance and the second shortest Hamming distance is less than a preset ratio, it is determined that the first feature point A matches the second feature point B corresponding to the shortest Hamming distance; for the second feature point B, it is determined whether the second feature point B matches the first feature point A;

[0032] When determining that the second feature point B matches the first feature point A, the first feature point A and the second feature point B are determined to be a matching point pair;

[0033] Angle and length histograms are built based on matching point pairs, and mainstream angles and mainstream lengths are selected.

[0034] The RANSAC algorithm is used to eliminate incorrect matching point pairs based on the mainstream angle and mainstream length, thereby obtaining valid matching point pairs.

[0035] In one embodiment, visual SLAM is used to track and locate the endoscopic catheter and reconstruct a map of the target tissue or organ to obtain pose data of the endoscopic catheter and SLAM map data corresponding to the target tissue or organ, including:

[0036] The pose data of the endoscopic catheter in the current frame is determined by constant velocity motion model, reference frame or relocalization, SLAM map data is built, and key frames are selected.

[0037] BA optimizes the pose data of keyframes and the SLAM map data;

[0038] The six degrees of freedom of the virtual endoscope are optimized using an optimization algorithm to maximize the similarity between the virtual endoscope image and the keyframe image. The position of the virtual endoscope corresponding to the keyframe in the coordinate system corresponding to the three-dimensional model data is recorded.

[0039] In one embodiment, the pose data of the BA-optimized keyframes and the SLAM map data, along with the optimization algorithm used to optimize the six degrees of freedom of the virtual endoscope to maximize the similarity between the virtual endoscope image and the keyframe image, are executed in a thread-scheduled manner.

[0040] In one embodiment, the coordinate system of the SLAM map data is registered with the coordinate system of the 3D model data to obtain a transformation matrix, including one of the following methods:

[0041] The local 3D point cloud model of tissues and organs in the SLAM map data is fused with the global 3D point cloud model of tissues and organs in the 3D model data to obtain a transformation matrix.

[0042] The pose data of the endoscopic catheter in the SLAM map data is fused with the centerline data extracted from the 3D model data to obtain the transformation matrix;

[0043] At least four points are selected on the local tissue and organ 3D point cloud model in the SLAM map data and the global tissue and organ 3D point cloud model in the 3D model data for registration to obtain the transformation matrix;

[0044] The position of the endoscope catheter where the keyframe is located in the coordinate system of the SLAM map data is fused with the position of the virtual endoscope corresponding to the keyframe in the coordinate system of the 3D model data to obtain the transformation matrix.

[0045] This specification also provides an endoscopic navigation and positioning device, comprising:

[0046] The acquisition module is used to acquire endoscopic images of target tissues and organs; the endoscopic images include a first image captured by a monocular camera installed in the endoscope before the endoscope performs a preset action and a second image captured after the preset action is performed;

[0047] The reconstruction module is used to determine the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue organ based on the first image and the second image using visual SLAM.

[0048] The generation module is used to acquire three-dimensional model data of the target tissue or organ; it is also used to register the coordinate system of the SLAM map data with the coordinate system of the three-dimensional model data to generate virtual endoscopic navigation data based on the pose data of the endoscopic catheter.

[0049] This specification also provides an endoscopic navigation and positioning system, comprising:

[0050] An endoscope, wherein the endoscope has an illumination source and a monocular camera at the distal end of its catheter; the monocular camera is used to acquire endoscopic images of target tissues and organs;

[0051] An image processor is configured to acquire endoscopic images from the endoscope; the endoscopic images include a first image captured by the monocular camera before the endoscope performs a preset action and a second image captured after the preset action is performed; it is also configured to determine the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue / organ based on visual SLAM using the first and second images; it is also configured to acquire three-dimensional model data of the target tissue / organ; and register the coordinate system of the SLAM map data with the coordinate system of the three-dimensional model data to generate virtual endoscopic navigation data based on the pose data of the endoscopic catheter.

[0052] An image display is used to acquire the virtual endoscope navigation data from the image processor and to display images based on the virtual endoscope navigation data.

[0053] This specification also provides a medical device, including a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the endoscopic navigation and positioning method described in any of the above embodiments.

[0054] This specification also provides a computer-readable storage medium storing computer instructions that, when executed, implement the steps of the endoscopic navigation and positioning method described in any of the above embodiments.

[0055] This specification provides an endoscopic navigation and positioning method that can acquire endoscopic images of target tissues and organs during endoscopic surgery. The endoscopic images include a first image captured by a monocular camera within the endoscope before the endoscope performs a preset action, and a second image captured after the preset action. Based on visual SLAM, the first and second images can be used to determine the pose data of the endoscopic catheter and the corresponding SLAM map data of the target tissue and organ. Three-dimensional model data of the target tissue and organ can be acquired. By registering the coordinate system of the SLAM map data with the coordinate system of the three-dimensional model data, virtual endoscopic navigation data is generated based on the pose data. This provides doctors with virtual endoscopic images to guide navigation during endoscopic surgery, providing real-time position and orientation of the endoscopic catheter, facilitating endoscopic surgery and improving patient experience. In the above-described scheme, visual SLAM-based localization and navigation can achieve coordinate initialization, camera pose calculation, framework mapping, and navigation solely relying on monocular endoscopic images. This results in faster speed and higher tracking and positioning accuracy. Furthermore, it eliminates the need for binoculars and additional electromagnetic or inertial sensors, reducing system complexity and saving costs while maintaining accuracy. Virtual endoscopic navigation data can provide surgeons with a more intuitive 3D experience during surgery, facilitating endoscopic procedures, improving accuracy and efficiency, reducing surgical difficulty, and enhancing patient experience. Attached Figure Description

[0056] The accompanying drawings, which are included to provide a further understanding of this specification and form part of it, do not constitute a limitation thereof. In the drawings:

[0057] Figure 1 A flowchart of an endoscopic navigation and positioning method according to one embodiment of this specification is shown;

[0058] Figure 2 A flowchart illustrating the initialization of SLAM coordinates for an endoscopic navigation and positioning method according to one embodiment of this specification is shown.

[0059] Figure 3 A flowchart of feature extraction in an endoscopic navigation and positioning method according to one embodiment of this specification is shown;

[0060] Figure 4 A flowchart of feature point matching in an endoscopic navigation and positioning method according to one embodiment of this specification is shown;

[0061] Figure 5 A flowchart of the bronchoscopy tracking process in an embodiment of the endoscopic navigation and positioning method of this specification is shown.

[0062] Figure 6A flowchart illustrating coordinate system registration in an endoscopic navigation and positioning method according to one embodiment of this specification is shown.

[0063] Figure 7 A schematic diagram of the mechanical structure of a bronchoscopic robot using an endoscopic navigation and positioning method according to one embodiment of this specification is shown.

[0064] Figure 8 A schematic diagram of a bronchoscopy examination scenario is shown in an embodiment of the endoscopic navigation and positioning method of this specification.

[0065] Figure 9 A schematic diagram of the catheter structure of an endoscopic navigation and positioning method according to one embodiment of this specification is shown;

[0066] Figure 10 A schematic diagram of catheter tip imaging in an endoscopic navigation and positioning method according to an embodiment of this specification is shown;

[0067] Figure 11 A flowchart of an endoscopic navigation and positioning method according to one embodiment of this specification is shown;

[0068] Figure 12 A schematic diagram of feature matching for an endoscopic navigation and positioning method according to an embodiment of this specification is shown.

[0069] Figure 13 A flowchart illustrating the virtual bronchoscope pose optimization process of the endoscopic navigation and positioning method in one embodiment of this specification is shown.

[0070] Figure 14 A schematic diagram of a bronchoscope image is shown in one embodiment of the endoscopic navigation and positioning method described in this specification.

[0071] Figure 15 A schematic diagram of the bronchial tree in an embodiment of the endoscopic navigation and positioning method of this specification is shown;

[0072] Figure 16 A schematic diagram of virtual bronchial navigation is shown in one embodiment of the endoscopic navigation and positioning method of this specification;

[0073] Figure 17 A schematic diagram of an endoscope navigation and positioning device according to one embodiment of this specification is shown;

[0074] Figure 18 A schematic diagram of an endoscope navigation and positioning system according to one embodiment of this specification is shown;

[0075] Figure 19 A schematic diagram of a medical device according to one embodiment of this specification is shown. Detailed Implementation

[0076] The principles and spirit of this specification will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this specification, and are not intended to limit the scope of this specification in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0077] Those skilled in the art will recognize that the embodiments described in this specification can be implemented as a system, apparatus, method, or computer program product. Therefore, the disclosure of this specification can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0078] This specification provides an endoscopic navigation and positioning method through its embodiments. Figure 1 A flowchart of an endoscopic navigation and positioning method according to one embodiment of this specification is shown. While this specification provides method operation steps or apparatus structures as illustrated in the following embodiments or figures, more or fewer operation steps or module units may be included in the method or apparatus based on conventional or non-inventive effort. In steps or structures where there is no logically necessary causal relationship, the execution order of these steps or the module structure of the apparatus is not limited to the execution order or module structure described in the embodiments and figures of this specification. When the method or module structure is applied in actual devices or end products, it can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed processing environment) according to the method or module structure shown in the embodiments or figures.

[0079] Specifically, such as Figure 1 As shown, an endoscopic navigation and positioning method provided in one embodiment of this specification may include the following steps:

[0080] Step S101: Obtain endoscopic images of the target tissue or organ; the endoscopic images include a first image captured by a monocular camera installed in the endoscope before the endoscope performs a preset action and a second image captured after the preset action is performed.

[0081] The method in this embodiment can be applied to computer equipment or image processors, etc. The image processor can acquire endoscopic images of the target tissue or organ during endoscopic surgery. The endoscope here can include a bronchoscope, colonoscope, gastroscope, laryngoscope, etc. Correspondingly, the target tissue or organ can be the bronchus, intestine, stomach, pharynx, etc. A monocular camera can be provided at the distal end of the endoscope's catheter, which can acquire endoscopic images during the surgery. The endoscopic images can include a first image acquired by the monocular camera before the endoscope performs a preset action and a second image acquired after the preset action is performed.

[0082] Step S102: Based on visual SLAM, the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue organ are determined using the first image and the second image.

[0083] In this embodiment, visual SLAM (Simultaneous Localization and Mapping) can be used to track and locate the endoscopic catheter and reconstruct maps of the target tissues and organs. In this embodiment, visual SLAM can be based on a monocular camera. The image processor can use a first image and a second image to track and locate the endoscopic catheter using visual SLAM to obtain the pose data of the endoscopic catheter. The image processor can also reconstruct maps of the target tissues and organs based on visual SLAM to obtain SLAM map data corresponding to the target tissues and organs.

[0084] Step S103: Obtain the three-dimensional model data of the target tissue / organ; register the coordinate system of the SLAM map data with the coordinate system of the three-dimensional model data to generate virtual endoscopic navigation data based on the pose data of the endoscopic catheter.

[0085] The image processor can also acquire 3D model data of the target tissue or organ. This 3D model data can be constructed based on data acquired in advance by CT or MRI. The coordinate system of the SLAM map data can be registered with the coordinate system of the 3D model data. After registration, virtual endoscopic navigation data can be generated based on the pose data of the endoscopic catheter. The virtual endoscopic navigation data can include the 3D model data, as well as the endoscopic image and endoscopic pose data in the coordinate system of the 3D model data. The virtual endoscopic navigation data can be sent to an image display. The image display can then show the virtual endoscopic navigation data, facilitating endoscopic surgery or examinations performed by the physician.

[0086] In the above embodiments, localization and navigation based on visual SLAM rely solely on monocular endoscopic images to achieve coordinate initialization, camera pose calculation, mapping, and navigation. This results in faster speed and higher tracking and positioning accuracy. Furthermore, it eliminates the need for binoculars and additional electromagnetic or inertial sensors, reducing system complexity and saving costs while maintaining accuracy. Virtual endoscopic navigation data can provide surgeons with a more intuitive 3D experience during surgery, facilitating endoscopic procedures, improving accuracy and efficiency, reducing surgical difficulty, and enhancing patient experience.

[0087] In some embodiments of this specification, determining the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue organ based on the first image and the second image using visual SLAM may include: initializing visual SLAM coordinates using the first image and the second image; tracking and locating the endoscopic catheter based on visual SLAM and reconstructing the map of the target tissue organ to obtain the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue organ.

[0088] In this embodiment, to perform visual SLAM coordinate initialization, a first image acquired before the endoscope performs a preset action and a second image acquired after the preset action are obtained. Using multiple frames of endoscope images, visual SLAM coordinate initialization can be performed. In monocular vision, the translation of two frames of endoscope images can be normalized, effectively fixing a scale. The motion of the monocular camera and the 3D position of feature points can be calculated. After initialization, the camera motion can be calculated; the trajectory and map units after initialization are the scale fixed during initialization. Therefore, initialization is an unavoidable step in monocular SLAM. After initialization, the image processor can track and locate the endoscope catheter based on visual SLAM and reconstruct a map of the target tissue / organ, obtaining the pose data of the endoscope catheter and the corresponding SLAM map data of the target tissue / organ. Through this method, the pose data of the endoscope catheter and the SLAM map data can be obtained, facilitating subsequent positioning and navigation.

[0089] In some embodiments of this specification, the preset actions include rotational and translational actions. The preset actions can be a series of rotational and translational actions. The initial two frames of images need to have a certain degree of translation, and subsequent trajectories and maps will be based on this translation step. Monocular initialization cannot consist solely of pure rotation; a certain degree of translation is required. Without translation, monocular initialization will fail. In this embodiment, the endoscope can be controlled to perform a series of forward, backward, or bending actions, acquiring image data before and after the actions for SLAM coordinate initialization.

[0090] In some embodiments of this specification, the coordinate system of the SLAM map data is registered with the coordinate system of the 3D model data to generate virtual endoscopic navigation data based on the pose data of the endoscopic catheter. This includes: registering the coordinate system of the SLAM map data with the coordinate system of the 3D model data to obtain a transformation matrix; and using the transformation matrix to transform the pose data into the coordinate system corresponding to the 3D model data to generate virtual endoscopic navigation data. The virtual endoscopic navigation data may include the 3D model data, as well as the endoscopic image and endoscopic pose data in the coordinate system of the 3D model data. The virtual endoscopic navigation data can be displayed on an image display, facilitating endoscopic surgery or examinations performed by physicians.

[0091] In some embodiments of this specification, visual SLAM coordinate initialization using the first image and the second image may include: extracting features from the first image to obtain first feature points; extracting features from the second image to obtain second feature points; matching the first feature points and the second feature points to obtain matching point pairs; generating a rotation matrix and a unit translation vector corresponding to the monocular camera based on the matching point pairs; calculating a displacement vector based on the displacement length data in the preset action and the unit translation vector; generating map points corresponding to the matching point pairs using the rotation matrix and the displacement vector; and determining that visual SLAM coordinate initialization is successful when the number of map points is greater than a preset number.

[0092] The advantage of monocular visual SLAM is its low cost, but its biggest limitation is the inability to know the distance from the object to the camera. Therefore, the poses of the two images obtained through epipolar geometry and the triangulated map points lack scale information. Thus, motion control is needed to control the camera to execute preset actions, using the known displacement length of the camera to recover scale information. The preset action is a sequence of actions including forward, bending, and backward movements. If initialization fails, a second preset action is executed until initialization succeeds.

[0093] In one embodiment, the preset action sequence can be in the following form: First time: move forward m1 cm, Second time: bend upward n1 degrees and move forward m2 cm, Third time: bend downward to the right n2 degrees and move forward m3 cm, Fourth time: bend to the left n3 degrees and move backward m4 cm, etc. The preset action is composed of forward, bend, backward and other operations, and needs to return to the starting position within N action cycles.

[0094] For details, please refer to Figure 2 The flowchart illustrating the initialization of SLAM coordinates for an endoscopic navigation and positioning method according to an embodiment of this specification is shown. Figure 2 The following explanation uses a bronchoscope as an example. Figure 2 As shown, the steps for SLAM coordinate initialization include:

[0095] Step 1: Capture bronchoscope image I1. Image I1 needs to meet the image quality requirements, with clear images and obvious bronchial features. Then, feature point P1 is extracted through strict conditions.

[0096] Step 2: Motion control executes preset actions;

[0097] Step 3: Capture bronchoscope image I2. Image I2 needs to meet the image quality requirements, with clear images and obvious bronchial features. Then, feature points P2 are extracted under strict conditions.

[0098] Step 4: Perform strict and constrained matching on feature points P1 and P2 to obtain matching point pairs;

[0099] Step 5: Use the RANSAC algorithm to remove incorrect matching pairs and keep only the useful ones.

[0100] Step 6: Normalize the feature point coordinates of the retained feature point pairs, calculate the fundamental matrix, and decompose it to obtain R and t; R is the camera rotation matrix, and t is the camera translation vector.

[0101] Step 7: Based on the preset action executed in this motion control, obtain the known displacement length m, multiply it by the unit displacement vector to obtain the displacement vector t' of the restored scale;

[0102] Step 8: Recover the map points corresponding to the feature points using the triangulation algorithm, check the disparity angle and position of the map points, and retain the map points that meet the conditions.

[0103] Step 9: Determine if the number of map points is greater than the preset threshold. If yes, the initialization is successful; otherwise, repeat the coordinate initialization process using the next preset action.

[0104] In some embodiments described herein, feature extraction of a first image to obtain a first feature point may include: downsampling the first image to generate N pyramid images at different scales; where N is an integer greater than 2; extracting feature points from the N pyramid images at different scales; filtering the extracted feature points using a quadtree to obtain a first feature point uniformly distributed in the image; calculating the gray-level centroid of the first feature point, and calculating the orientation angle of the first feature point based on the position of the gray-level centroid and the position of the first feature point; and calculating the feature descriptor of the first feature point to represent the image information around the first feature point in binary form.

[0105] For details, please refer to Figure 3This diagram illustrates a feature extraction flowchart in an endoscopic navigation and positioning method according to one embodiment of this specification. Each feature point includes location, size, orientation, score, and descriptive information. For example... Figure 3 As shown, the image can be downsampled to generate n-layer pyramids of different scales, with the number of pyramid layers needing to be greater than two. Feature points are extracted from the n-layer images at different scales using either the Fast or Harris algorithm. Quadtree filtering is used to select feature points, and non-maximum suppression is applied to densely populated regions to ensure a uniform distribution of feature points across the image. Uniform feature points improve pose calculation accuracy. A region of a certain size is selected centered on the feature point, and its gray-level centroid is calculated. The orientation angle of the feature point is then calculated based on its position and the location of the feature point. A BRIEF feature descriptor is calculated to represent the image information surrounding the feature point in binary form. This feature point is extracted from the current image and filtered for density using non-maximum suppression. Through this method, rigorous feature extraction can be performed on endoscopic images.

[0106] In some embodiments described herein, matching the first feature point and the second feature point to obtain a matching point pair may include: taking a first feature point A in the first image, and selecting a preset region centered on the first feature point A as a specified range for searching matching features in the second image; calculating the Hamming distance between the feature descriptor of the first feature point A and the feature descriptor of the second feature point within the specified range in the second image; sorting the calculated Hamming distances in descending order to obtain the shortest Hamming distance and the second shortest Hamming distance; when the ratio between the shortest Hamming distance and the second shortest Hamming distance is less than a preset ratio, determining that the first feature point A matches the second feature point B corresponding to the shortest Hamming distance; for the second feature point B, determining whether the second feature point B matches the first feature point A; when the second feature point B matches the first feature point A, determining that the first feature point A and the second feature point B are a matching point pair; establishing an angle histogram and a length histogram based on the matching point pair, and selecting the mainstream angle and mainstream length; using the RANSAC algorithm based on the mainstream angle and mainstream length to eliminate erroneous matching point pairs and obtain valid matching point pairs.

[0107] For details, please refer to Figure 4 The diagram illustrates a feature point matching flowchart in an endoscopic navigation and positioning method according to one embodiment of this specification. Figure 4 As shown, the feature point matching process may include the following steps.

[0108] Step 1. Take any feature fa from image A, and expand a certain area centered on the position of fa to define the range for searching for matching features in image B.

[0109] Step 2. Take the feature descriptor of feature fa and calculate its Hamming distance d to all feature descriptors within range r in image B.

[0110] Step 3. Sort the obtained distances and select the features with the shortest and second shortest distances, denoted as d1 and d2.

[0111] Step 4. Nearest Neighbor and Second Nearest Neighbor Distance Ratio: Take the feature fa of image A and search for the two features that are closest to it to image B. The distances are denoted as d1 and d2 respectively. Only when d1 / d2 is less than a certain threshold is it considered a good match.

[0112] Step 5. Cross-checking: If feature fb is the best match for feature fa in image B, and feature fa is also the best match for feature fb in image A, then fa and fb are considered a good match.

[0113] Step 6. Build an angle histogram based on the angle difference of the matched feature points, and select the mainstream angle of the histogram.

[0114] Step 7. Find the minimum matching distance and select only matching point pairs whose distance is n times greater than the minimum distance.

[0115] Step 8. Figure A is divided into multiple sub-regions. It is assumed that the length and direction of the matching feature vectors in each region should be consistent. Therefore, a histogram of the length and angle of the matching feature vectors is constructed, and the mainstream length and angle are selected.

[0116] Step 9. RANSAC removes incorrect matching pairs.

[0117] Step 10. The reconstructed 3D points need to be within a certain range in front of the camera to filter out the correct matching feature point pairs.

[0118] By using the above method to initialize feature point matching through constrained feature matching, the tracking accuracy of SLAM can be improved.

[0119] In some embodiments described herein, the endoscopic catheter is tracked and located based on visual SLAM, and the target tissue or organ is reconstructed to obtain the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue or organ. This may include: determining the pose data of the endoscopic catheter in the current frame through a constant velocity motion model, a reference frame, or relocalization; establishing SLAM map data and selecting keyframes; optimizing the pose data of the keyframes and the SLAM map data using BA; optimizing the six degrees of freedom of the virtual endoscope using an optimization algorithm to maximize the similarity between the virtual endoscope image and the keyframe image; and recording the position of the virtual endoscope corresponding to the keyframe in the coordinate system corresponding to the three-dimensional model data.

[0120] Specifically, bronchoscopy tracking can include three threads: a front-end visual odometry thread, a back-end optimization and mapping thread, and a virtual bronchoscope pose optimization thread. The front-end visual odometry thread estimates the current frame pose using a constant velocity motion model, reference frames, or relocalization. This includes image feature extraction, feature matching, PnP solving for camera pose, triangulation to obtain the 3D spatial location of feature points, and keyframe determination and recording. The back-end optimization and mapping thread optimizes the camera pose and bronchial tree mapping of keyframes using BA. The virtual bronchoscope pose optimization thread uses optimization algorithms to optimize the six degrees of freedom of the virtual bronchoscope, maximizing the similarity between the virtual bronchoscope image and the keyframe image, and records the position of the virtual bronchoscope in the CT coordinate system corresponding to the keyframe.

[0121] Please refer to Figure 5 The diagram shows an endoscope tracking flowchart in an embodiment of the endoscope navigation and positioning method described in this specification. Figure 5 The following explanation uses bronchoscopy as an example. Figure 5 As shown, the three different tracing modes are invoked in the order of installation priority and whether tracing was successful.

[0122] The constant velocity motion model assumes that the camera moves at a constant speed. First, the estimated value of the pose Posecurr for the current frame is calculated based on the motion velocity Vlast and the pose Poselast of the previous frame: Posecurr = Vlast × Poselast, where Vlast represents the transformation from the previous frame to the current frame, i.e., the motion velocity. Then, pose optimization is performed to update the motion velocity Vcurr for the current frame: Vcurr = Posecurr × inverse(Poselast).

[0123] The reference frame pose estimation uses a bag-of-words approach to accelerate feature matching between the current frame and the reference frame. Reference frame pose estimation is initiated when the constant velocity motion model fails to track. Tracking failure is determined by whether the number of reconstructed map points exceeds a threshold.

[0124] When both pose estimation methods fail, relocalization is used to estimate the pose. First, candidate keyframes similar to the current frame are searched in the keyframe database using the bag-of-words method. Then, each candidate keyframe is traversed and its features are tracked, matched, and filtered to find matching points with the current frame using the bag-of-words method.

[0125] The current frame image needs to meet the image quality criteria, that is, the image is clear and the bronchial features are obvious.

[0126] There are two conditions for keyframe determination: in terms of time, the time since the previous keyframe cannot be too long; in terms of quality, the current frame must observe a sufficient number of map points, and the degree of overlap with the previous keyframes should not be too great.

[0127] Once the keyframe is successfully determined, two other threads are started to perform BA (Browser Alignment) to optimize the camera pose and virtual bronchoscope to optimize the pose, respectively.

[0128] In the above embodiments, performing fine-grained optimization of the real bronchoscope pose, mapping, and virtual bronchoscope pose only on keyframes can improve the robustness and positioning accuracy of the algorithm.

[0129] In some embodiments described herein, the pose data of the optimized keyframes and the SLAM map data, along with the optimization of the six degrees of freedom of the virtual endoscope using an optimization algorithm to maximize the similarity between the virtual endoscope image and the keyframe image, can be executed in a multi-threaded manner. In the above embodiments, multi-threaded parallel computation is performed, and thread locks are used to prevent interference between threads, thereby improving computational efficiency.

[0130] In some embodiments described herein, registering the coordinate system of the SLAM map data with the coordinate system of the 3D model data to obtain a transformation matrix may include one of the following methods: fusing the local organ 3D point cloud model in the SLAM map data with the global organ 3D point cloud model in the 3D model data to obtain a transformation matrix; fusing the pose data of the endoscopic catheter in the SLAM map data with the centerline data extracted from the 3D model data to obtain a transformation matrix; selecting at least four points on the local organ 3D point cloud model in the SLAM map data and the global organ 3D point cloud model in the 3D model data for registration to obtain a transformation matrix; or fusing the position of the endoscopic catheter where the keyframe is located in the coordinate system of the SLAM map data with the position of the virtual endoscope corresponding to the keyframe in the coordinate system of the 3D model data to obtain a point-to-point transformation matrix.

[0131] Please refer to Figure 6 The diagram shows a coordinate system registration flowchart in an endoscopic navigation and positioning method according to an embodiment of this specification. Figure 6 The following explanation uses bronchoscopy as an example. Figure 6As shown, coordinate system registration can take, but is not limited to, the following four forms: Form 1: Includes a local bronchial tree point cloud model A1 reconstructed using SLAM technology and a global bronchial tree point cloud model A2 reconstructed from CT image segmentation. The registration method uses the ICP correlation algorithm to fuse the local A1 point cloud with the global A2 point cloud to obtain the transformation relationship between the patient and the CT coordinate system. Form 2: Can include a camera trajectory B1 obtained from SLAM technology and a bronchial segmentation reconstruction and extraction of the centerline B2 from the CT image. The ICP correlation algorithm is used to fuse the local trajectory B1 with the global centerline B2 to obtain the transformation relationship between the patient and the CT coordinate system. Form 3: Includes a local bronchial tree point cloud model C1 reconstructed using SLAM technology and a global bronchial tree point cloud model C2 reconstructed from CT image segmentation. The physician manually selects >= 4 point pairs on models C1 and C2 for registration. Format 4: Record the position D1 of the real bronchoscope in the patient coordinate system for each keyframe, and optimize the six degrees of freedom of the virtual bronchoscope using the optimization algorithm of the virtual bronchoscope position optimization thread to maximize the similarity between the virtual bronchoscope image and the keyframe image. Record the position D2 of the virtual bronchoscope in the CT coordinate system corresponding to each keyframe, and fuse the point pairs D1 and D2 using the ICP algorithm to obtain the positions in the patient and CT coordinate systems. In the above embodiments, various optional patient-CT registration methods can be used for registration.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. For details, please refer to the foregoing descriptions of the relevant processing embodiments; they will not be repeated here.

[0133] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0134] The above method will be described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this specification and does not constitute an improper limitation of this specification.

[0135] This specific embodiment provides a bronchoscopy navigation and localization method based on visual SLAM. This embodiment does not add extra electromagnetic sensors, saving costs and eliminating the need for extensive time-consuming virtual endoscopic images (VB). It employs improved visual SLAM for real-time estimation of the bronchoscopy camera pose, resulting in faster speed and higher tracking and localization accuracy. This embodiment utilizes improved SLAM technology, employing stricter feature extraction criteria and constrained feature point matching to avoid mismatches. The improved SLAM technology used relies solely on monocular bronchoscopy images for camera pose calculation, mapping, and navigation, eliminating the need for binoculars and additional inertial sensors. Special initialization operations address scale loss during SLAM coordinate initialization. Multi-threaded parallel computation, with thread locks preventing interference, improves computational efficiency. Fine-grained optimization of the real bronchoscopy pose, mapping, and virtual bronchoscopy pose is performed only on keyframes, improving algorithm robustness and localization accuracy. Various optional patient-CT registration methods can be employed.

[0136] Please refer to Figure 7 This diagram illustrates the mechanical structure of a bronchoscope robot using an endoscopic navigation and positioning method according to one embodiment of this specification. Figure 7 As shown, the mechanical structure of the bronchoscopic robot mainly consists of the following four parts:

[0137] Trolley 701: The trolley is equipped with a robotic arm, and its internal space will house the robot control host and image computing platform, among other devices.

[0138] Robotic arm 702: The robotic arm is mounted on a trolley, and the end of the robotic arm is connected to a mounting plate. Its built-in driver can control the forward and backward movement of the tube in the trachea.

[0139] Mounting plate 703: The end of the robotic arm is equipped with a mounting plate with a slide rail. The built-in driver can control the forward and backward movement of the guide tube.

[0140] Catheter 704: The catheter contains at least one guidewire to control the bending of the catheter.

[0141] Please refer to Figure 8 This diagram illustrates a bronchoscopy scenario in an embodiment of the endoscopic navigation and positioning method described in this specification. Figure 8 As shown, a bronchoscopy scenario may include a bronchoscopic robot 801, a patient cart 802, and an image display and navigation device 803. The robot can control a robotic arm to insert a flexible tube from the patient's mouth or nose into the patient's lower respiratory tract for examination and treatment in the trachea and bronchi.

[0142] Please refer to Figure 9 The diagram shows the structure of a bronchoscope catheter. Figure 9 The left side shows a three-dimensional schematic diagram of the catheter. At least one guidewire is inserted inside the catheter, and the guidewire can be extended and shortened, so that the end of the catheter can be bent in at least one direction. Figure 9 The right side shows a schematic diagram of the cross-section of the catheter tip. (See diagram below.) Figure 9 As shown, a camera is placed at the end of the duct to capture images of the inside of the bronchus, with a light source on each side of the camera. Figure 10 A schematic diagram of the catheter tip imaging is shown.

[0143] Please refer to Figure 11 A flowchart of an endoscopic navigation and positioning method according to an embodiment of this specification is shown. Figure 11 As shown, the visual SLAM-based bronchoscopy navigation and localization method in this embodiment may include the following steps: Initialization, including SLAM coordinate initialization and initialization of the patient-CT coordinate system transformation relationship. Real bronchoscopy tracking, using improved SLAM technology for real-time estimation of bronchoscopic pose during surgery and reconstruction of the point cloud model of the bronchial tree. Patient-CT registration, used to obtain the transformation relationship between the real bronchoscopic image and the CT image. Virtual bronchial navigation, providing the physician with virtual bronchoscopic images to guide intraoperative navigation, and providing the physician with a virtual bronchial tree to display the duct position and orientation in real time.

[0144] This specific embodiment also provides a bronchoscope navigation and positioning device based on visual SLAM, including an image acquisition module, a navigation and positioning algorithm module, a motion control module, and a bronchoscope robot.

[0145] The image acquisition module is used to capture real bronchoscopic images in real time and to determine the image quality.

[0146] The navigation and positioning algorithm module includes image quality assessment, tracking and positioning, 3D registration, and virtual bronchoscopy image generation.

[0147] The motion control module is used to control the forward movement and bending of the bronchoscope according to the doctor's operating instructions or according to the preset motion plan.

[0148] Bronchoscopy robots are used to perform bronchoscopy procedures.

[0149] In one embodiment of this specification, the coordinate initialization of SLAM may include the following steps: inserting a bronchoscope into the main airway, after manual confirmation by a doctor, the motion control module begins to execute a preset action to advance the bronchus a fixed distance m, the image acquisition module captures two frames of images before and after the motion execution, obtains the correspondence between feature points between the 2D and 2D images, solves the camera rotation matrix and displacement vector between the two frames of images according to the epipolar geometry principle, because the displacement vector is a normalized vector and loses scale information, multiplying the displacement vector by the value m gives the true displacement vector, thereby obtaining the initial pose of the bronchoscope, and then performing triangulation to obtain the 3D position of the feature points.

[0150] In one embodiment of this specification, the initialization of the patient-CT coordinate system transformation relationship may include: traversing the bronchoscope through the main airway and secondary branches or deeper tertiary branches, using SLAM to track and locate the bronchoscope and reconstruct the bronchial tree point cloud model, and using a patient-CT registration method to obtain the initial transformation matrix between the two.

[0151] In one embodiment of this specification, the SLAM technology used for bronchoscopy tracking may include three threads: a front-end visual odometry thread, which estimates the pose of the current frame through a constant velocity motion model, reference frame, or relocalization, specifically including image feature extraction, feature matching, PnP to solve for camera pose, triangulation to obtain the 3D spatial position of feature points, and keyframe determination and recording; a back-end optimization and mapping thread, which optimizes the camera pose and bronchial tree mapping of keyframes using BA; and a virtual bronchoscope position optimization thread, which uses optimization algorithms to optimize the six degrees of freedom of the virtual bronchoscope to maximize the similarity between the virtual bronchoscope image and the keyframe image, and records the position of the virtual bronchoscope in the CT coordinate system corresponding to the keyframe.

[0152] In one embodiment of this specification, the improved SLAM technology can achieve more accurate 3D-2D feature point matching through stricter feature extraction criteria, constrained 3D feature point spatial range, and constrained 2D feature point image range.

[0153] In some embodiments of this specification, the constrained feature matching process may include the following steps:

[0154] Step 1. Take any feature fa from image A, and expand a certain area centered on the position of fa to define the range for searching for matching features in image B.

[0155] Step 2. Take the feature descriptor of feature fa and calculate its Hamming distance d to all feature descriptors within range r in image B.

[0156] Step 3. Sort the obtained distances and select the features with the shortest and second shortest distances, denoted as d1 and d2.

[0157] Step 4. Nearest Neighbor and Second Nearest Neighbor Distance Ratio: Take the feature fa of image A and search for the two features that are closest to it to image B. The distances are denoted as d1 and d2 respectively. Only when d1 / d2 is less than a certain threshold is it considered a good match.

[0158] Step 5. Cross-checking: If feature fb is the best match for feature fa in image B, and feature fa is also the best match for feature fb in image A, then fa and fb are considered a good match.

[0159] Step 6. Build an angle histogram based on the angle difference of the matched feature points, and select the mainstream angle of the histogram.

[0160] Step 7. Find the minimum matching distance and select only matching point pairs whose distance is n times greater than the minimum distance.

[0161] Step 8. Figure A is divided into multiple sub-regions. It is assumed that the length and direction of the matching feature vectors in each region should be consistent. Therefore, a histogram of the length and angle of the matching feature vectors is constructed, and the mainstream length and angle are selected.

[0162] 9. RANSAC removes incorrect matching pairs.

[0163] 10. The reconstructed 3D points need to be within a certain range in front of the camera to select the correct matching feature point pairs.

[0164] Figure 12 A schematic diagram of feature matching for an endoscopic navigation and positioning method according to one embodiment of this specification is shown. Figure 12 The above figure exemplifies a schematic diagram of the connection between two images for feature matching. Figure 12 The figure below shows an example feature matching vector.

[0165] In one embodiment of this specification, patient-CT registration may include, but is not limited to, the following four forms: Form 1: including a local bronchial tree point cloud model A1 reconstructed using SLAM technology and a global bronchial tree point cloud model A2 reconstructed from CT image segmentation. The registration method uses an ICP correlation algorithm to fuse the local A1 point cloud with the global A2 point cloud to obtain the transformation relationship between the patient and the CT coordinate system; Form 2: may include a camera trajectory B1 obtained using SLAM technology and a bronchial segmentation reconstruction and extraction of the centerline B2 from the CT image. The ICP correlation algorithm is used to fuse the local trajectory B1 with the global centerline B2 to obtain the transformation relationship between the patient and the CT coordinate system; Formula 3: Includes a local bronchial tree point cloud model C1 reconstructed using SLAM technology and a global bronchial tree point cloud model C2 reconstructed from CT image segmentation. The doctor manually selects >= 4 point pairs on models C1 and C2 for registration. Formula 4: Record the position D1 of the real bronchoscope in the patient coordinate system for each keyframe, and optimize the 6 degrees of freedom of the virtual bronchoscope using the optimization algorithm of the virtual bronchoscope position optimization thread to maximize the similarity between the virtual bronchoscope image and the keyframe image. Record the position D2 of the virtual bronchoscope in the CT coordinate system corresponding to each keyframe. The point pairs D1 and D2 are fused using the ICP algorithm to obtain the positions in the patient and CT coordinate systems.

[0166] Figure 13 A flowchart illustrating the virtual bronchoscope pose optimization process of an endoscopic navigation and positioning method according to one embodiment of this specification is shown. Figure 13 As shown, virtual bronchoscopy pose optimization can include the following: Camera pose includes camera position and orientation. The orientation can be represented by Euler angles, quaternions, or by the camera's focal point and upward direction. The initial camera position P2 can be obtained by combining the position P1 obtained from the real bronchoscopy tracking and positioning algorithm with the patient-CT registration transformation matrix T, where P2 = T × P1. The initial camera orientation is the same as that obtained from the real bronchoscopy tracking and positioning algorithm. Virtual bronchoscopy images can be generated using OpenGL or VTK for rendering. Please refer to [reference needed]. Figure 14 The diagram shows a schematic image of a bronchoscope in this embodiment. Figure 14 The left image in the image is a real bronchoscope image, while the right image is a virtual bronchoscope image generated using OpenGL or VTK rendering.

[0167] It can render bronchial tree models obtained from CT segmentation, set colors, and place cameras and light sources, etc. Please refer to [reference needed]. Figure 15 The diagram shows a schematic representation of the bronchial tree in this embodiment. Figure 15The left side of the image shows a bronchial tree model reconstructed from CT data segmentation, while the middle side shows a bronchial tree point cloud model, which can be a point cloud reconstructed using SLAM technology or a point cloud reconstructed from CT data segmentation. Figure 15 The diagram on the right shows the bronchial tree centerline or camera trajectory.

[0168] Similarity calculation involves determining the similarity between a virtual bronchoscope image (VB) and a real bronchoscope image (RB). The pose of the virtual camera corresponding to the VB with the highest similarity is considered the optimized pose. The similarity metric can be a combination of mutual information, brightness, contrast, and structural features. The optimization algorithm can use a non-differentiable algorithm. The goal is to iteratively generate different VBs based on the camera pose according to the optimization strategy, calculate their similarity with the RB, and obtain the VB with the highest similarity and its pose. This step is performed in a separate thread, without affecting the calculation of other threads, and is only initiated after keyframes are acquired. After completion, it enters a waiting state.

[0169] In some embodiments of this specification, virtual bronchoscopic images can be provided to the physician to guide intraoperative navigation, and a virtual bronchial tree can be provided to display the position and orientation of the catheter in real time. Please refer to... Figure 16 This illustrates a schematic diagram of virtual bronchial navigation in this specific embodiment. Figure 16 The left image shows a schematic diagram of virtual bronchial tree navigation, with black lines indicating the position and orientation of the duct. Figure 16 The right image in the diagram shows a schematic diagram of virtual bronchoscopy image navigation. The black lines in the diagram indicate the path to reach the target point.

[0170] The bronchial tracking and localization algorithm in this specific embodiment can be applied to the navigation and localization of bronchoscopic robots, as well as to the navigation and localization of other endoscopes in other tissues of the human body, such as the intestines, pharynx, and stomach. In this specific embodiment, no additional electromagnetic sensors are added, saving costs; a series of virtual endoscopic images (VB) are not needed for each frame of bronchoscopic localization, saving computation time; an improved visual SLAM is used to estimate the bronchoscope camera pose in real time, resulting in faster speed and higher tracking and localization accuracy; the improved visual SLAM only requires a monocular image as input, eliminating the need for binocular images or other sensors, thus reducing system complexity while maintaining accuracy; the bronchial tree model reconstructed by SLAM has realistic bronchial colors, providing doctors with a more intuitive 3D effect during surgery.

[0171] Based on the same inventive concept, this specification also provides an endoscopic navigation and positioning device, as described in the following embodiments. Since the principle by which the endoscopic navigation and positioning device solves the problem is similar to that of the endoscopic navigation and positioning method, the implementation of the endoscopic navigation and positioning device can refer to the implementation of the endoscopic navigation and positioning method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated. Figure 17 This is a structural block diagram of an endoscopic navigation and positioning device according to an embodiment of this specification, such as... Figure 17 As shown, it includes: acquisition module 171, reconstruction module 172 and generation module 173. The structure is described below.

[0172] The acquisition module 171 is used to acquire endoscopic images of the target tissue or organ; the endoscopic images include a first image acquired by a monocular camera installed in the endoscope before the endoscope performs a preset action and a second image acquired after the preset action is performed.

[0173] The reconstruction module 172 is used to determine the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue organ based on the first image and the second image using visual SLAM.

[0174] The generation module 173 is used to acquire the three-dimensional model data of the target tissue organ; it is also used to register the coordinate system of the SLAM map data with the coordinate system of the three-dimensional model data to generate virtual endoscopic navigation data based on the pose data of the endoscopic catheter.

[0175] In some embodiments of this specification, the preset actions include rotational actions and translational actions.

[0176] In some embodiments of this specification, the reconstruction module may be specifically used to: initialize visual SLAM coordinates using the first image and the second image; track and locate the endoscopic catheter based on visual SLAM and reconstruct the map of the target tissue organ to obtain the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue organ.

[0177] In some embodiments of this specification, the generation module may be specifically used to: register the coordinate system of the SLAM map data with the coordinate system of the three-dimensional model data to obtain a transformation matrix; and use the transformation matrix to transform the pose data into the coordinate system corresponding to the three-dimensional model data to generate virtual endoscope navigation data.

[0178] In some embodiments of this specification, the initialization module may be specifically used for: extracting features from a first image to obtain a first feature point; extracting features from a second image to obtain a second feature point; matching the first feature point and the second feature point to obtain a matching point pair; generating a rotation matrix and a unit translation vector corresponding to the monocular camera based on the matching point pair; calculating a displacement vector based on the displacement length data in the preset action and the unit translation vector; generating map points corresponding to the matching point pair using the rotation matrix and the displacement vector; and determining that the visual SLAM coordinate initialization is successful when the number of map points is greater than a preset number.

[0179] In some embodiments described herein, feature extraction of a first image to obtain a first feature point may include: downsampling the first image to generate N pyramid images at different scales; where N is an integer greater than 2; extracting feature points from the N pyramid images at different scales; filtering the extracted feature points using a quadtree to obtain a first feature point uniformly distributed in the image; calculating the gray-level centroid of the first feature point, and calculating the orientation angle of the first feature point based on the position of the gray-level centroid and the position of the first feature point; and calculating the feature descriptor of the first feature point to represent the image information around the first feature point in binary form.

[0180] In some embodiments described herein, matching the first feature point and the second feature point to obtain a matching point pair may include: taking a first feature point A in the first image, and selecting a preset region centered on the first feature point A as a specified range for searching matching features in the second image; calculating the Hamming distance between the feature descriptor of the first feature point A and the feature descriptor of the second feature point within the specified range in the second image; sorting the calculated Hamming distances in descending order to obtain the shortest Hamming distance and the second shortest Hamming distance; when the ratio between the shortest Hamming distance and the second shortest Hamming distance is less than a preset ratio, determining that the first feature point A matches the second feature point B corresponding to the shortest Hamming distance; for the second feature point B, determining whether the second feature point B matches the first feature point A; when the second feature point B matches the first feature point A, determining that the first feature point A and the second feature point B are a matching point pair; establishing an angle histogram and a length histogram based on the matching point pair, and selecting the mainstream angle and mainstream length; using the RANSAC algorithm based on the mainstream angle and mainstream length to eliminate erroneous matching point pairs and obtain valid matching point pairs.

[0181] In some embodiments described herein, the initialization module may also be used to: determine the pose data of the endoscopic catheter in the current frame through a constant velocity motion model, a reference frame, or relocation; establish SLAM map data; and select key frames; optimize the pose data of the key frames and the SLAM map data using a BA (Balanced Assay) algorithm; optimize the six degrees of freedom of the virtual endoscope using an optimization algorithm to maximize the similarity between the virtual endoscope image and the key frame image; and record the position of the virtual endoscope corresponding to the key frame in the coordinate system corresponding to the three-dimensional model data.

[0182] In some embodiments described herein, the pose data of the BA-optimized keyframes and the SLAM map data, along with the optimization of the six degrees of freedom of the virtual endoscope using an optimization algorithm to maximize the similarity between the virtual endoscope image and the keyframe image, can be executed in a multi-threaded manner.

[0183] In some embodiments described herein, the generation module may be used for one of the following: fusing the local tissue / organ 3D point cloud model in the SLAM map data with the global tissue / organ 3D point cloud model in the 3D model data to obtain a transformation matrix; fusing the pose data of the endoscopic catheter in the SLAM map data with the centerline data extracted from the 3D model data to obtain a transformation matrix; selecting at least four points on the local tissue / organ 3D point cloud model in the SLAM map data and the global tissue / organ 3D point cloud model in the 3D model data for registration to obtain a transformation matrix; and fusing the position of the endoscopic catheter where the keyframe is located in the coordinate system of the SLAM map data with the position of the virtual endoscope corresponding to the keyframe in the coordinate system corresponding to the 3D model data to obtain a point-to-point transformation matrix.

[0184] As can be seen from the above description, the embodiments of this specification achieve the following technical effects: Based on visual SLAM for localization and navigation, coordinate initialization, camera pose calculation, mapping, and navigation can be achieved solely using monocular endoscopic images. This results in faster speed and higher tracking and positioning accuracy. Furthermore, it eliminates the need for binoculars and additional electromagnetic or inertial sensors, reducing system complexity and saving costs while maintaining accuracy. Virtual endoscopic navigation data can provide surgeons with a more intuitive 3D experience during surgery, facilitating endoscopic procedures, improving accuracy and efficiency, reducing surgical difficulty, and enhancing patient experience.

[0185] Based on the same inventive concept, this specification also provides an endoscope navigation and positioning system in the embodiments, as described in the following embodiments. Figure 18 This is a structural block diagram of an endoscopic navigation and positioning device according to an embodiment of this specification, such as... Figure 18 As shown, it includes an endoscope 181, an image processor 182, and an image display 183.

[0186] The endoscope 181 is equipped with an illumination source and a monocular camera at the distal end of its catheter; the monocular camera is used to acquire endoscopic images of the target tissues and organs during endoscopic surgery.

[0187] The image processor 182 is used to acquire endoscopic images from the endoscope; the endoscopic images include a first image captured by the monocular camera before the endoscope performs a preset action and a second image captured after the preset action is performed; it is also used to determine the pose data of the endoscopic catheter and the SLAM map data corresponding to the target tissue organ based on visual SLAM using the first image and the second image; it is also used to acquire three-dimensional model data of the target tissue organ; and to register the coordinate system of the SLAM map data with the coordinate system of the three-dimensional model data to generate virtual endoscopic navigation data based on the pose data of the endoscopic catheter.

[0188] The image display 183 is used to acquire the virtual endoscope navigation data from the image processor and to display images based on the virtual endoscope navigation data.

[0189] This specification also provides a medical device, which can be found in the following description. Figure 19 The diagram shown illustrates the structural composition of a medical device based on the endoscopic navigation and positioning method provided in the embodiments of this specification. Specifically, the medical device may include an input device 191, a processor 192, and a memory 193. The memory 193 stores processor-executable instructions. When the processor 192 executes the instructions, it implements the steps of the endoscopic navigation and positioning method described in any of the above embodiments.

[0190] In this embodiment, the input device can specifically be one of the main devices for information exchange between the user and the computer system. The input device may include a keyboard, mouse, camera, scanner, light pen, handwriting input tablet, voice input device, etc.; the input device is used to input raw data and programs for processing these data into the computer. The input device can also receive data transmitted from other modules, units, and devices. The processor can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. The memory can specifically be a memory device used to store information in modern information technology. The memory can include multiple layers; in digital systems, anything that can store binary data can be considered memory; in integrated circuits, a circuit without physical form but with storage function is also called memory, such as RAM, FIFO, etc.; in a system, a storage device with physical form is also called memory, such as a memory stick, TF card, etc.

[0191] The specific functions and effects of the medical device in this embodiment can be explained by comparison with other embodiments, and will not be repeated here.

[0192] This specification also provides a computer storage medium based on an endoscopic navigation and positioning method in its embodiments. The computer storage medium stores computer program instructions that, when executed, implement the steps of the endoscopic navigation and positioning method described in any of the above embodiments.

[0193] In this embodiment, the storage medium includes, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured according to standards specified in the communication protocol for network connection communication.

[0194] In this embodiment, the specific functions and effects implemented by the program instructions stored in the computer storage medium can be explained by comparison with other embodiments, and will not be repeated here.

[0195] Obviously, those skilled in the art will understand that the modules or steps of the embodiments described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this specification are not limited to any particular combination of hardware and software.

[0196] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this specification should not be determined by reference to the above description, but rather by reference to the foregoing claims and the full scope of their equivalents.

[0197] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to the embodiments described herein by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A medical device, characterized by The method comprises a processor and a memory for storing processor-executable instructions, the processor executes the instructions to implement the steps of the endoscope navigation positioning method, the endoscope navigation positioning method comprises: An endoscope image of a target tissue organ is acquired; the endoscope image comprises a first image collected by a monocular camera arranged in an endoscope before a preset action is performed and a second image collected after the preset action is performed; the preset action comprises a rotation action and a translation action; Visual SLAM coordinate initialization is performed using the first image and the second image, and pose data of an endoscope catheter and SLAM map data corresponding to the target tissue organ are determined based on visual SLAM using the first image and the second image; wherein, when the visual SLAM coordinate initialization is performed using the first image and the second image, a camera rotation matrix and a unit displacement vector between the first image and the second image are solved according to an epipolar geometry principle, the unit displacement vector is multiplied by displacement distance data of the translation action in the preset action to obtain a displacement vector; Three-dimensional model data of the target tissue organ is acquired; a coordinate system of the SLAM map data is registered with a coordinate system of the three-dimensional model data to generate virtual endoscope navigation data based on the pose data of the endoscope catheter.

2. The medical device of claim 1, wherein, The pose data of the endoscope catheter and the SLAM map data corresponding to the target tissue organ are determined based on visual SLAM using the first image and the second image, comprising: The endoscope catheter is tracked and positioned and the target tissue organ is map reconstructed based on visual SLAM to obtain the pose data of the endoscope catheter and the SLAM map data corresponding to the target tissue organ.

3. The medical device of claim 1, wherein, The coordinate system of the SLAM map data is registered with the coordinate system of the three-dimensional model data to generate virtual endoscope navigation data based on the pose data of the endoscope catheter, comprising: The coordinate system of the SLAM map data is registered with the coordinate system of the three-dimensional model data to obtain a transformation matrix; The pose data is converted into a coordinate system corresponding to the three-dimensional model data using the transformation matrix to generate virtual endoscope navigation data.

4. The medical device of claim 2, wherein, The visual SLAM coordinate initialization is performed using the first image and the second image, comprising: Feature extraction is performed on the first image to obtain first feature points, and feature extraction is performed on the second image to obtain second feature points; Matching is performed on the first feature points and the second feature points to obtain a matching point pair; Based on the matching point pair, a rotation matrix and a unit translation vector corresponding to the monocular camera are generated; a displacement vector is calculated based on displacement length data in the preset action and the unit translation vector; The matching point pair corresponds to a map point generated using the rotation matrix and the displacement vector; When the number of map points is greater than a preset number, it is determined that the visual SLAM coordinate initialization is successful.

5. The medical device of claim 4, wherein, The matching is performed on the first feature points and the second feature points to obtain a matching point pair, comprising: Taking a first feature point A in the first image as a center, a preset region is selected as a specified range for searching a matching feature in the second image; A Hamming distance between a feature descriptor of the first feature point A and a feature descriptor of a second feature point in the specified range in the second image is calculated, and the calculated Hamming distances are arranged in descending order to obtain a shortest Hamming distance and a second shortest Hamming distance; When a ratio between the shortest Hamming distance and the second shortest Hamming distance is less than a preset ratio, it is determined that the first feature point A and a second feature point B corresponding to the shortest Hamming distance are matched, and whether the second feature point B and the first feature point A are matched is determined for the second feature point B; When it is determined that the second feature point B and the first feature point A are matched, it is determined that the first feature point A and the second feature point B are a matched point pair; Based on the matched point pair, an angle histogram and a length histogram are established, and a mainstream angle and a mainstream length are selected; An RANSAC algorithm is used to eliminate incorrect matched point pairs based on the mainstream angle and the mainstream length, and valid matched point pairs are obtained.

6. The medical device of claim 2, wherein, The endoscope catheter is tracked and positioned based on visual SLAM, and the target tissue organ is map reconstructed, to obtain pose data of the endoscope catheter and SLAM map data corresponding to the target tissue organ, including: The pose data of the endoscope catheter of the current frame is determined through a constant speed motion model, a reference frame or repositioning, the SLAM map data is established, and a key frame is selected; The pose data of the key frame and the SLAM map data are optimized by BA; An optimization algorithm is used to optimize 6 degrees of freedom of the virtual endoscope, so that the similarity between the virtual endoscope image and the key frame image is maximum, and the position of the virtual endoscope corresponding to the key frame in the coordinate system corresponding to the three-dimensional model data is recorded.

7. The medical device of claim 6, wherein, The BA optimizes the pose data of the key frame and the SLAM map data, and the optimization algorithm is used to optimize 6 degrees of freedom of the virtual endoscope, so that the similarity between the virtual endoscope image and the key frame image is maximum, which are executed in separate threads.

8. The medical device of claim 3, wherein, The coordinate system of the SLAM map data is registered with the coordinate system of the three-dimensional model data to obtain a transformation matrix, including one of the following modes: Point cloud fusion is performed on the local tissue organ three-dimensional point cloud model in the SLAM map data and the global tissue organ three-dimensional point cloud model in the three-dimensional model data to obtain a transformation matrix; The pose data of the endoscope catheter in the SLAM map data is fused with the centerline data extracted from the three-dimensional model data to obtain a transformation matrix; At least four points are selected on the local tissue organ three-dimensional point cloud model in the SLAM map data and the global tissue organ three-dimensional point cloud model in the three-dimensional model data for registration to obtain a transformation matrix; Point pair fusion is performed on the position of the endoscope catheter corresponding to the key frame in the coordinate system of the SLAM map data and the position of the virtual endoscope corresponding to the key frame in the coordinate system corresponding to the three-dimensional model data to obtain a transformation matrix.

9. An endoscopic navigation positioning device, characterized by, including: The acquisition module is configured to acquire an endoscope image of a target tissue organ; the endoscope image comprises a first image captured by a monocular camera arranged in an endoscope before the endoscope performs a preset action and a second image captured after the preset action is performed; the preset action comprises a rotation action and a translation action; The reconstruction module is configured to perform visual SLAM coordinate initialization using the first image and the second image, determine pose data of the endoscope catheter and SLAM map data corresponding to the target tissue organ based on visual SLAM using the first image and the second image; wherein the reconstruction module is specifically configured to, when performing visual SLAM coordinate initialization using the first image and the second image, solve a camera rotation matrix and a unit displacement vector between the first image and the second image according to an epipolar geometry principle, multiply the unit displacement vector by displacement distance data of the translation action in the preset action to obtain a displacement vector; The generation module is configured to acquire three-dimensional model data of the target tissue organ; and further configured to register a coordinate system of the SLAM map data and a coordinate system of the three-dimensional model data to generate virtual endoscope navigation data based on the pose data of the endoscope catheter.

10. An endoscopic navigation positioning system, characterized by, Comprise: An endoscope, a catheter end of the endoscope is provided with an illumination light source and a monocular camera; the monocular camera is configured to capture an endoscope image of a target tissue organ; An image processor is configured to acquire the endoscope image from the endoscope; the endoscope image comprises a first image captured by the monocular camera before the endoscope performs a preset action and a second image captured after the preset action is performed; the preset action comprises a rotation action and a translation action; Further configured to perform visual SLAM coordinate initialization using the first image and the second image, determine pose data of the endoscope catheter and SLAM map data corresponding to the target tissue organ based on visual SLAM using the first image and the second image; further configured to acquire three-dimensional model data of the target tissue organ; register a coordinate system of the SLAM map data and a coordinate system of the three-dimensional model data to generate virtual endoscope navigation data based on the pose data of the endoscope catheter; wherein when performing visual SLAM coordinate initialization using the first image and the second image, the image processor is configured to solve a camera rotation matrix and a unit displacement vector between the first image and the second image according to an epipolar geometry principle, multiply the unit displacement vector by displacement distance data of the translation action in the preset action to obtain a displacement vector; An image display is configured to acquire the virtual endoscope navigation data from the image processor and perform image display based on the virtual endoscope navigation data.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, the instructions are executed to implement the steps of the endoscope navigation positioning method, the endoscope navigation positioning method comprises: An endoscope image of a target tissue organ is acquired; the endoscope image includes a first image captured by a monocular camera arranged in the endoscope before a preset action is performed by the endoscope and a second image captured after the preset action is performed; the preset action includes a rotation action and a translation action; Visual SLAM coordinate initialization is performed using the first image and the second image; pose data of an endoscope catheter and SLAM map data corresponding to the target tissue organ are determined based on visual SLAM using the first image and the second image; wherein, when the visual SLAM coordinate initialization is performed using the first image and the second image, a camera rotation matrix and a unit displacement vector between the first image and the second image are solved according to the epipolar geometry principle, and the unit displacement vector is multiplied by displacement distance data of the translation action in the preset action to obtain a displacement vector; Three-dimensional model data of the target tissue organ is acquired; a coordinate system of the SLAM map data is registered with a coordinate system of the three-dimensional model data to generate virtual endoscope navigation data based on the pose data of the endoscope catheter.

Images