Image registration method and system based on virtual bronchoscope navigation

Abstract

The invention discloses an image registration method and system based on virtual bronchoscope navigation, and the method comprises the steps: generating a navigation path of a target nodule through lung segment segmentation according to the target nodule and a preset bronchus three-dimensional model; collecting a real-time image at a preset sampling point in the navigation path by moving a bronchoscope, and converting the real-time image into a depth map; and registering the depth map and the bronchus three-dimensional model through a rough matching method and a fine matching method to obtain a registration result. The position of the bronchoscope is adjusted through image matching, accurate navigation of the bronchoscope is achieved, and the operation cost is reduced.

Description

Technical Field

[0001] This application belongs to the technical field of medical devices, and particularly relates to an image registration method and system based on virtual bronchoscope navigation. Background Art

[0002] As a tool to assist doctors in the treatment of lung nodule diseases, a bronchoscope is usually manually controlled by a doctor to move in the trachea of a patient's lungs with an image acquisition device, collect lung data, and match the actual position of the bronchoscope with a three-dimensional model of the tracheal tree reconstructed for the patient's lungs through the collected image data to achieve precise positioning of the device. However, this data acquisition method is cumbersome to operate, increases the operation burden on doctors during surgery, and affects the surgical efficiency.

[0003] Existing bronchoscope automation technologies mainly include two types. One is electromagnetic navigation, which uses electromagnetic technology to achieve high-precision real-time positioning function, can provide accurate navigation to the target area for physicians during bronchoscopy, and this technology can provide real-time feedback during the operation to assist physicians in adjusting the navigation path in a timely manner. However, electromagnetic navigation has limitations, can only be used in specific operating environments, is restricted by the installation position of the device, and the device and its maintenance costs are relatively high. The other is virtual navigation, which is mainly based on software, can run on multiple hardware platforms, has higher flexibility, and the development and maintenance costs are relatively low. However, the performance of virtual navigation depends to a large extent on the quality of the input medical images, and poor image quality may affect the navigation effect. Therefore, compared with electromagnetic navigation, the positioning accuracy of virtual navigation is slightly insufficient. Summary of the Invention

[0004] This application proposes an image registration method and system based on virtual bronchoscope navigation, which adjusts the position of the bronchoscope through image matching to achieve precise navigation of the bronchoscope and reduce the operation cost.

[0005] The first aspect of this application provides an image registration method based on virtual bronchoscope navigation, and the method includes: Generating a navigation path for the target nodule through lung segment segmentation according to the target nodule and a preset three-dimensional bronchial model; By moving the bronchoscope, collecting real-time images at preset sampling points in the navigation path, and converting the real-time images into depth maps; Registering the depth map with the three-dimensional bronchial model through a rough matching method and a fine matching method to obtain a registration result.

[0006] The above solution first performs lung segment segmentation on the three-dimensional bronchial model to quickly locate the lung segment where the target nodule is located, and then selects the most suitable navigation path according to the position of the target nodule, ensuring that the navigation path definitely includes the target nodule, thus improving the accuracy of path planning. Then, by moving the bronchoscope along the navigation path to collect real-time images and converting the images into depth maps, the computational complexity for subsequent image registration is reduced. Finally, through the combination of two matching methods, namely rough matching and fine matching, the depth map collected in real time is compared with the virtual image provided by the three-dimensional bronchial model to obtain the registration result for confirming whether the bronchoscope deviates from the navigation path, and timely adjusting the position of the bronchoscope to achieve precise navigation of the bronchoscope. Moreover, the above process of image registration does not need to be carried out in a specific operating environment, has no equipment installation restrictions, and reduces the operating cost.

[0007] In a possible implementation method of the first aspect, according to the target nodule and the preset three-dimensional bronchial model, a navigation path for the target nodule is generated through lung segment segmentation, specifically as follows: Based on pulmonary medical imaging, the three-dimensional bronchial model and the model starting point are generated, and the centerline of the three-dimensional bronchial model is extracted; According to the centerline, all feasible paths from the model starting point to the target nodule are extracted from the three-dimensional bronchial model; Based on the bifurcation points in the feasible paths, the lung segment where the target nodule is located and the navigation path including the lung segment are determined through lung segment segmentation.

[0008] The above solution first scans the lungs to generate a three-dimensional bronchial model, and then extracts the centerline of the model for planning the path of image sampling. First, all feasible paths from the model starting point to the target nodule are determined according to the centerline, and then the model is segmented by determining the bifurcation points in the path to accurately locate the lung segment where the target nodule is located. Thus, the shortest path including the lung segment where the target nodule is located can be selected as the navigation path, avoiding incorrect paths obtained due to blindly pursuing the shortest path.

[0009] In a possible implementation method of the first aspect, by moving the bronchoscope, real-time images at preset sampling points in the navigation path are collected, and the real-time images are converted into depth maps, specifically as follows: According to the preset spacing, the sampling points in the navigation path are marked; wherein, the positions of the sampling points include the straight segments, curved segments, and bifurcation points of the three-dimensional bronchial model; Starting from the starting point of the three-dimensional bronchial model, the bronchoscope is controlled to move along the navigation path, and real-time images at the sampling points are collected; The first images that meet the preset quality requirements are screened out from the real-time images, and the first images are converted into depth maps through the adversarial network model.

[0010] In the above solution, corresponding sampling points are first set on the navigation path at a preset interval. The evenly distributed sampling points can ensure that the subsequent image sampling of the bronchoscope can cover the entire path, providing sufficient data support for image registration. Then, starting from the starting point of the model, the bronchoscope is moved for image sampling, and the obtained high-quality images are converted into depth maps to reduce the computational complexity for subsequent image registration.

[0011] In a possible implementation method of the first aspect, converting the first image into a depth map through an adversarial network model further includes: Taking the sampling point corresponding to the first image as the center point, determining the radius based on the position of the center point in the navigation path, and generating a conical base; According to a preset angle range, deflecting four different random angles relative to the center point on the conical base to obtain four quadrant points; Converting the first image into the depth map corresponding to the center point through an adversarial network model, and adjusting the angle, image start position, and image end position of the depth map according to the quadrant points to generate the depth maps corresponding to each quadrant point, thereby obtaining all the depth maps corresponding to the first image.

[0012] In the above solution, since the real-time image of a single angle collected by the bronchoscope can provide limited information, which has a negative impact on the accuracy of subsequent image registration, it is necessary to set multiple evenly distributed quadrant points for the sampling points to expand the observation angle of the bronchoscope to obtain more information. By deflecting the sampling points at random angles, four quadrant points in different orientations are obtained, and then the angle of the depth map is adjusted according to the quadrant points to obtain depth maps with multiple perspectives, improving the diversity and comprehensiveness of the images. The obtained depth maps with different angles are closer to the observation mode of the bronchoscope, providing sufficient data support for subsequent image registration.

[0013] In a possible implementation method of the first aspect, registering the depth map with the three-dimensional bronchial model through a rough matching method and a fine matching method to obtain a registration result, specifically: According to the depth map, adjusting the camera parameters of the three-dimensional bronchial model through a fine matching method; Generating a virtual bronchial image according to the camera parameters; Performing rough matching on the depth map and the virtual bronchial image to obtain a registration result.

[0014] The above solution determines whether the position of the bronchoscope is consistent with the position of the bronchial three-dimensional model through rough matching to ensure that the bronchoscope does not deviate from the preset navigation path. On the basis of rough matching, the camera position of the bronchial three-dimensional model is further adjusted through fine matching to further improve the visual consistency of the virtual bronchial image provided by the model and ensure the accuracy of the rough matching result.

[0015] In a possible implementation method of the first aspect, according to the depth map, the camera parameters of the bronchial three-dimensional model are adjusted by the fine matching method, specifically: Obtain the real-time reference image provided by the bronchial three-dimensional model; Calculate the hash values of the real-time reference image and the depth map to obtain the image matching accuracy; When the image matching accuracy is less than the first threshold, adjust the camera position of the bronchial three-dimensional model according to the depth map; When the image matching accuracy is not less than the first threshold, according to the camera position of the bronchial three-dimensional model, Update the current moving direction and the current bifurcation point of the bronchoscope in the depth map through the tracking algorithm.

[0016] In a possible implementation method of the first aspect, the depth map and the virtual bronchial image are roughly matched to obtain a registration result, specifically: Respectively perform feature extraction on the depth map and the virtual bronchial image through a preset convolutional neural network to obtain the first feature vector of the depth map and the second feature vector of the virtual bronchial image; According to the first feature vector and the second feature vector, calculate the feature similarity between the depth map and the virtual bronchial image, and screen the virtual bronchial image according to the feature similarity to obtain the second threshold number of reference images; Calculate the image hash value of each reference image by the block-based average hash algorithm for the brightness value of the reference image; Perform weighted calculation on the image hash value of each reference image and the feature similarity through the similarity calculation formula to obtain the similarity of each reference image; Count the total number of the reference images with the similarity greater than the third threshold to obtain the registration result; Wherein, when the total number exceeds the fourth threshold, the registration result is that the current position of the bronchoscope is consistent with the camera position of the bronchial three-dimensional model.

[0017] The above solution first extracts features from the depth map and the virtual bronchus image, and calculates the feature similarity between the two images through the obtained feature vectors. Then, the reference images meeting a certain accuracy are screened out through the feature similarity, reducing the subsequent calculation amount. Then, by performing weighted calculation on the image hash value and the feature similarity, the similarity of each reference image relative to the virtual bronchus image is obtained. When the number of reference images meeting a certain similarity exceeds a certain value, it indicates that the position of the bronchoscope corresponding to the depth map matches the camera position of the bronchus three-dimensional model, and the bronchoscope does not deviate from the navigation path. Even if the matching is not successful, the position of the bronchoscope can be adjusted according to the registration result, so that the accurate navigation of the bronchoscope can be quickly realized.

[0018] In a possible implementation method of the first aspect, the similarity calculation formula is specifically: Score = Sf * w1 + Si * w2; In the formula, Score is the similarity, S f is the feature similarity, S i is the image hash value, w 1 , w 2 are the image weights.

[0019] In a possible implementation method of the first aspect, it further includes: According to the registration result, adjust the current position of the bronchoscope until the bronchoscope reaches the target nodule along the navigation path.

[0020] The second aspect of the present application provides an image registration system based on virtual bronchoscope navigation, and the system includes: A navigation path generation module, a depth map generation module, and an image registration module; Among them, the navigation path generation module is used to generate the navigation path of the target nodule through lung segment segmentation according to the target nodule and the preset bronchus three-dimensional model; The depth map generation module is used to collect real-time images at preset sampling points in the navigation path by moving the bronchoscope, and convert the real-time images into depth maps; The image registration module is used to register the depth map with the bronchus three-dimensional model through a rough matching method and a fine matching method to obtain a registration result. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the present application, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a schematic diagram of the specific process of an image registration method based on virtual bronchoscope navigation provided by an embodiment of the present application; Figure 2 It is a schematic diagram of the lungs of an image registration method based on virtual bronchoscope navigation provided by an embodiment of the present application; Figure 3 It is a schematic diagram of path selection based on lung segment segmentation of an image registration method based on virtual bronchoscope navigation provided by an embodiment of the present application Figure 4 It is a diagram of the image extraction result of an image registration method based on virtual bronchoscope navigation provided by an embodiment of the present application; Figure 5 It is a schematic diagram of depth map adjustment of an image registration method based on virtual bronchoscope navigation provided by an embodiment of the present application; Figure 6 It is a diagram of block-based average hash value calculation of an image registration method based on virtual bronchoscope navigation provided by an embodiment of the present application; Figure 7 It is a diagram showing the bifurcation position of the lungs of an image registration method based on virtual bronchoscope navigation provided by an embodiment of the present application; Figure 8 It is a diagram of the registration result of an image registration method based on virtual bronchoscope navigation provided by an embodiment of the present application; Figure 9 It is a specific structural diagram of an image registration system based on virtual bronchoscope navigation provided by an embodiment of the present application. Specific Embodiments

[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without making creative efforts shall fall within the protection scope of the present application.

[0024] It should be understood that the step numbers used in the text are only for convenience of description and are not intended to limit the order of execution of the steps.

[0025] First Embodiment Bronchoscope navigation technology is often used to assist in the treatment of pulmonary nodule diseases. By collecting pulmonary image data in a preset navigation path, doctors can fully understand the patient's lung conditions. The existing bronchoscope navigation technologies are mainly divided into electromagnetic navigation and virtual navigation. Electromagnetic navigation can provide real-time feedback to doctors to ensure the accuracy of navigation, but it has high requirements for equipment and complex operations. Virtual navigation has low requirements for equipment and flexible operations, but its navigation accuracy is poor and largely depends on the quality of the input medical images. Poor image quality may affect the navigation effect. Therefore, how to integrate the advantages of these two navigation technologies and eliminate the influence of their disadvantages to obtain a bronchoscope navigation method that is easy to operate, highly economical, and has high precision, so that doctors can smoothly find the target nodule, is the main research direction of the embodiments of this application.

[0026] As Figure 1 shown, Figure 1 This is a schematic flowchart of a specific process of an image registration method based on virtual bronchoscope navigation provided by an embodiment of this application. The image registration method based on virtual bronchoscope navigation in this embodiment includes steps S1 to S3, which are described in detail as follows: Step S1, according to the target nodule and the preset three-dimensional bronchial model, generate the navigation path of the target nodule through lung segment segmentation.

[0027] In the embodiments of this application, the navigation path is planned first, mainly based on the constructed three-dimensional bronchial model and the region where the target nodule is located to select and plan the most suitable path.

[0028] In the initial stage of path planning, the lungs of the patient are medically imaged through CT scanning to identify the end of the bronchus to construct a three-dimensional bronchial model and determine the starting point of the model. Among them, the starting point is the entrance for the bronchoscope to enter the three-dimensional bronchial model for image sampling.

[0029] Specifically, during the process of lung medical imaging, the starting point can be determined according to the direction of CT scanning. If the CT scanning is performed along the foot-to-head direction of the human body, and this direction corresponds to the Z-axis in the coordinate system of the three-dimensional model, then the starting point is located at the minimum value of the Z-axis; if the CT scanning is performed along the head-to-foot direction of the human body, then the starting point is located at the maximum value of the Z-axis.

[0030] In addition, since the starting point is usually located in the trachea with a relatively large radius, the position of the starting point can be quickly and accurately located in the three-dimensional bronchial model based on this physical feature, thereby improving the accuracy and efficiency of registration.

[0031] Optionally, in the embodiments of the present application, the VMTK library is used to identify the bronchus and the starting point. The VMTK library is an open-source vascular modeling toolkit and can also be used for the extraction of the centerline of the three-dimensional model of the bronchial tube.

[0032] Then, the centerline of the bronchus is extracted from the three-dimensional model of the bronchus, and the branching relationship of the model is constructed to complete the path planning. First, all feasible paths of the target nodule in the model and the bifurcation points of these feasible paths are extracted according to the centerline. Then, the model is segmented into pulmonary segments based on these bifurcation points, and the specific pulmonary segment where the target nodule is located is accurately located through the segmentation result, which helps the doctor accurately understand the position of the nodule and select the most appropriate navigation path.

[0033] Figure 2 A schematic diagram of the lungs is provided. Figure 3 A schematic diagram of path selection based on pulmonary segment segmentation in the embodiments of the present application is provided. As shown in the figure, Figure 2 The different pulmonary segments connected by the bronchus are shown, and the relevant pulmonary segments above are explained below; Figure 3 Target in it is the location where the target nodule is located, Path is the path, and Lobe boundary is the boundary line for the segmentation of the pulmonary segments of the lobar bronchi. In path planning, usually the path with the shortest distance between the starting point and the target nodule is searched for. However, in fact, due to the complex structure of the lungs, it is unreliable to make a selection only based on the distance between the target nodule and the path, because the path with the shortest distance may not belong to the pulmonary segment where the target nodule is located, which may result in the selected navigation path ultimately unable to reach the target nodule. Therefore, it is very necessary to determine the path through pulmonary segment segmentation technology, which can ensure that the obtained navigation path can definitely reach the target nodule.

[0034] Step S2, by moving the bronchoscope, real-time images at preset sampling points in the navigation path are collected, and the real-time images are converted into depth maps.

[0035] In the embodiments of the present application, the sampling points in the navigation path are set by a preset spacing. The sampling points include the straight segments, curved segments, and bifurcation points of the three-dimensional model of the bronchus. Specifically, the sampling points are distributed at all positions on the navigation path, usually evenly distributed at a fixed spacing to fully cover the entire path.

[0036] Then, starting from the starting point of the three-dimensional model of the bronchus, the bronchoscope is moved along the navigation path, and multiple real-time images are collected at the sampling points. Then, images with higher quality and meeting the preset quality standards are selected and converted into corresponding depth maps through the adversarial network model. Among them, the adversarial network model is called GAN for short and includes a generative model and a discriminative model, which can be used for image conversion.

[0037] Optionally, the real-time image is an RGB image.

[0038] Figure 4 The provided image extraction result diagram shows the depth map obtained at the bifurcation point of the navigation path. In the left figure, the red line represents the navigation path, and LNB, LUL, and LID are respectively three bifurcation points on the navigation path; in the right figure, the upper part of the first column is the real-time image at the bifurcation point LUL, and the lower part is the corresponding depth map; the second column is the real-time image at the bifurcation point LID, and the lower part is the corresponding depth map. Compared with the real-time image, the depth map is clearer and can show the three-dimensional scene inside the bronchus more meticulously, further improving the accuracy of image registration.

[0039] Considering that the obtained depth map is actually generated based on the perspective of the bronchoscope, this single-perspective depth map cannot meet the actual needs, especially at the bifurcation point and in a complex environment, and cannot better provide the three-dimensional information of the internal environment of the bronchoscope. Therefore, as an improvement to the above solution, the embodiments of the present application also generate corresponding conical bases for each sampling point on the navigation path to expand the viewing angle, and obtain depth maps with multiple perspectives to improve the diversity and comprehensiveness of the depth map, making it closer to the actual observation mode of the bronchoscope.

[0040] Specifically, for a certain sampling point, take this sampling point as the center point of the conical base, determine the radius based on the position of the center point in the navigation path, and generate the corresponding conical base. Among them, the top of the cone is the next point of this sampling point in the navigation path. Then set four quadrant points on the conical base. The first quadrant point is deflected by a random angle relative to the center point, and the range of this random angle is from 0° to 90°; then based on the first quadrant point, deflect 90°, 180°, and 270° clockwise on the conical base respectively to obtain the remaining three quadrant points. This can ensure that the arranged quadrant points are evenly distributed on the conical base, obtaining a depth map with a comprehensive perspective. Finally, adjust the angle, image start position, and image end position of the depth map of the center point according to the angles of these quadrant points to obtain depth maps with multiple perspectives, so as to enhance the realism and diversity of the depth map and facilitate subsequent image registration.

[0041] Figure 5 The provided depth map adjustment schematic diagram shows the method for collecting depth maps with different perspectives in the embodiments of the present application. In the figure, R is the sampling point, D is the next bifurcation point of the navigation path, a conical base is constructed with R as the center point, and the 4 green points in the conical base are the corresponding quadrant points. Designate a point in front of D as the viewing angle of the quadrant point. Adjust the depth map with the viewing angles of these quadrant points pointing to the upcoming next bifurcation point, and depth maps with different perspectives at R can be obtained, enriching the authenticity of the images collected by the bronchoscope.

[0042] Step S3: Register the depth map with the three-dimensional bronchial model by means of rough matching and fine matching to obtain a registration result.

[0043] In the embodiments of the present application, two image matching methods are provided: rough matching and fine matching, to comprehensively evaluate the similarity of images, thereby improving the reliability and stability of the matching process and accurately positioning the current position of the bronchoscope. Rough matching is mainly used to quickly verify whether the current position of the bronchoscope is at the correct bifurcation point on the navigation path in the initial stage of registration, and at the same time roughly determine the position of the camera in the three-dimensional bronchial model. However, since rough matching is based on the collected depth maps, these images may not cover all the three-dimensional positions that the bronchoscope can actually reach. Therefore, it is necessary to further refine and adjust the position and angle of the camera in the three-dimensional bronchial model through fine matching on the basis of rough matching to improve the positioning accuracy of the bronchoscope in actual situations.

[0044] Further, in the process of rough matching, a preset convolutional neural network is first used to extract features from the depth map collected by the actual bronchoscope and converted by the GAN model and the virtual bronchial images collected by the virtual camera, respectively, to obtain the first feature vector of the depth map and the second feature vector of the virtual bronchial images. Among them, the virtual bronchial images are several virtual depth maps collected in advance by the virtual camera of the three-dimensional bronchial model at each bifurcation point on the navigation path. Calculate the feature similarity between the first feature vector and the second feature vector, and then screen the depth map through the feature similarity. First, exclude all virtual bronchial images that are completely inconsistent with the current depth map according to the feature similarity, such as virtual bronchial images that are in a flipped state with respect to the depth map; then select the second threshold number of virtual bronchial images with the highest feature similarity from the remaining virtual bronchial images as reference images for image registration again. Screening a part of the data that is irrelevant to the depth map itself through the feature similarity before image registration reduces the subsequent calculation amount and improves the efficiency of image registration.

[0045] Specifically, in the process of calculating the feature similarity, the feature similarity is calculated for the depth map and the virtual bronchial image corresponding to the same sampling point, and then the virtual bronchial images are screened based on the depth map of the same sampling point. In subsequent image registration, the depth map and the reference image of the same sampling point are also registered, and in fine matching, the image matching accuracy of the same sampling point is also calculated.

[0046] Optionally, the convolutional neural network used in the embodiments of the present application for feature extraction of the depth map and the virtual bronchial images is a trained convolutional neural network with the fully connected layer removed. The second threshold is to select the remaining 50 virtual bronchial images with the highest feature similarity as reference images.

[0047] Then, the block - based average hashing algorithm is used to calculate these 50 reference images. First, a reference image is evenly divided into several small blocks, and the average brightness value of each small block is calculated, and the image hash value of the reference image is comprehensively generated. After obtaining the image hash value of each reference image, the image hash value of each reference image and the feature similarity are weighted and calculated through the similarity calculation formula to obtain the similarity between each reference image and the corresponding depth map.

[0048] Among them, the similarity calculation formula is specifically: Score = Sf * w1+Si * w2; In the formula, Score is the similarity, S f is the feature similarity, S i is the image hash value, w 1 、w 2 are image weights.

[0049] In addition to the above image registration through the similarity calculation formula, the embodiment of the present application also uses the SSIM technology for image registration. The SSIM technology is an index for measuring the similarity between two images, mainly by combining three factors: brightness, contrast, and structural information to evaluate the similarity between images, and calculating the corresponding SSIM value. The value range of the SSIM value is from 0 to 1, and the larger the value, the more similar the images are. When the two images are exactly the same, the SSIM value is 1.

[0050] By combining the SSIM value and the similarity between the reference image and the depth map, the total number of the reference images with the similarity greater than the third threshold is counted. When the total number exceeds the fourth threshold and the occurrence frequency of a certain reference image exceeds the set threshold, it can be considered that the rough matching is successful, and the current position of the bronchoscope is consistent with the camera position of the bronchial three - dimensional model, indicating that the current bronchoscope does not deviate from the preset navigation path.

[0051] Figure 6 A block - based average hash value calculation diagram is provided. The diagram shows that a certain reference image is divided into 5 image blocks, where 4 image blocks are distributed in the four corners (upper left, upper right, lower left, and lower right) of the reference image respectively, and the 5th image block is located in the center of the reference image, that is, the area within the red frame. The purpose of such division is to achieve a balance between the whole and the local, which can not only retain the global information but also be sensitive to local changes in the image. Calculate the hash values of these 5 image blocks respectively to compare the local features between images more finely. Then, the image hash value of the reference image is obtained by taking the average of these 5 hash values.

[0052] As an improvement to the above solution, the embodiments of the present application further adopt fine matching on the basis of rough matching to further refine and adjust the position and angle of the virtual camera in the three-dimensional bronchial model, so as to improve the positioning accuracy of the actual bronchoscope in rough matching. First, obtain the real-time reference image provided by the virtual camera of the three-dimensional bronchial model at the current position, and then calculate the hash value of the real-time reference image and the depth map collected by the actual bronchoscope and converted by the GAN model to obtain the image matching accuracy. When the image matching accuracy is less than the first threshold, it indicates that the camera parameters of the three-dimensional bronchial model need to be adjusted; when the image matching accuracy is not less than the first threshold, it indicates that the current camera parameters of the three-dimensional bronchial model do not affect the subsequent image registration accuracy.

[0053] Considering that the lens of the bronchoscope may shake in actual situations, the embodiments of the present application also adopt a tracking algorithm to update the markers of the bifurcation points and the moving direction of the bronchoscope in real time. After the rough matching is successful, the bifurcation points passed by the camera in the bronchoscope three-dimensional model and the current moving direction are highlighted and overlaid on the endoscopic image and depth map of the actual bronchoscope to further improve the accuracy of bronchoscope navigation.

[0054] The embodiments of the present application also provide Figure 6 to display the main bifurcation points involved in the navigation path. Based on these bifurcation points, The embodiments of the present application collected twenty test cases to demonstrate the image registration accuracy of the solution of the present application. Among them Figure 6 is used to display the key bifurcation points involved in each navigation path in these test cases. Each test case contains approximately 500 RGB images of the key bifurcation points passed by a complete navigation path, and approximately 2,000 reference depth map images collected in the three-dimensional bronchial model at the corresponding bifurcation points. By calculating the top 10 registration results of each RGB image. If a reference depth map image belongs to the bifurcation point of the RGB image, it is regarded as a successful match.

[0055] To ensure the representativeness of the test data, the twenty selected navigation paths cover the main bifurcation points of each bronchus, ensuring the comprehensiveness and rigor of the test. After testing, the average accuracy rate of the solution of the present application in 20 test cases is 90%, and the average processing time for each RGB image from image conversion, feature extraction to output of the matching result is 0.38 s, with high data processing efficiency.

[0056] Figure 7 To display the image registration accuracy of these twenty test cases, the top 10 registration results of the RGB images in 20 navigation paths are shown, and their registration accuracies are all higher than 80% and mostly around 90%.

[0057] Finally, according to the registration result, adjust the current position of the bronchoscope until the bronchoscope reaches the target nodule along the navigation path.

[0058] Implementing the embodiments of the present application has the following beneficial effects: In the embodiments of the present application, first perform lung segment segmentation on the bronchial three-dimensional model to quickly locate the lung segment where the target nodule is located, and then select the most suitable navigation path according to the position of the target nodule, ensuring that the navigation path definitely includes the target nodule, thereby improving the accuracy of path planning; then move the bronchoscope in the navigation path to collect real-time images and convert the images into depth maps, reducing the computational complexity for subsequent image registration; finally, through the combination of two matching methods, namely rough matching and fine matching, compare the depth map collected in real time with the virtual image provided by the bronchial three-dimensional model to obtain the registration result for confirming whether the bronchoscope deviates from the navigation path, and timely adjust the position of the bronchoscope to achieve precise navigation of the bronchoscope. Moreover, the process of the above image registration does not need to be carried out in a specific operating environment, has no equipment installation restrictions, and reduces the operating cost.

[0059] Second Embodiment Furthermore, in order to implement the image registration system based on virtual bronchoscope navigation corresponding to the above method embodiments to achieve the corresponding functions and technical effects, Figure 8 A structural diagram of an image registration system based on virtual bronchoscope navigation is provided. For ease of description, only the parts related to this embodiment are shown. The image registration system based on virtual bronchoscope navigation provided by the embodiments of the present application includes: A navigation path generation module 201, configured to generate a navigation path for the target nodule through lung segment segmentation according to the target nodule and a preset bronchial three-dimensional model.

[0060] In the embodiments of the present application, generate the bronchial three-dimensional model according to pulmonary medical imaging, and extract the centerline of the bronchial three-dimensional model.

[0061] Extract all feasible paths of the target nodule from the bronchial three-dimensional model according to the centerline.

[0062] Based on the bifurcation points in the feasible paths, determine the lung segment where the target nodule is located and the navigation path including the lung segment through lung segment segmentation.

[0063] A depth map generation module 202, configured to collect real-time images at preset sampling points in the navigation path by moving the bronchoscope, and convert the real-time images into depth maps.

[0064] In an embodiment of the present application, sampling points in the navigation path are marked according to a preset spacing; wherein, the positions of the sampling points include straight segments, curved segments, and bifurcation points of the bronchial three-dimensional model.

[0065] Starting from the starting point of the bronchial three-dimensional model, control the bronchoscope to move along the navigation path and collect real-time images at the sampling points.

[0066] First images that meet the preset quality requirements are screened out from the real-time images, and the first images are converted into depth maps through an adversarial network model.

[0067] An image registration module 203 is configured to register the depth map with the bronchial three-dimensional model through a rough matching method and a fine matching method to obtain a registration result.

[0068] In an embodiment of the present application, according to the depth map, the camera parameters of the bronchial three-dimensional model are adjusted through a fine matching method.

[0069] According to the camera parameters, a virtual bronchial image is generated.

[0070] The depth map is roughly matched with the virtual bronchial image to obtain a registration result.

[0071] In some embodiments, the navigation path generation module 201 further includes: First, plan the navigation path, mainly based on the constructed bronchial three-dimensional model and the region where the target nodule is located to select and plan the most suitable path.

[0072] In the initial stage of path planning, medical imaging of the patient's lungs is performed through CT scanning to identify the end of the bronchus to construct a bronchial three-dimensional model and determine the starting point of the model. Wherein, the starting point is the entrance for the bronchoscope to enter the bronchial three-dimensional model for image sampling.

[0073] Specifically, during the process of lung medical imaging, the starting point can be determined according to the direction of the CT scan. If the CT scan is performed along the foot-to-head direction of the human body, and this direction corresponds to the Z-axis in the coordinate system of the three-dimensional model, then the starting point is located at the minimum value of the Z-axis; if the CT scan is performed along the head-to-foot direction of the human body, then the starting point is located at the maximum value of the Z-axis.

[0074] In addition, since the starting point is usually located in the trachea with a relatively large radius, the position of the starting point can be quickly and accurately located in the bronchial tube three-dimensional model based on this physical feature, thereby improving the accuracy and efficiency of registration.

[0075] Optionally, in the embodiments of the present application, the VMTK library is used to identify the bronchus and the starting point. The VMTK library is an open-source vascular modeling toolkit and can also be used for extracting the centerline of the three-dimensional model of the bronchial tube.

[0076] Then, the centerline of the bronchus is extracted from the three-dimensional bronchial model, and the branching relationship of the model is constructed to complete the path planning. First, all feasible paths of the target nodule in the model and the bifurcation points of these feasible paths are extracted according to the centerline. Then, the model is segmented into lung segments based on these bifurcation points. The specific lung segment where the target nodule is located is accurately positioned through the segmentation result, which helps the doctor accurately understand the position of the nodule and select the most appropriate navigation path.

[0077] Specifically, in path planning, usually the path with the shortest distance between the starting point and the target nodule is searched. However, in fact, due to the complex structure of the lungs, it is unreliable to rely solely on the distance between the target nodule and the path for selection, because the path with the shortest distance may not belong to the lung segment where the target nodule is located, which may lead to the selected navigation path ultimately unable to reach the target nodule. Therefore, it is very necessary to determine the path through lung segment segmentation technology, which can ensure that the obtained navigation path can definitely reach the target nodule.

[0078] In some embodiments, the depth map generation module 202 further includes: First, sampling points in the navigation path are set through a preset spacing. The sampling points include the straight segments, curved segments, and bifurcation points of the three-dimensional bronchial model. To be precise, the sampling points are distributed at all positions on the navigation path and are usually evenly distributed at a fixed spacing to fully cover the entire path.

[0079] Then, starting from the starting point of the three-dimensional bronchial model, the bronchoscope is moved along the navigation path. Multiple real-time images are collected at the sampling points, and the images with higher quality and meeting the preset quality standards are selected and converted into corresponding depth maps through the adversarial network model. The adversarial network model is fully called GAN and includes a generative model and a discriminative model, which can be used for image conversion.

[0080] Optionally, the real-time image is an RGB image.

[0081] Considering that the obtained depth map is actually generated based on the perspective of the bronchoscope, this depth map with a single perspective cannot meet the actual needs. Especially at the bifurcation points and in complex environments, it cannot better provide the three-dimensional information of the internal environment of the bronchoscope. Therefore, as an improvement to the above solution, in the embodiments of the present application, a corresponding conical bottom surface is also generated for each sampling point on the navigation path to expand the viewing angle, and a depth map with multiple perspectives is obtained to improve the diversity and comprehensiveness of the depth map, making it closer to the actual observation mode of the bronchoscope.

[0082] Specifically, for a certain sampling point, this sampling point is used as the center point of the cone bottom surface, and the radius is determined according to the position of the center point in the navigation path to generate the corresponding cone bottom surface. The top of the cone is the next point of this sampling point in the navigation path. Then, four quadrant points are set on the cone bottom surface. The first quadrant point is formed by deflecting a random angle relative to the center point, and the range of this random angle is from 0° to 90°; then, based on the first quadrant point, it is deflected 90°, 180°, and 270° clockwise on the cone bottom surface respectively to obtain the remaining three quadrant points. This can ensure that the arranged quadrant points are evenly distributed on the cone bottom surface, obtaining a depth map with a comprehensive perspective. Finally, according to the angles of these quadrant points, the angles, the starting position of the image, and the ending position of the image of the center point's depth map are adjusted to obtain a depth map with multiple perspectives, so as to enhance the realism and diversity of the depth map and facilitate the subsequent image registration.

[0083] In some embodiments, the image registration module 203 further includes: In the embodiments of the present application, two image matching methods are provided: rough matching and fine matching, to comprehensively evaluate the similarity of images, thereby improving the reliability and stability of the matching process and accurately positioning the current position of the bronchoscope. Rough matching is mainly used to quickly verify whether the current position of the bronchoscope is at the correct bifurcation point in the navigation path at the initial stage of registration, and at the same time, roughly determine the position of the camera in the bronchial three-dimensional model. However, since rough matching is based on the collected depth maps, these images may not cover all the three-dimensional positions that the bronchoscope can actually reach. Therefore, it is necessary to further refine and adjust the position and angle of the camera in the bronchial three-dimensional model through fine matching on the basis of rough matching to improve the positioning accuracy of the bronchoscope in actual situations.

[0084] Further, in the process of rough matching, a preset convolutional neural network is first used to extract features from the depth map collected by the actual bronchoscope after being converted by the GAN model and the virtual bronchial image collected by the virtual camera, so as to obtain the first feature vector of the depth map and the second feature vector of the virtual bronchial image. Among them, the virtual bronchial image is a plurality of virtual depth maps collected in advance by the virtual camera of the bronchial three-dimensional model at each bifurcation point in the navigation path. Calculate the first feature vector and the second feature vector to obtain the feature similarity between the depth map and the virtual bronchial image, and then screen the depth map through the feature similarity. First, exclude all virtual bronchial images that are completely inconsistent with the current depth map according to the feature similarity, such as virtual bronchial images in a flipped state with respect to the depth map; then select the top second threshold number of virtual bronchial images with the highest feature similarity from the remaining virtual bronchial images as reference images for image registration again. Screening a part of the data that is irrelevant to the depth map itself through the feature similarity before image registration reduces the subsequent calculation amount and improves the efficiency of image registration.

[0085] Specifically, in the process of calculating the feature similarity, the feature similarity is calculated for the depth map and the virtual bronchial image corresponding to the same sampling point, and then the virtual bronchial image is screened based on the depth map of the same sampling point. In the subsequent image registration, the depth map of the same sampling point and the reference image are also registered.

[0086] Optionally, the convolutional neural network used in the embodiment of the present application is a trained convolutional neural network with the fully connected layer removed for feature extraction of the depth map and the virtual bronchial image. The second threshold is to select the top 50 remaining virtual bronchial images with the highest feature similarity as reference images.

[0087] Then, the block-based average hashing algorithm is used to calculate these 50 reference images. First, a reference image is evenly divided into several small blocks, and the average brightness value of each small block is calculated to comprehensively generate the image hash value of the reference image. After obtaining the image hash value of each reference image, the image hash value of each reference image and the feature similarity are weighted and calculated through the similarity calculation formula to obtain the similarity between each reference image and the corresponding depth map.

[0088] Among them, the similarity calculation formula is specifically: Score = Sf * w1 + Si * w2; In the formula, Score is the similarity, S f is the feature similarity, S i is the image hash value, w 1 、w 2 are image weights.

[0089] In addition to the above image registration using the similarity calculation formula, the embodiments of the present application also use the SSIM technology for image registration. The SSIM technology is an index for measuring the similarity between two images. It mainly evaluates the similarity between images by combining three factors: brightness, contrast, and structural information, and calculates the corresponding SSIM value. The value range of the SSIM value is from 0 to 1. The larger the value, the more similar the images are. When the two images are exactly the same, the SSIM value is 1.

[0090] By combining the SSIM value and the similarity between the reference image and the depth map, the total number of the reference images with the similarity greater than the third threshold is counted. When the total number exceeds the fourth threshold and the appearance frequency of a certain reference image exceeds the set threshold, it can be considered that the rough matching is successful, and the current position of the bronchoscope is consistent with the camera position of the bronchial three-dimensional model, indicating that the current bronchoscope does not deviate from the preset navigation path.

[0091] As an improvement to the above solution, the embodiments of the present application further use fine matching on the basis of rough matching to further refine and adjust the position and angle of the virtual camera in the bronchial three-dimensional model, so as to improve the positioning accuracy of the actual bronchoscope in rough matching. First, obtain the real-time reference image provided by the virtual camera of the bronchial three-dimensional model at the current position, and then calculate the hash value of the real-time reference image and the depth map collected by the actual bronchoscope and converted by the GAN model to obtain the image matching accuracy. When the image matching accuracy is less than the first threshold, it indicates that the camera parameters of the bronchial three-dimensional model need to be adjusted; when the image matching accuracy is not less than the first threshold, it indicates that the current camera parameters of the bronchial three-dimensional model do not affect the subsequent image registration accuracy.

[0092] Considering that the lens of the bronchoscope may shake in actual situations, the embodiments of the present application also adopt a tracking algorithm to update the markers of the bifurcation points and the moving direction of the bronchoscope in real time. After the rough matching is successful, the bifurcation points passed by the camera in the bronchoscope three-dimensional model and the current moving direction are highlighted and overlaid on the endoscopic image and depth map of the actual bronchoscope, further improving the accuracy of bronchoscope navigation.

[0093] Finally, according to the registration result, the current position of the bronchoscope is adjusted until the bronchoscope reaches the target nodule along the navigation path.

[0094] Implementing the embodiments of the present application has the following beneficial effects: In the embodiment of the present application, the lung segments of the bronchial three-dimensional model are first segmented to quickly locate the lung segment where the target nodule is located. Then, the most suitable navigation path is selected according to the position of the target nodule, ensuring that the navigation path definitely includes the target nodule, thus improving the accuracy of path planning. Then, real-time images are collected by moving the bronchoscope along the navigation path, and the images are converted into depth maps to reduce the computational complexity for subsequent image registration. Finally, by combining two matching methods, namely rough matching and fine matching, the depth maps collected in real time are compared with the virtual images provided by the bronchial three-dimensional model to obtain the registration result for confirming whether the bronchoscope deviates from the navigation path, and timely adjusting the position of the bronchoscope to achieve precise navigation of the bronchoscope. Moreover, the above process of image registration does not need to be carried out in a specific operating environment, without equipment installation restrictions, reducing the operating cost.

[0095] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present application. It should be understood that the above are only specific embodiments of the present application and are not used to limit the protection scope of the present application. In particular, for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An image registration method based on virtual bronchoscopy navigation, characterized in that: include: According to the target nodule and the preset bronchial 3D model, a navigation path of the target nodule is generated through lung segmentation; By moving the bronchoscope, real-time images at preset sampling points in the navigation path are collected, and the real-time images are converted into depth maps; The depth map is registered with the bronchial three-dimensional model through a coarse matching method and a fine matching method to obtain a registration result.

2. The image registration method based on virtual bronchoscopy navigation according to claim 1, characterized in that: The navigation path of the target nodule is generated by lung segmentation according to the target nodule and the preset bronchial three-dimensional model, specifically: Generate the bronchial three-dimensional model and the model starting point according to the lung medical imaging, and extract the center line of the bronchial three-dimensional model; According to the center line, extracting all feasible paths from the starting point of the model to the target nodule from the bronchial three-dimensional model; Based on the bifurcation point in the feasible path, the lung segment where the target nodule is located and the navigation path including the lung segment are determined by lung segmentation.

3. The image registration method based on virtual bronchoscopy navigation according to claim 1, characterized in that: The method of collecting a real-time image at a preset sampling point in the navigation path by moving the bronchoscope and converting the real-time image into a depth map is specifically as follows: According to a preset spacing, the sampling points in the navigation path are marked; wherein the positions of the sampling points include straight segments, curved segments and bifurcation points of the bronchial three-dimensional model; Starting from the starting point of the bronchial three-dimensional model, controlling the bronchoscope to move along the navigation path, and collecting the real-time image at the sampling point; A first image that meets a preset quality requirement is screened out from the real-time image, and the first image is converted into a depth map through an adversarial network model.

4. The image registration method based on virtual bronchoscopy navigation according to claim 3, characterized in that: The step of converting the first image into a depth map by using an adversarial network model further includes: Taking the sampling point corresponding to the first image as the center point and determining the radius according to the position of the center point in the navigation path, a cone base is generated; According to a preset angle range, four different random angles are deflected on the bottom surface of the cone relative to the center point to obtain four quadrant points; The first image is converted into a depth map corresponding to the center point through an adversarial network model, and the angle, image starting position and image ending position of the depth map are adjusted according to the quadrant point to generate a depth map corresponding to each quadrant point, thereby obtaining all depth maps corresponding to the first image.

5. The image registration method based on virtual bronchoscopy navigation according to claim 1, characterized in that: The registering the depth map with the bronchial three-dimensional model by a coarse matching method and a fine matching method to obtain a registration result is specifically: adjusting the camera parameters of the bronchial three-dimensional model by a fine matching method according to the depth map; generating a virtual bronchial image according to the camera parameters; The depth map is roughly matched with the virtual bronchial image to obtain a registration result.

6. The image registration method based on virtual bronchoscopy navigation according to claim 5, characterized in that: The camera parameters of the bronchial three-dimensional model are adjusted by a precise matching method according to the depth map, specifically: Acquire a real-time reference image provided by the bronchial three-dimensional model; Obtaining image matching accuracy by calculating hash values ​​of the real-time reference image and the depth map; When the image matching accuracy is less than the first threshold, the camera position of the bronchial three-dimensional model is adjusted according to the depth map; when the image matching accuracy is not less than the first threshold, according to the camera position of the bronchial three-dimensional model, The current moving direction and the current bifurcation point of the bronchoscope are updated in the depth map by a tracking algorithm.

7. The image registration method based on virtual bronchoscopy navigation according to claim 5, characterized in that: The coarse matching of the depth map and the virtual bronchial image to obtain a registration result is specifically as follows: Performing feature extraction on the depth map and the virtual bronchial image respectively through a preset convolutional neural network to obtain a first feature vector of the depth map and a second feature vector of the virtual bronchial image; Calculating feature similarity between the depth map and the virtual bronchial image according to the first feature vector and the second feature vector, and screening the virtual bronchial image according to the feature similarity to obtain a second threshold number of reference images; Calculating the brightness value of the reference image by a block-based average hash algorithm to obtain an image hash value of each reference image; Performing weighted calculation on the image hash value of each reference image and the feature similarity using a similarity calculation formula to obtain the similarity of each reference image; Counting the total number of the reference images whose similarity is greater than a third threshold, to obtain a registration result; When the total number exceeds a fourth threshold, the registration result is that the current position of the bronchoscope is consistent with the camera position of the bronchial three-dimensional model.

8. The image registration method based on virtual bronchoscopy navigation according to claim 7, characterized in that: The similarity calculation formula is specifically: Score=S f *w1+S i *w2; In the formula, Score is the similarity, S f is the feature similarity, S i is the image hash value, w1 and w2 are image weights.

9. The image registration method based on virtual bronchoscopy navigation according to any one of claims 1 to 8, characterized in that: Also includes: According to the registration result, the current position of the bronchoscope is adjusted until the bronchoscope reaches the target nodule along the navigation path.

10. An image registration system based on virtual bronchoscopy navigation, characterized in that: include: Navigation path generation module, depth map generation module and image registration module; The navigation path generation module is used to generate a navigation path of the target nodule by segmenting the lung segment according to the target nodule and the preset bronchial three-dimensional model; The depth map generation module is used to collect real-time images at preset sampling points in the navigation path by moving the bronchoscope, and convert the real-time images into depth maps; The image registration module is used to register the depth map with the bronchial three-dimensional model through a coarse matching method and a fine matching method to obtain a registration result.

Images