Bronchoscopy positioning method and device

Abstract

The bronchoscope positioning method and device can avoid the influence of texture changes, does not need manual initialization positioning, and can automatically resume positioning after the positioning is interrupted. It comprises: (1) a virtual camera for controlling the movement direction of the camera by increasing or decreasing Euler angles, a conical light source following the virtual bronchoscope movement is set, the spectrum and illumination range of the light source are adjusted according to the real conditions, and a texture map is made by using a real endoscope image; (2) the virtual endoscope records color images, corresponding depth images and bronchoscope poses at 30fps; the corresponding color images, depth images and pose data of the virtual bronchoscope are obtained from the reconstructed 3D model, the obtained data is divided into corresponding sub-regions to generate a virtual image set; (3) a positive and negative example comparison method is used to train an image descriptor extraction network, and sub-region positioning of the bronchoscope is realized through descriptor retrieval; (4) the depth information of the image is estimated by using a generation network, and the pose of the bronchoscope is obtained based on the depth information.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more particularly to a method for bronchoscope positioning and a bronchoscope positioning device. Background Technology

[0002] Lung cancer is one of the cancers with the highest incidence and mortality rates. During diagnosis and treatment, doctors first determine the location of the tumor based on CT images and then guide the bronchoscope to the lesion for examination. However, the bronchus has a complex structure, and there is no direct correlation between bronchoscope images and CT images. This poses a significant challenge for doctors when guiding the bronchoscope to the lesion, and can lead to missed lesions during examination. By constructing a 3D model of the bronchus using CT images and employing a positioning algorithm to determine the bronchus position, the current position of the endoscope can be displayed in real-time in the 3D model during diagnosis and treatment. This provides doctors with a clear view of the bronchoscope's location, eliminating the difficulties of manual positioning.

[0003] Bronchoscope positioning methods can be divided into two categories: electromagnetic sensor-based and image-based. Electromagnetic sensor-based positioning integrates an electromagnetic sensor at the end of the bronchoscope, calculating its position within an artificial electromagnetic field for positioning. This type of positioning method has the following drawbacks: (a) it is sensitive to movements such as breathing and heartbeat; (b) the proximity of conductors or ferromagnetic materials can interfere with the artificial electromagnetic field; and (c) integrating electromagnetic sensors into the small end of the bronchoscope is difficult.

[0004] Image-based localization methods use only bronchoscope images for localization, are not limited by external environments, and do not require modification of the bronchoscope. In research, inter-frame motion estimation algorithms are often used to calculate the bronchoscope's trajectory for localization; however, these methods require manual initialization of the bronchoscope pose and suffer from accumulated errors during localization. Jung et al. reconstructed a bronchial mesh model from CT images, acquired virtual images using a virtual bronchoscope, and used edge features to find the most similar virtual image to the input image during surgery. The approximate location of the bronchoscope was estimated based on the position of this virtual image. Merritt et al. used gradient features of the image for localization, but the weak texture of the bronchoscope image and unstable lighting resulted in less than ideal results. Deep learning can integrate diverse information and is increasingly widely used in endoscopic localization tasks. Zhao et al. used neural networks to directly map images to poses for bronchoscope localization, but even slight changes in the scene require model retraining. Hong et al. used neural networks to estimate the depth map of the image and used depth information for localization, but the existence of multiple depth map similarity regions within the cavity limits its performance when depth estimation is not entirely accurate. Song et al. used Siamese networks to directly generate image descriptors and divided the cavity into sub-regions. They first located the sub-regions based on the similarity between the descriptors of the real and virtual images, and then used the constraint relationship between 3D-2D corresponding points to determine the accurate pose. The localization effect was significantly improved, but the textures of the virtual and real images were different, making it difficult to find corresponding points. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the technical problem to be solved by the present invention is to provide a bronchoscope positioning method that can avoid the influence of texture changes, achieve positioning without manual initialization, and automatically recover positioning after positioning is interrupted.

[0006] The technical solution of this invention is: this bronchoscopy positioning method includes the following steps:

[0007] (1) A virtual camera whose movement direction is controlled by adding or subtracting Euler angles. The intrinsic parameters of the virtual camera are set to be consistent with the intrinsic parameters of the actual bronchoscope. In order to simulate the lighting conditions in the real environment, a cone-shaped light source that follows the movement of the virtual bronchoscope is set. The spectrum and illumination range of the light source are adjusted according to the real conditions, and texture maps are made using real endoscope graphics.

[0008] (2) The virtual endoscope records color images and their corresponding depth maps and bronchoscope poses at a speed of 30fps; during the virtual image acquisition stage, the virtual bronchoscope is used to obtain the corresponding color images, depth images and pose data from the reconstructed 3D model, and the obtained data is divided into corresponding sub-regions to generate a virtual image set.

[0009] (3) The image descriptor extraction network is trained by comparing positive and negative examples, and the sub-region localization of the bronchoscope is achieved by descriptor retrieval;

[0010] (4) Use the generative network to estimate the depth information of the image, and obtain the pose of the bronchoscope based on the depth information.

[0011] This invention uses positive and negative examples to train a descriptor extraction network for sub-region localization. During the training process, prior knowledge of hole information is introduced to guide the network to make judgments based on the branch structure, reducing the impact of texture changes. After determining the sub-region, the bronchoscope pose is predicted through depth information, achieving localization without manual initialization. Localization can also be automatically restored after interruption.

[0012] A bronchoscopy positioning device is also provided, which includes:

[0013] The virtual camera module is configured as a virtual camera whose movement direction is controlled by adding or subtracting Euler angles. The intrinsic parameters of the virtual camera are set to be consistent with the intrinsic parameters of the actual bronchoscope. In order to simulate the lighting conditions in the real environment, a cone-shaped light source is set to follow the movement of the virtual bronchoscope. The spectrum and illumination range of the light source are adjusted according to the real conditions, and texture maps are made using real endoscope graphics.

[0014] The data acquisition module is configured to record color images and their corresponding depth maps and bronchoscope poses at a speed of 30fps using a virtual endoscope. During the virtual image acquisition stage, the virtual bronchoscope is used to acquire the corresponding color images, depth images and pose data from the reconstructed 3D model, and the obtained data is divided into corresponding sub-regions to generate a virtual image set.

[0015] The descriptor module is configured to train the image descriptor extraction network using a positive and negative example comparison method, and to achieve sub-region localization of the bronchoscope through descriptor retrieval;

[0016] The depth information module is configured to estimate the depth information of the image using a generative network and obtain the pose of the bronchoscope based on the depth information. Attached Figure Description

[0017] Figure 1 This is a schematic flowchart of the bronchoscopic positioning method according to the present invention. Detailed Implementation

[0018] like Figure 1 As shown, this bronchoscopic positioning method includes the following steps:

[0019] (1) A virtual camera whose movement direction is controlled by adding or subtracting Euler angles. The intrinsic parameters of the virtual camera are set to be consistent with the intrinsic parameters of the actual bronchoscope. In order to simulate the lighting conditions in the real environment, a cone-shaped light source that follows the movement of the virtual bronchoscope is set. The spectrum and illumination range of the light source are adjusted according to the real conditions, and texture maps are made using real endoscope graphics.

[0020] (2) The virtual endoscope records color images and their corresponding depth maps and bronchoscope poses at a speed of 30fps; during the virtual image acquisition stage, the virtual bronchoscope is used to obtain the corresponding color images, depth images and pose data from the reconstructed 3D model, and the obtained data is divided into corresponding sub-regions to generate a virtual image set.

[0021] (3) The image descriptor extraction network is trained by comparing positive and negative examples, and the sub-region localization of the bronchoscope is achieved by descriptor retrieval;

[0022] (4) Use the generative network to estimate the depth information of the image, and obtain the pose of the bronchoscope based on the depth information.

[0023] This invention uses positive and negative examples to train a descriptor extraction network for sub-region localization. During the training process, prior knowledge of hole information is introduced to guide the network to make judgments based on the branch structure, reducing the impact of texture changes. After determining the sub-region, the bronchoscope pose is predicted through depth information, achieving localization without manual initialization. Localization can also be automatically restored after interruption.

[0024] Preferably, before step (1), a 3D mesh model of the bronchus is reconstructed from the CT image using threshold segmentation, and the bronchus is divided into different sub-regions according to the branching structure of the bronchus; each branch with a length of less than 1 cm is treated as a separate sub-region, and for the bronchus with a length of more than 2 cm, it is divided into two sub-regions, one before and one after, and located in the segment closer to the smaller branch.

[0025] Preferably, in step (2), images containing clear hole information are selected from the virtual image set as keyframe datasets. In addition, a pose optimization dataset for precise positioning is generated using a virtual bronchoscope system. The sampling method for the pose optimization dataset is to generate annular regions with radii increasing by 1.5 mm in a cross section with a 1 mm interval near the end of the sub-region. In each annular region, 8×n images with an angle of 0 to 10° to the center line direction are randomly generated evenly, where n represents the nth annular region. If the radius is less than 1.5 mm, only the points on the center line are used as reference positions.

[0026] Preferably, in step (3), the network used to generate the descriptor is called FDNet. The input of FDNet is a 3-channel color image, and the output includes a single-channel grayscale image and an image descriptor. Before training, the relative total difference detexturing algorithm is used to remove image details that may cause interference, and then the hole region is segmented using an adaptive threshold as a hole information map. The hole information map can reflect the branching structure of the bronchus. By optimizing the output grayscale image to make it close to the pre-extracted image hole information map, the hole information is introduced into the network as prior knowledge. In the training based on positive and negative example contrastive learning, FDNet is first used to extract descriptors for the baseline image, positive image, and negative image, respectively, during each training session. Simultaneously, prior knowledge of hole information is introduced into the network through the cross-entropy loss function. In the FDNet structure diagram, the blue blocks represent the encoding layer consisting of 4 sets of ResNet-v2 blocks, the yellow blocks represent the decoding layer consisting of 4 sets of convolutional layers, with the output being a single-channel information map, and the green blocks represent the fully connected encoding layer, with the output being a 1×128-dimensional descriptor. The loss function used to introduce the prior knowledge of hole information is cross-entropy loss.

[0027]

[0028] Where y represents the image hole information map, y ′ The information graph representing the network output, y and y ′ The median value ranges from 0 to 1.

[0029] Preferably, in step (3), in order for the loss function to be able to perceive the difference between positive and negative examples during the training process, Triplet Loss is used as the network's loss function, defined as follows:

[0030]

[0031] in, Represents the baseline image descriptor. The descriptor representing the sample and, The descriptor represents the negative sample; α is a hyperparameter representing the adjustment... and To determine the minimum distance between the two images, this paper sets α to 1. For the baseline image descriptors, positive sample descriptors, and negative sample descriptors extracted using FDNet, TripletLoss is used to reduce the Euclidean distance between the baseline image descriptors and positive sample descriptors, and to increase the Euclidean distance between the baseline image descriptors and negative sample descriptors, until the distance between the baseline image descriptors and positive sample descriptors is less than the distance between the baseline image descriptors and negative sample descriptors, and the difference between the two distances is not less than the set hyperparameter α. Then, the criterion for determining whether two images are in the same sub-region is whether the Euclidean distance between the descriptors of the two images is less than α.

[0032] Preferably, in step (4), a depth map of the real image is obtained using a depth estimation network. The input of the depth estimation network is a 3-channel color bronchoscopy image, and the output is a single-channel grayscale image representing the image depth. The network structure includes an encoder and a decoder. The decoder consists of a convolutional layer, 4 sets of ResNet-v2 blocks, and pooling layers. The decoder consists of 3 sets of convolutional layers and upsampling layers. The last convolutional layer in each ResNet-v2 block of the encoder is concatenated with the corresponding convolutional layer in the decoder.

[0033] Preferably, in step (4), the L1 loss of corresponding pixels between the depth map gold standard and the network prediction result is used to measure the brightness difference between the prediction result and the label, and the structural similarity loss SSIM is used to measure the structural difference between the prediction result and the label. The total loss function is as follows:

[0034]

[0035] in, As the gold standard for depth mapping, In the network prediction results, γ is a parameter that balances structural loss and brightness loss; in this paper, it is set to 0.85. SSIM measures the structural similarity between two images using the covariance, and its definition is as follows:

[0036]

[0037] Where A and B represent the input images, μ A and μ B σ represents the pixel mean of images A and B, respectively. A and σ A σ represents the variance of images A and B. AB This represents the covariance of images A and B.

[0038] Preferably, in step (4), in order to improve the processing speed, the image is downsampled with a spacing of 4 before depth estimation is performed; after obtaining the depth map, the depth map is divided into 8 sectors with the image center as the center. Each sector is divided into inner, middle and outer parts according to the distance from the pixel to the image center, resulting in 24 regions. Starting from the region with the largest average gray value, a difference hash calculation is performed from the outside to the inside in a clockwise order to obtain a 24-dimensional vector. The 8-bit gray histograms of the inner, middle and outer parts of the depth map are statistically analyzed. The statistical values ​​are normalized to between 0 and 2 to obtain 3 8-dimensional vectors. The two feature vectors obtained are combined into a 48-dimensional vector as a depth descriptor.

[0039] Preferably, before bronchoscopy, the method calculates the depth descriptor of the optimized dataset in each sub-region to generate an optimized descriptor set. During bronchoscopy, the depth descriptor of the current input image is first calculated, and the most similar descriptor is retrieved from the optimized descriptor set. The pose of the corresponding image is used as the current bronchoscopy pose. Since the image is circularly symmetrical, the direction of the line connecting the image center and the sub-region with the largest average depth in the outer ring is taken as the principal direction. The bronchoscopy pose is rotated around the principal optical axis so that the principal direction of the input image coincides with the principal direction of the corresponding virtual image, thus obtaining the final localization result.

[0040] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a bronchoscope positioning device, which is typically represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0041] The virtual camera module is configured as a virtual camera whose movement direction is controlled by adding or subtracting Euler angles. The intrinsic parameters of the virtual camera are set to be consistent with the intrinsic parameters of the actual bronchoscope. In order to simulate the lighting conditions in the real environment, a cone-shaped light source is set to follow the movement of the virtual bronchoscope. The spectrum and illumination range of the light source are adjusted according to the real conditions, and texture maps are made using real endoscope graphics.

[0042] The data acquisition module is configured to record color images and their corresponding depth maps and bronchoscope poses at a speed of 30fps using a virtual endoscope. During the virtual image acquisition stage, the virtual bronchoscope is used to acquire the corresponding color images, depth images and pose data from the reconstructed 3D model, and the obtained data is divided into corresponding sub-regions to generate a virtual image set.

[0043] The descriptor module is configured to train the image descriptor extraction network using a positive and negative example comparison method, and to achieve sub-region localization of the bronchoscope through descriptor retrieval;

[0044] The depth information module is configured to estimate the depth information of the image using a generative network and obtain the pose of the bronchoscope based on the depth information.

[0045] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for bronchoscope positioning, characterized in that: It includes the following steps: (1) A virtual camera whose movement direction is controlled by adding or subtracting Euler angles. The intrinsic parameters of the virtual camera are set to be consistent with the intrinsic parameters of the actual bronchoscope. In order to simulate the lighting conditions in the real environment, a cone-shaped light source is set to follow the movement of the virtual bronchoscope. The spectrum and illumination range of the light source are adjusted according to the real conditions, and texture maps are made using real endoscope graphics. (2) The virtual endoscope records color images and their corresponding depth maps and bronchoscope poses at a speed of 30fps; during the virtual image acquisition stage, the virtual bronchoscope is used to obtain the corresponding color images, depth images and pose data from the reconstructed 3D model, and the obtained data is divided into corresponding sub-regions to generate a virtual image set. (3) The image descriptor extraction network is trained by comparing positive and negative examples, and the sub-region localization of the bronchoscope is achieved by descriptor retrieval; (4) Use the generative network to estimate the depth information of the image, and obtain the pose of the bronchoscope based on the depth information; This method uses positive and negative examples to train a descriptor extraction network for sub-region localization. During the training process, prior knowledge of hole information is introduced to guide the network to make judgments based on the branch structure to reduce the impact of texture changes. After determining the sub-region, the bronchoscope pose is predicted through depth information, achieving localization without manual initialization and automatic recovery of localization after interruption. In step (3), the network used to generate the descriptor is called FDNet. The input of FDNet is a 3-channel color image, and the output includes a single-channel grayscale image and an image descriptor. Before training, the relative total difference detexturing algorithm is used to remove image details that may cause interference. Then, the hole region is segmented using an adaptive threshold as hole information. The hole information can reflect the branch structure of the bronchus. The output grayscale image is optimized to be close to the pre-extracted image hole information. The hole information is introduced into the network as prior knowledge. Training based on positive and negative example contrastive learning involves using FDNet to extract descriptors for the baseline image, positive example image, and negative example image during each training iteration. Simultaneously, the prior knowledge of the hole information introduced into the network is supervised and constrained using the cross-entropy loss function. In the FDNet structure diagram, the blue blocks represent the encoding layer consisting of four sets of ResNet-v2 blocks, the yellow blocks represent the decoding layer consisting of four sets of convolutional layers, outputting a single-channel information map, and the green blocks represent the fully connected encoding layer, outputting a 1×128-dimensional descriptor. The cross-entropy loss function is: （1） in, Represents an image hole information map. An information graph representing the network output. and The median value ranges from 0 to 1; In order for the loss function to be able to distinguish between positive and negative examples during training, Triplet Loss is used as the network's loss function, defined as follows: （2） in, Represents the baseline image descriptor. A descriptor representing the sample. Descriptors representing negative samples; It is a hyperparameter representing the adjustment and The minimum distance between them will Set to 1; For the baseline image descriptors, positive sample descriptors, and negative sample descriptors extracted using FDNet, the Euclidean distance between the labeled baseline image descriptors and positive sample descriptors is reduced using TripletLoss, while the Euclidean distance between the baseline image descriptors and negative sample descriptors is increased, until the distance between the baseline image descriptors and positive sample descriptors is less than the distance between the baseline image descriptors and negative sample descriptors, and the difference between the two distances is not less than the set hyperparameter. The criterion for determining whether two images belong to the same sub-region is whether the Euclidean distance between the descriptors of the two images is less than 1. .

2. The bronchoscopy positioning method according to claim 1, characterized in that: Before step (1), a 3D mesh model of the bronchus is reconstructed from the CT image using threshold segmentation, and the bronchus is divided into different sub-regions according to the branching structure of the bronchus; each branch with a length of less than 1 cm is treated as a separate sub-region, and for the bronchus with a length of more than 2 cm, it is divided into two sub-regions, one before and one after, and located in the segment closer to the smaller branch.

3. The bronchoscopy positioning method according to claim 2, characterized in that: In step (2), images containing clear hole information are selected from the virtual image set as keyframe datasets. In addition, a pose optimization dataset for precise positioning is generated using a virtual bronchoscope system. The sampling method for the pose optimization dataset is to generate annular regions with radii increasing by 1.5 mm in a cross section with a 1 mm interval near the end of the sub-region. In each annular region, 8×n images with an angle of 0~10° to the center line direction are randomly generated evenly, where n represents the nth annular region. If the radius is less than 1.5 mm, only the points on the center line are used as reference positions.

4. The bronchoscopy positioning method according to claim 3, characterized in that: In step (4), a depth map of the real image is obtained using a depth estimation network. The input of the depth estimation network is a 3-channel color bronchoscope image, and the output is a single-channel grayscale image representing the image depth. The network structure includes an encoder and a decoder. The encoder consists of a convolutional layer, 4 sets of ResNet-v2 blocks, and pooling layers. The decoder consists of 3 sets of convolutional layers and upsampling layers. The last convolutional layer in each ResNet-v2 block of the encoder is concatenated with the corresponding convolutional layer in the decoder.

5. The bronchoscopic positioning method according to claim 4, characterized in that: In step (4), the corresponding pixels between the depth map gold standard and the network prediction results are used. The loss is used to measure the difference in brightness between the predicted result and the label. The structural similarity loss (SSIM) measures the structural difference between the predicted result and the label. The total loss function is as follows: （3） in, As the gold standard for depth mapping, Network prediction results This parameter balances structural and luminance loss, and is set to 0.

85. SSIM measures the structural similarity between two images using the covariance between them, as defined below: （4） Where A and B represent the input images, and These represent the pixel mean values ​​of images A and B, respectively. and The variances of images A and B are represented. This represents the covariance of images A and B.

6. The bronchoscopy positioning method according to claim 5, characterized in that: In step (4), in order to improve the processing speed, the image is downsampled with a spacing of 4 before depth estimation is performed. After obtaining the depth map, the depth map is divided into 8 sectors with the image center as the center. Each sector is divided into inner, middle and outer parts according to the distance from the pixel to the image center, resulting in 24 regions. Starting from the region with the largest average gray value, a 24-dimensional vector is obtained by differential hashing in a clockwise order from the outside to the inside. The 8-bit gray histograms of the inner, middle and outer parts of the depth map are statistically analyzed. The statistical values ​​are normalized to between 0 and 2 to obtain 3 8-dimensional vectors. The two feature vectors obtained are combined into a 48-dimensional vector as a depth descriptor.

7. The bronchoscopy positioning method according to claim 6, characterized in that: Before bronchoscopy, this method calculates the depth descriptor of the optimized dataset in each sub-region to generate an optimized descriptor set. During bronchoscopy, the depth descriptor of the current input image is first calculated, and the most similar descriptor is retrieved from the optimized descriptor set. The pose of the corresponding image is used as the current bronchoscopy pose. Since the image is circularly symmetric, the direction of the line connecting the image center and the sub-region with the largest average depth in the outer ring is taken as the principal direction. The bronchoscopy pose is rotated around the principal optical axis so that the principal direction of the input image coincides with the principal direction of the corresponding virtual image, thus obtaining the final localization result.

8. The apparatus for the bronchoscopy positioning method according to claim 1, characterized in that: It includes: The virtual camera module is configured as a virtual camera whose movement direction is controlled by adding or subtracting Euler angles. The intrinsic parameters of the virtual camera are set to be consistent with the intrinsic parameters of the actual bronchoscope. In order to simulate the lighting conditions in the real environment, a cone-shaped light source is set to follow the movement of the virtual bronchoscope. The spectrum and illumination range of the light source are adjusted according to the real conditions, and texture maps are made using real endoscope graphics. The data acquisition module is configured to record color images and their corresponding depth maps and bronchoscope poses at a speed of 30fps using a virtual endoscope. During the virtual image acquisition stage, the virtual bronchoscope is used to acquire the corresponding color images, depth images and pose data from the reconstructed 3D model, and the obtained data is divided into corresponding sub-regions to generate a virtual image set. The descriptor module is configured to train the image descriptor extraction network using a positive and negative example comparison method, and to achieve sub-region localization of the bronchoscope through descriptor retrieval; The depth information module is configured to estimate the depth information of the image using a generative network and obtain the pose of the bronchoscope based on the depth information.

Images