Bronchoscope automatic navigation method and device, electronic equipment and storage medium

Abstract

The application provides a bronchoscope automatic navigation method and device, electronic equipment and storage medium, and relates to the technical field of bronchoscope. The bronchoscope automatic navigation method comprises the following steps: in response to the start of bronchoscope examination operation, a virtual bronchoscope model of a target patient is obtained; according to the virtual bronchoscope model, a navigation path of bronchoscope examination is obtained, so that the operator drives the bronchoscope to move to the region of interest in the bronchus of the target patient according to the navigation path; in response to receiving a real scene image of the bronchus of the target patient taken by the bronchoscope at the current position, depth information is extracted from the real scene image to obtain a real scene depth map corresponding to the current position; point cloud information is extracted from the real scene depth map to obtain the point cloud of the current position; and according to the real scene depth map and the point cloud of the current position, the current pose of the bronchoscope is obtained by using a projection function. The application can effectively improve the accuracy of bronchoscope automatic navigation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bronchoscopy, and in particular to a bronchoscope automatic navigation method and device, electronic equipment and storage medium. BACKGROUND

[0002] Bronchoscopy, as an important endoscopic examination method, plays a key role in clinical diagnosis and treatment. Traditionally, this technique relies on the experience and visual judgment of doctors to operate the bronchoscope, deep into the respiratory tract for observation and treatment. However, the complexity of the respiratory tract structure and the narrowness of the channel, especially when the lesion is located in the distal bronchioles, pose significant challenges to manual operation, requiring clinicians to have superb operating skills, and the precision limitation may affect the accuracy of diagnosis and the effectiveness of treatment.

[0003] To overcome these difficulties, bronchoscope automatic navigation technology has emerged and made significant progress. This technology combines three-dimensional imaging and advanced image processing techniques, such as computed tomography (CT) and convolutional neural networks (CNN), to achieve three-dimensional reconstruction of the patient's respiratory tract. Through the computer screen, doctors can visually see the three-dimensional structure of the tracheal tree, real-time images under the bronchoscope, and virtual bronchial pictures, greatly improving the accuracy and visualization level of bronchoscope navigation. The application of this technology not only reduces the risk of operation and patient trauma, but also significantly improves the observation and diagnosis of bronchial lesions, showing broad application prospects in the fields of early diagnosis of bronchial lung cancer, tumor resection, and lesion biopsy. However, the current bronchoscope automatic navigation technology still faces some challenges. Due to the complexity of the respiratory system, especially in dealing with abnormal bronchial morphology, structural deformation caused by respiration, or airway stenosis, existing bronchoscope navigation systems may not accurately adapt to these changes, resulting in positioning errors and insufficient precision.

[0004] Therefore, how to effectively improve the precision of bronchoscope automatic navigation is a technical problem to be solved. SUMMARY

[0005] In view of the above problems existing in the prior art, the present application provides a bronchoscope automatic navigation method, device, electronic equipment and storage medium to effectively improve the precision of bronchoscope automatic navigation.

[0006] The present application provides a bronchoscope automatic navigation method, comprising the following steps.

[0007] In response to starting of the bronchoscopy operation, a virtual bronchoscope model of the established target patient is acquired; wherein the virtual bronchoscope model is a three-dimensional structure model of a bronchial tree established according to a pulmonary airway map of the target patient; according to the virtual bronchoscope model, a navigation path of the bronchoscopy is acquired, so that the operator drives the bronchoscope to move in the bronchus of the target patient to the region of interest according to the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as a starting point, and the region of interest is a target point; in response to receiving a real scene image of the bronchus of the target patient taken by the bronchoscope at a current position, depth information is extracted from the real scene image to obtain a real scene depth map corresponding to the current position; point cloud information is extracted from the real scene depth map to obtain a point cloud of the current position; according to the real scene depth map and the point cloud of the current position, a current pose of the bronchoscope is obtained by using a projection function, so that the operator adjusts the moving path of the bronchoscope at the next time based on the current pose of the bronchoscope.

[0008] According to the bronchoscope automatic navigation method provided by the application, the extracting of the depth information from the real scene image to obtain the real scene depth map corresponding to the current position comprises: acquiring a feature map of the real scene image; adding random noise obeying a normal distribution to the feature map by using a forward noise adding model to obtain a noise feature map; and obtaining the real scene depth map according to the feature map and the noise feature map by using an inverse denoising model; wherein the inverse denoising model is a neural network model trained by using a training data set, the training data set comprises a plurality of groups of sample data carrying labels, the sample data is a noise feature map obtained by adding noise to a real scene image taken by the bronchoscope at different preset positions in the bronchus of a patient by using the forward noise adding model, and the label is a real scene depth map corresponding to the sample data.

[0009] According to the bronchoscope automatic navigation method provided by the application, the acquiring of the feature map of the real scene image comprises: processing the real scene image by using a multi-scale feature encoding network to obtain a plurality of scale feature maps of the real scene image; fusing the plurality of scale feature maps by using a weighted summation algorithm to obtain an aggregated feature map of the real scene image; and taking the aggregated feature map as the feature map of the real scene image.

[0010] According to the bronchoscope automatic navigation method provided by the application, the point cloud information is extracted from the real scene depth map to obtain the point cloud of the current position, which comprises: for each pixel point on the real scene depth map, the following operations are performed to obtain each data point of the point cloud: the x-axis coordinate and y-axis coordinate of the pixel point in the real scene depth map are transformed by using the imaging parameters of the bronchoscope to obtain the corresponding x-axis coordinate and y-axis coordinate of the pixel point in the point cloud coordinate system; and the data point corresponding to the pixel point in the point cloud is obtained according to the depth value of the pixel point, the corresponding x-axis coordinate and y-axis coordinate of the pixel point in the point cloud coordinate system.

[0011] According to the bronchoscope automatic navigation method provided by the application, the current pose of the bronchoscope is obtained by using a projection function according to the real scene image and the point cloud of the current position, which comprises: each data point of the point cloud is projected onto the two-dimensional plane corresponding to the real scene image according to the first initial estimated pose of the bronchoscope by using a projection function to obtain the first projection point corresponding to each data point; a first reprojection error is obtained according to the first distance between the first projection point corresponding to each data point of the point cloud and the pixel point of the real scene image corresponding to the data point by using a loss function; the first initial estimated pose of the bronchoscope is iteratively adjusted by using a preset algorithm to minimize the first reprojection error until the iteration termination condition is reached; and the current pose of the bronchoscope is obtained according to the pose of the bronchoscope when the first reprojection error is minimized.

[0012] According to the present invention, an automatic navigation method for a bronchoscope includes obtaining the current pose of the bronchoscope based on the pose of the bronchoscope when the first projection error is minimized. The method comprises: determining a second initial estimated pose of the bronchoscope based on the pose of the bronchoscope when the first projection error is minimized; projecting each data point of the point cloud onto a two-dimensional plane corresponding to the real-world image using a projection function according to the second initial estimated pose, thereby obtaining a second projection point corresponding to each data point; selecting multiple sets of projection data pairs that satisfy preset refinement estimation conditions from all projection data pairs corresponding to the data points of the point cloud; wherein each set of projection data pairs consists of a second projection point corresponding to a data point and a pixel group in the corresponding real-world image. The preset refinement estimation conditions include: each pixel in the projection data pair is located within a preset range of the second projection point of the projection data pair, and the distance between the second projection point of the projection data pair and the third projection point of its corresponding data point on the real-world image of the target patient's bronchus captured by the bronchoscope at the previous position is less than a preset distance threshold; using a loss function, a second projection error is obtained based on the second distance between the second projection point and the pixel in each projection data pair; using a preset algorithm, the second initial estimated pose of the bronchoscope is iteratively adjusted to minimize the second projection error until the iteration termination condition is met; the current pose of the bronchoscope is obtained based on the pose of the bronchoscope when the second projection error is minimized.

[0013] This invention also provides an automatic bronchoscopy navigation device, comprising the following modules: a first acquisition module, configured to acquire an established virtual bronchoscopy model of the target patient in response to the initiation of a bronchoscopy examination; wherein the virtual bronchoscopy model is a three-dimensional structural model of the bronchial tree established based on the airway diagram of the target patient's lungs; a second acquisition module, configured to acquire a navigation path for the bronchoscopy examination based on the virtual bronchoscopy model, so that the operator can drive the bronchoscope to move towards the region of interest within the bronchus of the target patient according to the navigation path; wherein the navigation path starts from the upper part of the main trachea of ​​the target patient. The region of interest is the target point; the third acquisition module is used to extract depth information from the real-scene image of the bronchus of the target patient taken by the bronchoscope at the current position in response to receiving the real-scene image of the bronchus of the target patient at the current position, and obtain the real-scene depth map corresponding to the current position; the fourth acquisition module is used to extract point cloud information from the real-scene depth map, and obtain the point cloud of the current position; the fifth acquisition module is used to obtain the current pose of the bronchoscope by using a projection function based on the real-scene depth map and the point cloud of the current position, so that the operator can adjust the movement path of the bronchoscope at the next moment based on the current pose of the bronchoscope.

[0014] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the bronchoscope automatic navigation method according to any one of the above when executing the computer program.

[0015] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executable by a processor to implement the bronchoscope automatic navigation method according to any one of the above.

[0016] The application further provides a computer program product, which includes a computer program, wherein the computer program is executable by a processor to implement the bronchoscope automatic navigation method according to any one of the above.

[0017] The bronchoscope automatic navigation method, device, electronic device and storage medium provided by the application can construct a point cloud of the current position by acquiring a real scene image of the bronchoscope at the current position in real time and extracting depth information and point cloud information from the real scene image. The current pose of the bronchoscope can be accurately calculated by using the real scene image and the point cloud of the current position and combining a projection function, and a clear and sufficient virtual guide interface can be provided. The operator can timely adjust the moving path of the bronchoscope based on the accurate current pose information of the bronchoscope, and realize rail flying based on route planning, thereby overcoming the positioning error and insufficient accuracy of the navigation path caused by the complexity of the respiratory system (such as abnormal bronchial morphology, structure deformation or airway stenosis caused by respiration, etc.). Therefore, the accuracy of bronchoscope automatic navigation can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0019] Figure 1 is a flowchart of the bronchoscope automatic navigation method provided by the application.

[0020] Figure 2 is a flowchart of the method for acquiring a real scene depth map corresponding to the current position provided by the application.

[0021] Figure 3 is a flowchart of the method for acquiring the current pose of the bronchoscope provided by the application.

[0022] Figure 4It is a structural schematic view of the bronchoscope automatic navigation device provided by the present application.

[0023] Figure 5 It is a structural schematic view of the electronic device provided by the present application. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0025] The present application will be described below with reference to the drawings. Figures 1-3 The bronchoscope automatic navigation method of the present application is described.

[0026] Figure 1 It is a flowchart of the bronchoscope automatic navigation method provided by the present application, as shown in the figure, the method comprises the following: Figure 1

[0027] Step 101, in response to the start of the bronchoscope examination operation, the virtual bronchoscope model of the target patient established is acquired.

[0028] The target patient is a patient who needs to be examined by bronchoscopy.

[0029] The virtual bronchoscope model is a three-dimensional structure model of the bronchial tree established according to the lung airway map of the target patient.

[0030] In the specific implementation process, the lung airway map of the target patient can be acquired in various ways, for example, the lung airway map of the target patient can be obtained by performing computed tomography (CT) on the target patient, which is not limited by the description in the present specification.

[0031] In the specific implementation process, the lung airway map can be finely segmented to construct an accurate three-dimensional airway grid model; based on the three-dimensional airway grid model, a highly realistic virtual bronchoscope model is established by using volume rendering technology. The virtual bronchoscope model can provide doctors with a more intuitive and comprehensive lung airway visualization tool.

[0032] Step 102, according to the virtual bronchoscope model, the navigation path of the bronchoscope examination is acquired, so that the operator drives the bronchoscope to move in the bronchus of the target patient to the region of interest according to the navigation path.

[0033] The navigation path takes the upper part of the main bronchus of the target patient as the starting point and the region of interest as the target point. ​

[0034] In bronchoscopy, the region of interest (ROI) can include a variety of regions where structures or abnormalities related to the airway of the lung and its related lesions are located, for example, a manually or automatically outlined nodule lesion.

[0035] In the specific implementation process, based on the virtual bronchoscope model, the center line of the airway of the target patient can be extracted to obtain the optimal path to the target point, which will serve as the navigation path to guide the subsequent examination or treatment operation. When determining the navigation path, it is necessary to ensure that the position of the viewpoint is always centered to obtain the most comprehensive field of view; at the same time, the viewpoint needs to maintain a safe distance from the airway wall surface to avoid collision or damage; in addition, it is also necessary to ensure that the entire path is strictly located inside the airway, thereby guaranteeing the accuracy and safety of the operation.

[0036] In the specific implementation process, the operator can be various implementation subjects, which are not limited by the description in this specification.

[0037] For example, the operator can be a doctor, who can drive the bronchoscope to move in the bronchus of the target patient towards the region of interest according to the navigation path in the visualized virtual bronchoscope model displayed on the display.

[0038] For another example, the operator can also be an interventional robot, which can drive the bronchoscope to move in the bronchus of the target patient towards the region of interest according to the navigation path extracted from the virtual bronchoscope model using a mechanical arm.

[0039] Step 103, in response to receiving the real scene image of the bronchus of the target patient taken by the bronchoscope at the current position, extracting depth information from the real scene image to obtain a real scene depth map corresponding to the current position.

[0040] The real scene image is an actual image taken by the bronchoscope in the bronchus of the target patient.

[0041] The real scene depth map is a gray-scale image, in which the value of each pixel point represents the distance from the corresponding point in the scene to the optical center of the bronchoscope.

[0042] In the specific implementation process, during the movement of the bronchoscope in the bronchus of the target patient, the bronchoscope can take real scene images of the bronchus at preset time intervals or preset distance intervals, and send the taken real scene images to the processing device for executing the method provided by the present application. After receiving the real scene image of the bronchus of the target patient taken by the bronchoscope at the current position, the processing device can extract depth information from the real scene image in various ways to obtain a real scene depth map corresponding to the current position.

[0043] For extracting depth information from the real scene image, one embodiment of obtaining the real scene depth map corresponding to the current position is described in detail in the related content of Figure 2 , which will not be repeated here.

[0044] Step 104, extracting point cloud information from the real scene depth map to obtain the point cloud of the current position.

[0045] In some embodiments, the following operations can be performed for each pixel point on the real scene depth map to obtain each data point of the point cloud:

[0046] Using the imaging parameters of the bronchoscope, the x-axis coordinate and the y-axis coordinate of the pixel point in the real scene depth map are coordinate-transformed to obtain the corresponding x-axis coordinate and y-axis coordinate of the pixel point in the point cloud coordinate system; according to the depth value of the pixel point, the corresponding x-axis coordinate and y-axis coordinate of the pixel point in the point cloud coordinate system, the data point corresponding to the pixel point in the point cloud is obtained.

[0047] In the specific implementation process, the imaging parameters of the bronchoscope can include the focal length of the bronchoscope on the x, y axes of the camera coordinate system.

[0048] For example only, For any point in the real scene depth map, The pixel value in the real scene depth map represents the distance between the position point on the bronchus represented by the bronchoscope and the optical center of the bronchoscope, i.e., the z-axis coordinate of the position point in the camera coordinate system of the bronchoscope. According to the depth value of , the corresponding x-axis coordinate and y-axis coordinate in the point cloud coordinate system, the data point corresponding to the point cloud can be obtained by using the coordinate conversion model shown below.

[0049] (1)

[0050] wherein, and respectively represent the focal length of the bronchoscope on the x, y axes of the camera coordinate system, which can be obtained according to the bronchoscope parameters; x, y respectively represent the x-axis coordinate and the y-axis coordinate of any pixel point (for example, ) in the real scene depth map in the real scene depth map; , , respectively represent the x-axis coordinate, the y-axis coordinate and the z-axis coordinate of the data point corresponding to any pixel point in the real scene depth map in the point cloud coordinate system.

[0051] ​For each pixel point in the real scene depth map, the coordinate transformation model shown in formula (1) can be used to obtain the data point in the point cloud corresponding to the pixel point, thereby obtaining the point cloud of the position where the bronchoscope captures the real scene image.

[0052] In step 105, according to the real scene image and the point cloud of the current position, the current pose of the bronchoscope is obtained by using the projection function, so that the operator adjusts the moving path of the bronchoscope at the next moment based on the current pose of the bronchoscope.

[0053] In the specific implementation process, the current pose of the bronchoscope can be obtained by using the projection function according to the real scene image and the point cloud of the current position in various ways, which is not limited by the description in this specification.

[0054] For an embodiment of obtaining the current pose of the bronchoscope according to the real scene image and the point cloud of the current position by using the projection function, please refer to the related content in Figure 3 , which will not be repeated here.

[0055] In the specific implementation process, the operator can timely adjust the moving path of the bronchoscope at the next moment based on the current pose of the bronchoscope, so as to ensure that the bronchoscope maintains a safe distance from the airway wall of the bronchus.

[0056] Figure 2 is the flowchart of the method for obtaining the real scene depth map corresponding to the current position provided by the application, as shown in Figure 2 , the method comprises the following steps:

[0057] In step 201, the feature map of the real scene image is obtained.

[0058] In the specific implementation process, the feature map of the real scene image can be obtained in various ways, which is not limited by the description in this specification.

[0059] In some embodiments, a feature extraction model with Swin Transformer structure as the backbone can be used to extract the feature map of the real scene image. The feature extraction model includes a multi-scale feature encoding network and an aggregation module. The multi-scale feature encoding network is used to block the input real scene image and map it to a token variable containing position embedding information, and extract visual features at different scales. The aggregation module is used to aggregate the visual features at different scales to obtain a feature map containing global and local information of the real scene image, which retains more feature information of the original image.

[0060] In the specific implementation process, the multi-scale feature encoding network can be used to process the real scene image to obtain the feature map of the real scene image at multiple scales.

[0061] For example only, a real scene image is represented as k scales Then the multi-scale feature encoding network contains k groups of convolution and down-sampling operations to predict the feature maps of scales . .

[0062] The aggregation module fuses the feature maps of multiple scales to obtain the aggregated feature map G of the real scene image using a weighted summation algorithm. The calculation formula is as follows:

[0063] (2)

[0064] wherein, ; represents an inflation convolution operation with an inflation rate of , which is used to increase the receptive field of the convolution kernel. Each feature map is adjusted to the same size by nearest interpolation.

[0065] The aggregated feature map obtained by formula (2) can be used as a visual condition to guide the inverse denoising model in step 203 to accurately estimate the real scene depth map corresponding to the real scene image in the deep latent space.

[0066] Finally, the aggregated feature map obtained above is used as the feature map of the real scene image.

[0067] Step 202: Adding random noise subject to a normal distribution to the feature map using a forward noise adding model to obtain a noise feature map.

[0068] The forward noise adding model is a latent variable model and can be used for generative tasks.

[0069] In the specific implementation process, the forward noise adding model can be a diffusion process as follows: :

[0070] (3)

[0071] wherein, is a conditional probability distribution, representing the probability distribution of the noise feature map at time step t under the condition that the feature map of the initial real scene image is given; is a parameter for controlling the degree of noise addition; and I is an identity matrix.

[0072] Adding random noise subject to a normal distribution to the feature map of the real scene image using formula (3) can process different real scene images into Gaussian noise blurred images, and then make the inverse denoising model in the following steps accurately predict the real scene depth map from the real scene image according to the visual condition obtained in step 201 in the Gaussian distribution space.

[0073] Step 203, obtaining the real scene depth map according to the feature map and the noise feature map by using the reverse denoising model.

[0074] In this step, the monocular depth estimation task is realized according to the feature map and the noise feature map through the visual condition guided denoising process , wherein c is the noise feature map, and z is the finally obtained real scene depth map. The depth distribution of the real scene depth map is iteratively corrected through the following formula , and converted into the final real scene depth map.

[0075] (4)

[0076] , wherein is a conditional probability distribution, representing the probability distribution of the real scene depth map at the previous time step t-1 given the real scene depth map at the current time step t and the noise feature map c; θ is a parameter of the model, which can be obtained through training; is a commonly used symbol of Gaussian distribution (Normal Distribution); represents the noise variance at the time step t .

[0077] , wherein is a reverse denoising model, used to remove noise from the image with noise step by step to predict the original depth map . The input of the reverse denoising model includes: the real scene depth map at the current time step t, the current time step t and the noise feature map c; the output is the probability distribution of the real scene depth map at the previous time step t-1.

[0078] In the specific implementation process, the reverse denoising model can be constructed based on a neural network in various ways. For example only, the reverse denoising model can include multiple convolutional layers for capturing input image features and removing noise step by step; skip connections or residual blocks can be used between each convolutional layer to enhance feature propagation and gradient flow; the time step t can be encoded as an additional input feature, or integrated into the network's weights or activation functions in a specific way.

[0079] The inverse denoising model is a neural network model trained by a training data set, the training data set includes a plurality of groups of sample data carrying labels, the sample data is a noise feature map obtained by adding noise to the real scene image photographed by the bronchoscope at different preset positions in the bronchus of the patient by using the forward noise adding model, and the label is a real scene depth map corresponding to the sample data.

[0080] Figure 3 is a flowchart of the method for obtaining the current pose of the bronchoscope provided by the application, as Figure 3 shown, the method comprises the following steps:

[0081] Step 301, using a projection function, projecting each data point of the point cloud onto a two-dimensional plane corresponding to the real scene image according to the first initial estimated pose of the bronchoscope, to obtain a first projection point corresponding to each data point.

[0082] In the specific implementation process, the first initial estimated pose can be set according to experience or experimental results, and is not limited by the description in this specification.

[0083] The projection function is used to project points in three-dimensional space onto a two-dimensional image plane. The implementation of the projection function depends on the camera's intrinsic parameters (such as focal length, optical center position, radial and tangential distortion parameters, etc.) and the camera's extrinsic parameters (i.e. pose, including rotation and translation).

[0084] Step 302, using a loss function, obtaining a first reprojection error according to the first distance between the first projection point corresponding to each data point of the point cloud and the pixel point corresponding to the data point of the real scene image.

[0085] The loss function can include but is not limited to: mean square error loss function, Huber loss function, etc.

[0086] In the specific implementation process, the first distance between the first projection point corresponding to each data point of the point cloud and the pixel point corresponding to the data point of the real scene image can be brought into the loss function to obtain the first reprojection error.

[0087] Step 303, using a preset algorithm, iteratively adjusting the first initial estimated pose of the bronchoscope to minimize the first reprojection error until the iteration termination condition is reached.

[0088] In the specific implementation process, a variety of preset algorithms can be used to iteratively adjust the first initial estimated pose of the bronchoscope, such as PnP (Perspective-n-Point) algorithm or ICP (Iterative Closest Point) algorithm, etc.

[0089] The termination condition can include that the first reprojection error reaches the minimum, or the number of iterations reaches a preset distance threshold, etc.

[0090] The steps 301-303 can be expressed by the following formulas.

[0091] (5)

[0092] wherein, represents a Huber loss function, is a matching pair composed of a first projection point corresponding to the jth data point of the point cloud and a pixel point in the real scene image corresponding to the data point; represents the jth pixel point of the kth real scene image; is a projection function using the camera estimated pose projects the jth data point of the point cloud onto the two-dimensional plane corresponding to the kth real scene image; represents finding the parameter P that minimizes the function followed by.

[0093] Step 304: obtaining the current pose of the bronchoscope according to the pose of the bronchoscope when the first re-projection error is minimized.

[0094] In some embodiments, the pose of the bronchoscope when the first re-projection error is minimized can be taken as the current pose of the bronchoscope.

[0095] In some embodiments, in order to obtain a more accurate estimation result of the current pose of the bronchoscope and make the navigation process more suitable for the bronchial scene, the pose of the bronchoscope when the first re-projection error is minimized can be refined based on a predetermined refinement estimation condition to select only part of the data points in the point cloud in front of the bronchoscope for refinement estimation of the pose of the bronchoscope. The specific steps are as follows.

[0096] According to the pose of the bronchoscope when the first re-projection error is minimized, a second initial estimation pose of the bronchoscope is determined.

[0097] In the implementation process, the pose of the bronchoscope when the first re-projection error is minimized can be taken as the second initial estimation pose, and the estimation pose of the bronchoscope can be adjusted from the second initial estimation pose.

[0098] Using the projection function, each data point of the point cloud is projected onto the two-dimensional plane corresponding to the real scene image according to the second initial estimation pose to obtain a second projection point corresponding to each data point.

[0099] For all the projection data pairs corresponding to the data points of the point cloud, a plurality of groups of projection data pairs satisfying a predetermined refinement estimation condition are selected. A group of projection data pairs is composed of a second projection point corresponding to a data point and a pixel point in the real scene image corresponding to the data point.

[0100] wherein the preset refinement estimation condition comprises: the pixel point in each projection data pair is located within a preset range of the second projection point of the projection data pair. For example, for the data point of the point cloud , only the corresponding point is selected from the pixel points within the preset range of the second projection point of the data point on the two-dimensional plane corresponding to the real scene image , wherein is a preset distance threshold.

[0101] The preset refinement estimation condition further comprises: the distance between the second projection point of the projection data pair and the third projection point of the data point on the real scene image of the bronchoscope in the bronchus of the target patient in the last position is less than a preset distance threshold. The constraint condition is expressed by the following formula.

[0102] , (6)

[0103] wherein is a preset distance threshold, is the estimated pose of the bronchoscope in the last position; is the second projection point corresponding to the b point; is the third projection point corresponding to the b point.

[0104] Through formula (6), the data points participating in the refinement estimation of the pose of the bronchoscope are required to satisfy the rule of continuous slow motion of the camera between adjacent frames. Thus, the pose estimation error of the bronchoscope caused by abnormal data points in the point cloud can be avoided, and the accuracy of the pose estimation of the bronchoscope in the bronchial scene with sparse texture is effectively improved.

[0105] Using the loss function, the second reprojection error is obtained according to the second distance between the second projection point of each group of projection data pairs and the pixel point.

[0106] Using a preset algorithm, the second initial estimated pose of the bronchoscope is iteratively adjusted to minimize the second reprojection error until an iteration termination condition is reached.

[0107] According to the pose of the bronchoscope when the second reprojection error is minimized, the current pose of the bronchoscope is obtained.

[0108] The bronchoscope automatic navigation device provided by the present application is described below. The bronchoscope automatic navigation device described below can be referred to in correspondence with the bronchoscope automatic navigation method described above.

[0109] Figure 4 is a structural schematic diagram of the bronchoscope automatic navigation device provided by the present application. As Figure 4 shown, the device 400 includes the following modules.

[0110] The first obtaining module 410 is configured to, in response to the start of the bronchoscopy operation, obtain a virtual bronchoscope model of an established target patient; wherein the virtual bronchoscope model is a three-dimensional structure model of a bronchial tree established according to a pulmonary airway map of the target patient.

[0111] The second obtaining module 420 is configured to, according to the virtual bronchoscope model, obtain a navigation path of the bronchoscopy, so that the operator drives the bronchoscope to move in the bronchus of the target patient to a region of interest according to the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as a starting point, and the region of interest is a target point.

[0112] The third obtaining module 430 is configured to, in response to receiving a real scene image of the bronchus of the target patient taken by the bronchoscope at a current position, extract depth information from the real scene image to obtain a real scene depth map corresponding to the current position.

[0113] The fourth obtaining module 440 is configured to extract point cloud information from the real scene depth map to obtain a point cloud of the current position.

[0114] The fifth obtaining module 450 is configured to, according to the real scene depth map and the point cloud of the current position, obtain a current pose of the bronchoscope by using a projection function, so that the operator adjusts a moving path of the bronchoscope at a next time based on the current pose of the bronchoscope.

[0115] Figure 5 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 5As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute a bronchoscope automatic navigation method, which includes: in response to starting of a bronchoscopy operation, acquiring an established virtual bronchoscope model of a target patient; wherein the virtual bronchoscope model is a three-dimensional structure model of a bronchial tree established according to a lung airway map of the target patient; according to the virtual bronchoscope model, acquiring a navigation path of the bronchoscopy, so that an operator drives a bronchoscope to move in a bronchus of the target patient to a region of interest according to the navigation path; wherein the navigation path takes an upper part of a main bronchus of the target patient as a starting point, and the region of interest is a target point; in response to receiving a real scene image of the bronchus of the target patient taken by the bronchoscope at a current position, extracting depth information from the real scene image to obtain a real scene depth map corresponding to the current position; extracting point cloud information from the real scene depth map to obtain a point cloud of the current position; and according to the real scene depth map and the point cloud of the current position, using a projection function to obtain a current pose of the bronchoscope, so that the operator adjusts a moving path of the bronchoscope at a next time based on the current pose of the bronchoscope.

[0116] In addition, the logical instruction in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0117] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the bronchoscope automatic navigation method provided by the above-mentioned methods. The method comprises: in response to the start of a bronchoscopy operation, obtaining a virtual bronchoscope model of a target patient that has been established; wherein the virtual bronchoscope model is a three-dimensional structure model of a bronchial tree established according to a lung airway map of the target patient; obtaining a navigation path of the bronchoscopy according to the virtual bronchoscope model, so that the operator drives the bronchoscope to move in the bronchus of the target patient to a region of interest in accordance with the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as the starting point, and the region of interest is the target point; in response to receiving a real scene image of the bronchus of the target patient taken by the bronchoscope at a current position, extracting depth information from the real scene image to obtain a real scene depth map corresponding to the current position; extracting point cloud information from the real scene depth map to obtain a point cloud of the current position; and using a projection function to obtain a current pose of the bronchoscope according to the real scene depth map and the point cloud of the current position, so that the operator adjusts the moving path of the bronchoscope at the next time based on the current pose of the bronchoscope.

[0118] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the bronchoscope automatic navigation method provided by the above-mentioned methods. The method comprises: in response to the start of a bronchoscopy operation, obtaining a virtual bronchoscope model of a target patient that has been established; wherein the virtual bronchoscope model is a three-dimensional structure model of a bronchial tree established according to a lung airway map of the target patient; obtaining a navigation path of the bronchoscopy according to the virtual bronchoscope model, so that the operator drives the bronchoscope to move in the bronchus of the target patient to a region of interest in accordance with the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as the starting point, and the region of interest is the target point; in response to receiving a real scene image of the bronchus of the target patient taken by the bronchoscope at a current position, extracting depth information from the real scene image to obtain a real scene depth map corresponding to the current position; extracting point cloud information from the real scene depth map to obtain a point cloud of the current position; and using a projection function to obtain a current pose of the bronchoscope according to the real scene depth map and the point cloud of the current position, so that the operator adjusts the moving path of the bronchoscope at the next time based on the current pose of the bronchoscope.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0120] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A bronchoscope automatic navigation method, characterized by, The method comprises: in response to the start of the bronchoscopy operation, obtaining a virtual bronchoscope model of the established target patient; wherein the virtual bronchoscope model is a three-dimensional structure model of the bronchial tree established according to the lung airway map of the target patient; according to the virtual bronchoscope model, obtaining a navigation path of the bronchoscopy, so that the operator drives the bronchoscope to move in the bronchus of the target patient to the region of interest according to the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as the starting point, and the region of interest is the target point; in response to receiving the real scene image of the bronchus of the target patient taken by the bronchoscope at the current position, extracting depth information from the real scene image to obtain a real scene depth map corresponding to the current position; extracting point cloud information from the real scene depth map to obtain the point cloud of the current position; according to the real scene depth map and the point cloud of the current position, using a projection function to obtain the current pose of the bronchoscope, so that the operator adjusts the moving path of the bronchoscope at the next time based on the current pose of the bronchoscope; wherein the current pose of the bronchoscope is obtained by using the projection function according to the real scene depth map and the point cloud of the current position, comprising: using the projection function, projecting each data point of the point cloud onto the two-dimensional plane corresponding to the real scene depth map according to the first initial estimated pose of the bronchoscope, to obtain the first projection point corresponding to each data point; using a loss function, obtaining a first reprojection error according to the first distance between the first projection point corresponding to each data point of the point cloud and the pixel point of the real scene depth map corresponding to the data point; using a preset algorithm, iteratively adjusting the first initial estimated pose of the bronchoscope to minimize the first reprojection error until the iteration termination condition is reached; determining the second initial estimated pose of the bronchoscope according to the pose of the bronchoscope when the first reprojection error is minimized; using the projection function, projecting each data point of the point cloud onto the two-dimensional plane corresponding to the real scene depth map according to the second initial estimated pose, to obtain the second projection point corresponding to each data point; for all projection data pairs corresponding to the point cloud, selecting multiple groups of projection data pairs that satisfy a preset refined estimation condition; wherein a group of the projection data pair is composed of the second projection point corresponding to one data point and the pixel point corresponding thereto in the real scene depth map, and the preset refined estimation condition includes: the pixel point in each of the projection data pairs is within the preset range of the second projection point of the projection data pair, and the distance between the second projection point of the projection data pair and the third projection point of the data point corresponding thereto on the real scene depth map taken by the bronchoscope at the previous position is less than a preset distance threshold; using a loss function, obtaining a second reprojection error according to the second distance between the second projection point and the pixel point of each group of the projection data pair; iteratively adjust a second initial estimated pose of the bronchoscope using a preset algorithm to minimize the second re-projection error until an iteration termination condition is reached; obtain a current pose of the bronchoscope according to the pose of the bronchoscope when the second re-projection error is minimized.

2. The bronchoscope automatic navigation method of claim 1, wherein, The extracting depth information from the real scene image to obtain a real scene depth map corresponding to the current position comprises: obtaining a feature map of the real scene image; adding random noise conforming to a normal distribution to the feature map using a forward noise adding model to obtain a noise feature map; obtaining the real scene depth map according to the feature map and the noise feature map using an inverse noise removing model, wherein the noise removing model is a neural network model trained using a training data set, the training data set comprises a plurality of groups of sample data carrying labels, the sample data is a noise feature map obtained by adding noise to a real scene image captured by the bronchoscope at different preset positions in the bronchus of a patient using the forward noise adding model, and the label is a real scene depth map corresponding to the sample data.

3. The bronchoscope automatic navigation method of claim 2, wherein, The obtaining the feature map of the real scene image comprises: processing the real scene image using a multi-scale feature encoding network to obtain a plurality of scale feature maps of the real scene image; fusing the plurality of scale feature maps using a weighted summation algorithm to obtain an aggregated feature map of the real scene image; taking the aggregated feature map as the feature map of the real scene image.

4. The bronchoscope automatic navigation method of claim 1, wherein, The extracting point cloud information from the real scene depth map to obtain a point cloud of the current position comprises: for each pixel point on the real scene depth map, performing the following operations to obtain each data point of the point cloud: performing coordinate transformation on the x-axis coordinate and the y-axis coordinate of the pixel point in the real scene depth map using the imaging parameters of the bronchoscope to obtain the corresponding x-axis coordinate, y-axis coordinate and z-axis coordinate of the pixel point in the point cloud coordinate system; obtaining the data point of the pixel point in the point cloud according to the depth value of the pixel point, the corresponding x-axis coordinate, y-axis coordinate and z-axis coordinate of the pixel point in the point cloud coordinate system.

5. A bronchoscope automatic navigation device, characterized by, comprise: a first obtaining module configured to, in response to starting of a bronchoscopy operation, obtain a virtual bronchoscope model of a target patient that has been established, wherein the virtual bronchoscope model is a three-dimensional structure model of a bronchial tree established according to a lung airway map of the target patient; a second obtaining module configured to, according to the virtual bronchoscope model, obtain a navigation path of the bronchoscopy, so that an operator drives a bronchoscope to move in a bronchus of the target patient to a region of interest in accordance with the navigation path, wherein the navigation path takes an upper part of a main bronchus of the target patient as a starting point, and the region of interest is a target point; a third obtaining module configured to, in response to receiving a real scene image of the bronchus of the target patient captured by the bronchoscope at a current position, extract depth information from the real scene image to obtain a real scene depth map corresponding to the current position; a fourth obtaining module configured to extract point cloud information from the real scene depth map to obtain a point cloud of the current position. The fifth acquisition module is configured to obtain the current pose of the bronchoscope according to the real scene depth map and the point cloud of the current position by using a projection function, so that the operator adjusts the moving path of the bronchoscope at the next time based on the current pose of the bronchoscope. The current pose of the bronchoscope is obtained according to the real scene depth map and the point cloud of the current position by using a projection function, and includes the following steps: Each data point of the point cloud is projected onto the two-dimensional plane corresponding to the real scene depth map according to the first initial estimated pose of the bronchoscope by using a projection function, to obtain a first projection point corresponding to each data point; A first reprojection error is obtained according to the first distance between the first projection point corresponding to each data point of the point cloud and the pixel point of the real scene depth map corresponding to the data point by using a loss function; The first initial estimated pose of the bronchoscope is iteratively adjusted by using a preset algorithm to minimize the first reprojection error until an iteration termination condition is reached; The second initial estimated pose of the bronchoscope is determined according to the pose of the bronchoscope when the first reprojection error is minimized; Each data point of the point cloud is projected onto the two-dimensional plane corresponding to the real scene depth map according to the second initial estimated pose, to obtain a second projection point corresponding to each data point; For all projection data pairs corresponding to the data points of the point cloud, a plurality of projection data pairs satisfying a preset refined estimation condition are selected; wherein one projection data pair is composed of a second projection point corresponding to a data point and a pixel point in the real scene depth map corresponding to the data point, and the preset refined estimation condition includes that the pixel point in each projection data pair is within a preset range of the second projection point of the projection data pair, and the distance between the second projection point of the projection data pair and the third projection point of the data point on the real scene depth map of the bronchus of the target patient photographed by the bronchoscope at the previous position is less than a preset distance threshold; A second reprojection error is obtained according to the second distance between the second projection point and the pixel point of each projection data pair by using a loss function; The second initial estimated pose of the bronchoscope is iteratively adjusted by using a preset algorithm to minimize the second reprojection error until an iteration termination condition is reached; The current pose of the bronchoscope is obtained according to the pose of the bronchoscope when the second reprojection error is minimized.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the bronchoscope automatic navigation method of any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the bronchoscope automatic navigation method of any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the bronchoscope automatic navigation method of any one of claims 1 to 4.

Images