Bronchoscope automatic navigation method and device, electronic equipment and storage medium

Abstract

The invention provides an automatic navigation method and device for a bronchoscope, electronic equipment and a storage medium, and relates to the technical field of bronchoscopes. The automatic navigation method for the bronchoscope comprises the following steps: in response to starting of bronchoscope examination operation, acquiring an established virtual bronchoscope model of a target patient; according to the virtual bronchoscope model, acquiring a navigation path of bronchoscope examination, so that an operator drives the bronchoscope to move towards the region of interest in the bronchus of the target patient according to the navigation path; in response to a received live-action image, shot by the bronchoscope at the current position, in the bronchus of the target patient, depth information is extracted from the live-action image, and a live-action depth map corresponding to the current position is obtained; extracting point cloud information from the live-action depth map to obtain the point cloud of the current position; and obtaining the current pose of the bronchoscope by using a projection function according to the live-action depth map and the point cloud of the current position. The automatic navigation precision of the bronchoscope can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of bronchoscopes, and in particular to a bronchoscope automatic navigation method, device, electronic equipment and storage medium. Background Art

[0002] Bronchoscopy, as an important endoscopic examination method, plays a key role in clinical diagnosis and treatment. Traditionally, this technology relies on the doctor's experience and visual judgment to operate the bronchoscope and conduct deep observation and treatment in the respiratory tract. However, the complexity of the respiratory tract structure and the narrowness of the passage, especially when the lesion is located in the distal bronchioles, pose significant challenges to manual operation, which not only requires clinicians to have superb operating skills, but also may affect the accuracy of diagnosis and the effectiveness of treatment due to precision limitations.

[0003] In order to overcome these difficulties, bronchoscopic automatic navigation technology has emerged and has made significant progress. This technology combines three-dimensional imaging with advanced image processing technologies, such as computed tomography (CT) and convolutional neural networks (CNN), to achieve three-dimensional reconstruction of the patient's respiratory tract. Doctors can intuitively see the three-dimensional structure of the tracheal tree, real-time images under the bronchoscope, and virtual bronchial images through the computer screen, which greatly improves the accuracy and visualization of bronchoscopic navigation. The application of this technology not only reduces operational risks and patient trauma, but also significantly improves the observation and diagnosis capabilities of endobronchial lesions, showing broad application prospects in the fields of early diagnosis of bronchial lung cancer, tumor resection, and lesion biopsy. However, the current bronchoscopic automatic navigation technology still faces some challenges. Due to the complexity of the respiratory system, especially when dealing with abnormal bronchial morphology, structural deformation caused by respiratory action, or airway stenosis, the existing bronchoscopic navigation system may not be able to accurately adapt to these changes, resulting in positioning errors and insufficient accuracy.

[0004] Therefore, how to effectively improve the accuracy of bronchoscope automatic navigation is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems existing in the prior art, the present invention provides a bronchoscope automatic navigation method, device, electronic equipment and storage medium to effectively improve the accuracy of bronchoscope automatic navigation.

[0006] The present invention provides a bronchoscope automatic navigation method, comprising the following steps.

[0007] In response to the start of the bronchoscopic examination operation, a virtual bronchoscope model of the target patient that has been established is obtained; wherein the virtual bronchoscope model is a three-dimensional structural model of the bronchial tree established based on the lung airway map of the target patient; according to the virtual bronchoscope model, a navigation path for the bronchoscopic examination is obtained, so that the operator drives the bronchoscope to move to the region of interest in the bronchus of the target patient according to the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as the starting point, and the region of interest as the target point; in response to receiving a real-life image of the bronchus of the target patient taken by the bronchoscope at the current position, depth information is extracted from the real-life image to obtain a real-life depth map corresponding to the current position; point cloud information is extracted from the real-life depth map to obtain a point cloud at the current position; according to the real-life depth map and the point cloud at the current position, a projection function is used to obtain the current posture of the bronchoscope, so that the operator can adjust the movement path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

[0008] According to a bronchoscope automatic navigation method provided by the present invention, the depth information is extracted from the real-scene image to obtain the real-scene depth map corresponding to the current position, including: obtaining a feature map of the real-scene image; using a forward noising model to add random noise that obeys a normal distribution to the feature map to obtain a noise feature map; using an inverse denoising model to obtain the real-scene depth map according to the feature map and the noise feature map; wherein the inverse denoising model is a neural network model trained using a training data set, the training data set includes multiple groups of sample data with labels, the sample data is a noise feature map obtained by adding noise to real-scene images taken by the bronchoscope at different preset positions in the patient's bronchus using the forward noising model, and the label is the real-scene depth map corresponding to the sample data.

[0009] According to a bronchoscope automatic navigation method provided by the present invention, the step of obtaining the feature map of the real-scene image includes: using a multi-scale feature encoding network to process the real-scene image to obtain feature maps of multiple scales of the real-scene image; using a weighted sum algorithm to fuse the feature maps of the multiple scales to obtain an aggregated feature map of the real-scene image; and using the aggregated feature map as the feature map of the real-scene image.

[0010] According to a bronchoscope automatic navigation method provided by the present invention, the point cloud information is extracted from the real-scene depth map to obtain the point cloud of the current position, including: for each pixel point on the real-scene depth map, performing the following operations to obtain each data point of the point cloud: using the imaging parameters of the bronchoscope to perform coordinate transformation on the x-axis coordinate and y-axis coordinate of the pixel point in the real-scene depth map to obtain the x-axis coordinate and y-axis coordinate corresponding to the pixel point in the point cloud coordinate system; according to the depth value of the pixel point and the x-axis coordinate and y-axis coordinate corresponding to the pixel point in the point cloud coordinate system, obtain the data point in the point cloud corresponding to the pixel point.

[0011] According to a bronchoscope automatic navigation method provided by the present invention, the current posture of the bronchoscope is obtained by using a projection function based on the real-scene image and the point cloud of the current position, including: using a projection function, according to a first initial estimated posture of the bronchoscope, projecting each data point of the point cloud onto a two-dimensional plane corresponding to the real-scene image, to obtain a first projection point corresponding to each data point; using a loss function, according to a first distance between the first projection point corresponding to each data point of the point cloud and a pixel point of the real-scene image corresponding to the data point, to obtain a first reprojection error; using a preset algorithm, iteratively adjusting the first initial estimated posture of the bronchoscope to minimize the first reprojection error until an iteration termination condition is reached; and obtaining the current posture of the bronchoscope based on the posture of the bronchoscope when the first reprojection error is minimized.

[0012] According to a bronchoscope automatic navigation method provided by the present invention, the current posture of the bronchoscope is obtained according to the posture of the bronchoscope when the first reprojection error is minimized, including: determining the second initial estimated posture of the bronchoscope according to the posture of the bronchoscope when the first reprojection error is minimized; using a projection function, according to the second initial estimated posture, projecting each data point of the point cloud onto a two-dimensional plane corresponding to the real-scene image to obtain a second projection point corresponding to each data point; for all the projection data pairs corresponding to the data points of the point cloud, selecting multiple groups of projection data pairs that meet preset refined estimation conditions; wherein a group of the projection data pairs is composed of a second projection point corresponding to one of the data points and a pixel point group in the real-scene image corresponding to it. The preset refined estimation conditions include: each pixel point in the projection data pair is located within a preset range of the second projection point of the projection data pair, and the distance between the second projection point of the projection data pair and the third projection point of the corresponding data point on the real-life image of the target patient's bronchus taken by the bronchoscope at the previous position is less than a preset distance threshold; using a loss function, a second reprojection error is obtained according to the second distance between the second projection point and the pixel point of each group of the projection data pairs; using a preset algorithm, iteratively adjusting the second initial estimated posture of the bronchoscope to minimize the second reprojection error until the iteration termination condition is reached; and obtaining the current posture of the bronchoscope according to the posture of the bronchoscope when the second reprojection error is minimized.

[0013] The present invention also provides a bronchoscope automatic navigation device, comprising the following modules: a first acquisition module, for acquiring an established virtual bronchoscope model of a target patient in response to the start of a bronchoscope examination operation; wherein the virtual bronchoscope model is a three-dimensional structural model of a bronchial tree established based on a lung airway map of the target patient; a second acquisition module, for acquiring a navigation path for the bronchoscope examination based on the virtual bronchoscope model, so that the operator drives the bronchoscope to move toward an area of ​​interest in the bronchus of the target patient according to the navigation path; wherein the navigation path takes the upper part of the main trachea of ​​the target patient as a starting point , the region of interest is the target point; a third acquisition module is used to extract depth information from the real-scene image in response to receiving a real-scene image of the target patient's bronchus taken by the bronchoscope at the current position, and obtain a real-scene depth map corresponding to the current position; a fourth acquisition module is used to extract point cloud information from the real-scene depth map to obtain a point cloud at the current position; a fifth acquisition module is used to obtain the current posture of the bronchoscope based on the real-scene depth map and the point cloud at the current position using a projection function, so that the operator can adjust the moving path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the automatic bronchoscope navigation method as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the automatic bronchoscope navigation method as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned automatic bronchoscope navigation methods.

[0017] The bronchoscope automatic navigation method, device, electronic device and storage medium provided by the present invention can construct a point cloud at the current position by acquiring the real-life image of the bronchoscope at the current position in real time and extracting depth information and point cloud information therefrom. By using the real-life image and point cloud at the current position, combined with the projection function, the current position of the bronchoscope can be accurately calculated, and a clear virtual guidance interface with sufficient field of view can be provided. The operator can adjust the moving path of the bronchoscope in time based on the accurate current position information of the bronchoscope, realize tracked flight based on route planning, and overcome the positioning error and insufficient precision of the navigation path caused by the complexity of the respiratory system (such as abnormal bronchial morphology, structural deformation caused by respiratory action or airway stenosis, etc.). Thereby, the accuracy of automatic navigation of the bronchoscope can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic flow chart of the automatic bronchoscope navigation method provided by the present invention.

[0020] Figure 2 It is a flowchart of a method for obtaining a real-scene depth map corresponding to a current position provided by the present invention.

[0021] Figure 3 It is a flowchart of the method for obtaining the current posture of a bronchoscope provided by the present invention.

[0022] Figure 4It is a structural schematic diagram of the bronchoscope automatic navigation device provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Combine the following Figure 1-Figure 3 The automated bronchoscopic navigation method of the present invention is described.

[0026] Figure 1 is a flow chart of the automatic bronchoscope navigation method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: In response to the initiation of a bronchoscopic examination operation, an established virtual bronchoscopic model of a target patient is obtained.

[0027] Target patients are those for whom bronchoscopy is indicated.

[0028] The virtual bronchoscopy model is a three-dimensional structural model of the bronchial tree established based on the lung airway map of the target patient.

[0029] In the specific implementation process, the lung airway map of the target patient can be obtained in a variety of ways. For example, the lung airway map of the target patient can be obtained by performing a computed tomography (CT) scan on the target patient, which is not limited to the description of this specification.

[0030] In the specific implementation process, the lung airway map can be finely segmented to construct an accurate three-dimensional airway mesh model; based on the three-dimensional airway mesh model, a highly realistic virtual bronchoscope model is established using volume rendering technology. The virtual bronchoscope model can provide doctors with a more intuitive and comprehensive lung airway visualization tool.

[0031] Step 102: Obtain a navigation path for bronchoscopic examination according to the virtual bronchoscope model, so that the operator can drive the bronchoscope to move toward the region of interest in the bronchus of the target patient according to the navigation path.

[0032] The navigation path takes the upper part of the main trachea of ​​the target patient as the starting point and the region of interest as the target point.

[0033] In bronchoscopy, the region of interest (ROI) can include various structures or abnormal areas related to the pulmonary airways and their related lesions, such as nodular lesions that are manually or automatically outlined.

[0034] In the specific implementation process, the center line of the target patient's airway can be extracted based on the virtual bronchoscope model to obtain the optimal path to the target point. This path will be used as a navigation path to guide subsequent inspections or treatment operations. When determining the navigation path, it is necessary to ensure that the viewpoint is always centered to obtain the most comprehensive field of view; at the same time, the viewpoint must maintain a safe distance from the airway wall to avoid collision or damage; in addition, it is necessary to ensure that the entire path is strictly located inside the airway to ensure the accuracy and safety of the operation.

[0035] In the specific implementation process, the operator can be a variety of implementation entities and is not limited by the description of this specification.

[0036] For example, the operator may be a doctor, who may drive the bronchoscope to move toward the region of interest in the bronchus of the target patient according to the navigation path in the visualized virtual bronchoscope model displayed on the display.

[0037] For another example, the operator may also be an interventional robot, which may use a robotic arm to drive the bronchoscope to move toward the region of interest in the bronchus of the target patient according to a navigation path extracted from the virtual bronchoscope model.

[0038] Step 103 : In response to receiving a real-scene image of the bronchus of the target patient captured by the bronchoscope at the current position, extracting depth information from the real-scene image to obtain a real-scene depth map corresponding to the current position.

[0039] The real-scene image is the actual image captured by the bronchoscope in the bronchus of the target patient.

[0040] The real-scene depth map is a grayscale image in which the value of each pixel represents the distance from the corresponding point in the scene to the optical center of the bronchoscope.

[0041] In a specific implementation process, when the bronchoscope moves in the bronchus of the target patient, it can capture real-life images of the bronchus at preset time intervals or preset distance intervals, and send the captured real-life images to a processing device that executes the method provided by the present invention. After receiving the real-life images of the bronchus of the target patient captured by the bronchoscope at the current position, the processing device can extract depth information from the real-life images in a variety of ways to obtain a real-life depth map corresponding to the current position.

[0042] For an embodiment of extracting depth information from a real scene image to obtain a real scene depth map corresponding to the current position, see Figure 2 The relevant content in will not be repeated here.

[0043] Step 104: extract point cloud information from the real-scene depth map to obtain a point cloud at the current position.

[0044] In some embodiments, the following operations may be performed for each pixel point on the real scene depth map to obtain each data point of the point cloud: The imaging parameters of the bronchoscope are used to transform the x-axis and y-axis coordinates of the pixel points in the real-scene depth map to obtain the x-axis and y-axis coordinates of the pixel points in the point cloud coordinate system; the data points in the point cloud corresponding to the pixel points are obtained according to the depth value of the pixel points and the x-axis and y-axis coordinates of the pixel points in the point cloud coordinate system.

[0045] In a specific implementation process, the imaging parameters of the bronchoscope may include the focal length of the bronchoscope on the x and y axes of the camera coordinate system.

[0046] Just as an example, is any point in the real-life depth map, The pixel value in the real scene depth map represents The distance between the position point on the bronchus represented by and the optical center of the bronchoscope is the z-axis coordinate of the position point in the camera coordinate system of the bronchoscope. The coordinate system conversion model shown below can be used. The depth value of The corresponding x-axis coordinates and y-axis coordinates in the point cloud coordinate system are obtained The corresponding data points in the point cloud.

[0047] (1) in, and They represent the focal lengths of the bronchoscope on the x and y axes of the camera coordinate system, respectively, and can be obtained based on the bronchoscope parameters; x and y are any pixel point in the real-view depth map (for example, ) in the x-axis and y-axis coordinates of the real-life depth map; , , They are respectively the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the data point in the point cloud corresponding to any pixel point in the real-life depth map in the point cloud coordinate system.

[0048] For each pixel point in the real-scene depth map, the coordinate system conversion model shown in formula (1) can be used to obtain the data point in the point cloud corresponding to the pixel point, thereby obtaining the point cloud of the position where the bronchoscope is located when taking the real-scene image.

[0049] Step 105: According to the real-scene image and the point cloud at the current position, a projection function is used to obtain the current posture of the bronchoscope, so that the operator can adjust the moving path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

[0050] In the specific implementation process, the current position of the bronchoscope can be obtained by using a projection function based on the real-scene image and the point cloud of the current position in a variety of ways, which is not limited to the description of this specification.

[0051] For an embodiment of obtaining the current position of the bronchoscope by using a projection function according to the real scene image and the point cloud of the current position, see Figure 3 The relevant content in will not be repeated here.

[0052] During the specific implementation process, the operator can timely adjust the moving path of the bronchoscope at the next moment based on the current position of the bronchoscope to ensure that the bronchoscope maintains a safe distance from the airway wall of the bronchus.

[0053] Figure 2 is a flow chart of a method for obtaining a real scene depth map corresponding to a current position provided by the present invention, such as Figure 2 As shown, the method includes the following: Step 201: Obtain a feature map of a real scene image.

[0054] In the specific implementation process, the feature map of the real scene image can be obtained in a variety of ways, which are not limited to the description of this specification.

[0055] In some embodiments, a feature extraction model with a Swin Transformer structure as the backbone can be used to extract feature maps of real-scene images. The feature extraction model includes a multi-scale feature encoding network and an aggregation module. The multi-scale feature encoding network is used to divide the input real-scene image into blocks, map it into token variables containing position embedding information, and extract visual features at different scales; the aggregation module is used to aggregate visual features at different scales to obtain a feature map containing global and local information of the real-scene image, retaining more feature information of the original image.

[0056] In a specific implementation process, a multi-scale feature encoding network can be used to process the real scene image to obtain feature maps of multiple scales of the real scene image.

[0057] As an example, a real scene image is represented as k scales , then the multi-scale feature encoding network contains k groups of convolution and downsampling operations to predict the scale Feature map .

[0058] The aggregation module uses a weighted summation algorithm to fuse feature maps of multiple scales to obtain the aggregate feature map G of the real scene image. The calculation formula is as follows: (2) in, ; The expansion rate is The expansion convolution operation is used to increase the receptive field of the convolution kernel. Each feature map Resize to the same size by nearest interpolation.

[0059] The aggregated feature map obtained by formula (2) can be used as a visual condition to guide the inverse denoising model in step 203 to accurately estimate the real scene depth map corresponding to the real scene image in the depth latent space.

[0060] Finally, the aggregated feature map obtained above is used as the feature map of the real scene image.

[0061] Step 202: using the forward noise model, add random noise that obeys the normal distribution to the feature map to obtain a noise feature map.

[0062] The forward noising model is a latent variable model that can be used for generative tasks.

[0063] In the specific implementation process, the forward noise model can be the following diffusion process: : (3) in, is a conditional probability distribution, indicating the feature map of a given initial real scene image Under the condition of , the noise characteristic diagram at time step t The probability distribution of is a parameter for controlling the degree of noise addition; I is the unit matrix.

[0064] By using formula (3), random noise that obeys the normal distribution is added to the feature map of the real scene image, and different real scene images can be processed into images blurred by Gaussian noise, so that the inverse denoising model in the following steps can accurately predict the real scene depth map from the real scene image in the Gaussian distribution space according to the visual conditions obtained in step 201.

[0065] Step 203: using the inverse denoising model, obtain a real scene depth map according to the feature map and the noise feature map.

[0066] In this step, the monocular depth estimation task is achieved based on the feature map and noise feature map through the denoising process guided by visual conditions. , where c is the noise feature map and z is the final real scene depth map. Specifically, the depth distribution of the real scene depth map is iteratively corrected through the following formula: , converted into the final real-life depth map.

[0067] (4) in, is a conditional probability distribution, representing the real scene depth map at a given current time step t Under the condition of and noise feature map c, the real scene depth map of the previous time step t−1 The probability distribution of θ is the parameter of the model, which can be obtained through training. It is the commonly used symbol for Gaussian distribution (Normal Distribution); Indicates the time step t The noise variance at .

[0068] in, It is an inverse denoising model used to denoise images from noisy images. The noise is gradually removed to predict the original depth map The input of the inverse denoising model includes: the real scene depth map at the current time step t , the current time step t and the noise feature map c; the output is the real scene depth map of the previous time step t−1 The probability distribution of .

[0069] In the specific implementation process, the inverse denoising model can be constructed based on the neural network in a variety of ways. As an example, the inverse denoising model can include multiple convolutional layers to capture input image features and gradually remove noise; skip connections or residual blocks can be used between convolutional layers to enhance feature propagation and gradient flow; the time step t can be encoded as an additional input feature, or integrated into the network weight or activation function in a specific way.

[0070] The inverse denoising model is a neural network model trained using a training data set. The training data set includes multiple groups of sample data with labels. The sample data is a noise feature map obtained by adding noise to the real-life images taken by a bronchoscope at different preset positions in the patient's bronchi using a forward denoising model. The label is the real-life depth map corresponding to the sample data.

[0071] Figure 3 is a flow chart of a method for obtaining the current position of a bronchoscope provided by the present invention, such as Figure 3 As shown, the method includes the following: Step 301: Using a projection function, according to a first initial estimated position of the bronchoscope, project each data point of the point cloud onto a two-dimensional plane corresponding to the real-scene image to obtain a first projection point corresponding to each data point.

[0072] In a specific implementation process, the first initial estimated posture can be set based on experience or experimental results and is not limited to the description in this specification.

[0073] The projection function is used to project a point in three-dimensional space onto a two-dimensional image plane. The implementation of the projection function depends on the camera's intrinsic parameters (such as focal length, optical center position, radial and tangential distortion parameters, etc.) and the camera's extrinsic parameters (i.e., position, including rotation and translation).

[0074] Step 302: using the loss function, a first reprojection error is obtained according to a first distance between a first projection point corresponding to each data point of the point cloud and a pixel point of the real scene image corresponding to the data point.

[0075] The loss function may include but is not limited to: mean square error loss function, Huber loss function, etc.

[0076] In a specific implementation process, the first distance between the first projection point corresponding to each data point of the point cloud and the pixel point of the real scene image corresponding to the data point can be substituted into the loss function to obtain a first reprojection error.

[0077] Step 303: using a preset algorithm, iteratively adjust the first initial estimated position of the bronchoscope to minimize the first reprojection error until an iteration termination condition is reached.

[0078] In a specific implementation process, a variety of preset algorithms, such as a PnP (Perspective-n-Point) algorithm or an ICP (Iterative Closest Point) algorithm, etc., may be used to iteratively adjust the first initial estimated posture of the bronchoscope.

[0079] The termination condition may include that the first reprojection error is minimized, or the number of iterations reaches a preset distance threshold, etc.

[0080] The above steps 301 to 303 can be expressed using the following formula.

[0081] (5) in, represents the Huber loss function, is a matching pair consisting of the first projection point corresponding to the jth data point of the point cloud and the pixel point of the real scene image corresponding to the data point; Represents the jth pixel of the kth frame of the real scene image; is the projection function, which uses the camera to estimate the pose The jth data point of the point cloud Project it onto the two-dimensional plane corresponding to the k-th frame of the real scene image; It means finding the parameter P that minimizes the function that follows.

[0082] Step 304: Obtain the current posture of the bronchoscope according to the posture of the bronchoscope when the first reprojection error is minimized.

[0083] In some embodiments, the posture of the bronchoscope when the first reprojection error is minimized can be used as the current posture of the bronchoscope.

[0084] In some embodiments, in order to obtain a more accurate estimation result of the current position and posture of the bronchoscope and make the navigation process more suitable for bronchial scenes, based on the position and posture of the bronchoscope obtained above when the first reprojection error is minimized, by presetting a refined estimation condition, only some data points in the point cloud in front of the bronchoscope are selected to perform a refined estimation of the position and posture of the bronchoscope. The specific steps are as follows.

[0085] A second initial estimated pose of the bronchoscope is determined based on the pose of the bronchoscope when the first reprojection error is minimized.

[0086] During the specific implementation process, the posture of the bronchoscope when the first reprojection error is minimized can be used as the second initial estimated posture, and the estimated posture of the bronchoscope can be adjusted starting from the second initial estimated posture.

[0087] Using the projection function, according to the second initial estimated posture, each data point of the point cloud is projected onto a two-dimensional plane corresponding to the real scene image to obtain a second projection point corresponding to each data point.

[0088] Among the projection data pairs corresponding to all data points of the point cloud, multiple groups of projection data pairs that meet the preset refinement estimation conditions are selected. A group of projection data pairs consists of a second projection point corresponding to a data point and a pixel point in the real scene image corresponding to the second projection point.

[0089] The preset refinement estimation condition includes: each pixel point in the projection data pair is located within a preset range of the second projection point of the projection data pair. , only from the preset range of the second projection point on the two-dimensional plane corresponding to the real image Select the corresponding point from the pixels in is the preset distance threshold.

[0090] The preset refinement estimation condition also includes: and the distance between the second projection point of the projection data pair and the third projection point of the corresponding data point on the real scene image of the bronchus of the target patient taken by the bronchoscope at the previous position is less than the preset distance threshold. The constraint condition is expressed by the formula as follows.

[0091] , (6) in is the preset distance threshold, is the estimated posture of the bronchoscope in the previous position; is the second projection point corresponding to point b; is the third projection point corresponding to point b.

[0092] Formula (6) constrains the data points involved in the refined estimation of the bronchoscope pose to meet the law of continuous slow motion of the camera between adjacent frames. This can avoid the bronchoscope pose estimation bias caused by abnormal data points in the point cloud and effectively improve the accuracy of bronchoscope pose estimation in bronchial scenes with sparse textures.

[0093] The loss function is used to obtain a second reprojection error according to a second distance between a second projection point and a pixel point of each group of projection data pairs.

[0094] Using a preset algorithm, the second initial estimated pose of the bronchoscope is iteratively adjusted to minimize the second reprojection error until the iteration termination condition is reached.

[0095] According to the posture of the bronchoscope when the second projection error is minimized, the current posture of the bronchoscope is obtained.

[0096] The following is a description of the automatic bronchoscope navigation device provided by the present invention. The automatic bronchoscope navigation device described below and the automatic bronchoscope navigation method described above can be referenced to each other.

[0097] Figure 4 Schematic diagram of the structure of the automatic navigation device for bronchoscope provided by the present invention. Figure 4 As shown, the device 400 includes the following modules.

[0098] The first acquisition module 410 is used to acquire an established virtual bronchoscope model of a target patient in response to the initiation of a bronchoscopic examination operation; wherein the virtual bronchoscope model is a three-dimensional structural model of a bronchial tree established based on a lung airway map of the target patient.

[0099] The second acquisition module 420 is used to acquire the navigation path of the bronchoscopic examination according to the virtual bronchoscope model, so that the operator can drive the bronchoscope to move toward the area of ​​interest in the bronchus of the target patient according to the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as the starting point, and the area of ​​interest is the target point.

[0100] The third acquisition module 430 is used to extract depth information from the real-scene image in the bronchus of the target patient captured by the bronchoscope at the current position in response to receiving the real-scene image, so as to obtain a real-scene depth map corresponding to the current position.

[0101] The fourth acquisition module 440 is used to extract point cloud information from the real-scene depth map to obtain the point cloud of the current position.

[0102] The fifth acquisition module 450 is used to obtain the current posture of the bronchoscope according to the real-scene depth map and the point cloud of the current position using a projection function, so that the operator can adjust the moving path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

[0103] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530 and a communications bus 540, wherein the processor 510, the communications interface 520 and the memory 530 communicate with each other via the communications bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a bronchoscope automatic navigation method, the method comprising: in response to the start of a bronchoscope examination operation, obtaining an established virtual bronchoscope model of a target patient; wherein the virtual bronchoscope model is a three-dimensional structural model of a bronchial tree established based on a lung airway map of the target patient; according to the virtual bronchoscope model, obtaining a navigation path for the bronchoscope examination, so that the operator drives the bronchoscope to move to an area of ​​interest in the bronchus of the target patient according to the navigation path; wherein the navigation path is based on the target patient. The upper part of the main bronchus of the target patient is taken as the starting point, and the region of interest is taken as the target point; in response to receiving a real-scene image of the target patient's bronchus taken by the bronchoscope at the current position, depth information is extracted from the real-scene image to obtain a real-scene depth map corresponding to the current position; point cloud information is extracted from the real-scene depth map to obtain a point cloud at the current position; according to the real-scene depth map and the point cloud at the current position, a projection function is used to obtain the current posture of the bronchoscope, so that the operator can adjust the moving path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

[0104] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0105] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the bronchoscope automatic navigation method provided by the above methods, the method comprising: in response to the start of a bronchoscopic examination operation, obtaining an established virtual bronchoscope model of a target patient; wherein the virtual bronchoscope model is a three-dimensional structural model of a bronchial tree established based on a lung airway map of the target patient; based on the virtual bronchoscope model, obtaining a navigation path for the bronchoscope examination so that the operator drives the bronchoscope in accordance with the navigation path. The target patient moves toward the region of interest within the bronchus; wherein the navigation path takes the upper part of the main bronchus of the target patient as a starting point, and the region of interest as a target point; in response to receiving a real-scene image of the target patient's bronchus taken by the bronchoscope at a current position, extracting depth information from the real-scene image to obtain a real-scene depth map corresponding to the current position; extracting point cloud information from the real-scene depth map to obtain a point cloud of the current position; and obtaining the current posture of the bronchoscope using a projection function based on the real-scene depth map and the point cloud of the current position, so that the operator can adjust the moving path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

[0106] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the bronchoscope automatic navigation method provided by the above-mentioned methods, the method comprising: in response to the initiation of a bronchoscope examination operation, obtaining an established virtual bronchoscope model of a target patient; wherein the virtual bronchoscope model is a three-dimensional structural model of a bronchial tree established based on a lung airway map of the target patient; and obtaining a navigation path for the bronchoscope examination based on the virtual bronchoscope model, so that the operator drives the bronchoscope to move toward an interested region in the bronchus of the target patient according to the navigation path. wherein the navigation path takes the upper part of the main bronchus of the target patient as a starting point and the region of interest as a target point; in response to receiving a real-scene image of the target patient's bronchus taken by the bronchoscope at a current position, extracting depth information from the real-scene image to obtain a real-scene depth map corresponding to the current position; extracting point cloud information from the real-scene depth map to obtain a point cloud at the current position; and obtaining a current posture of the bronchoscope using a projection function based on the real-scene depth map and the point cloud at the current position, so that the operator can adjust the moving path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bronchoscope automatic navigation method, characterized in that: include: In response to the initiation of the bronchoscopic examination operation, obtaining an established virtual bronchoscopic model of the target patient; wherein the virtual bronchoscopic model is a three-dimensional structural model of the bronchial tree established according to the lung airway map of the target patient; According to the virtual bronchoscope model, a navigation path for the bronchoscopic examination is obtained, so that the operator drives the bronchoscope to move toward the region of interest in the bronchus of the target patient according to the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as the starting point, and the region of interest is the target point; In response to receiving a real-scene image of the bronchus of the target patient captured by the bronchoscope at the current position, extracting depth information from the real-scene image to obtain a real-scene depth map corresponding to the current position; Extracting point cloud information from the real-view depth map to obtain a point cloud at the current position; According to the real-life depth map and the point cloud of the current position, a projection function is used to obtain the current posture of the bronchoscope, so that the operator can adjust the moving path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

2. The bronchoscope automatic navigation method according to claim 1, characterized in that: The extracting depth information from the real scene image to obtain a real scene depth map corresponding to the current position includes: Acquire a feature map of the real scene image; Using a forward noise addition model, random noise that obeys a normal distribution is added to the feature map to obtain a noise feature map; The real-life depth map is obtained according to the feature map and the noise feature map by using an inverse denoising model; wherein the denoising model is a neural network model trained using a training data set, the training data set includes multiple groups of sample data with labels, the sample data is a noise feature map obtained by adding noise to the real-life image taken by the bronchoscope at different preset positions in the patient's bronchus by using the forward denoising model, and the label is the real-life depth map corresponding to the sample data.

3. The bronchoscope automatic navigation method according to claim 2, characterized in that: The step of obtaining a feature map of the real scene image comprises: Processing the real scene image using a multi-scale feature encoding network to obtain feature maps of multiple scales of the real scene image; Using a weighted sum algorithm, the feature maps of the multiple scales are fused to obtain an aggregated feature map of the real scene image; The aggregated feature map is used as the feature map of the real scene image.

4. The bronchoscope automatic navigation method according to claim 1, characterized in that: The step of extracting point cloud information from the real scene depth map to obtain the point cloud at the current position includes: For each pixel point on the real scene depth map, perform the following operations to obtain each data point of the point cloud: Using the imaging parameters of the bronchoscope, coordinate transformation is performed on the x-axis coordinate and the y-axis coordinate of the pixel point in the real-view depth map to obtain the x-axis coordinate, the y-axis coordinate and the z-axis coordinate corresponding to the pixel point in the point cloud coordinate system; According to the depth value of the pixel point, the x-axis coordinate, y-axis coordinate and z-axis coordinate corresponding to the pixel point in the point cloud coordinate system, the data point in the point cloud corresponding to the pixel point is obtained.

5. The bronchoscope automatic navigation method according to claim 1, characterized in that: The method of obtaining the current posture of the bronchoscope by using a projection function according to the real scene image and the point cloud of the current position includes: Using a projection function, according to a first initial estimated position of the bronchoscope, each data point of the point cloud is projected onto a two-dimensional plane corresponding to the real scene image to obtain a first projection point corresponding to each data point; Obtaining a first reprojection error according to a first distance between the first projection point corresponding to each data point of the point cloud and a pixel point of the real scene image corresponding to the data point by using a loss function; Iteratively adjusting the first initial estimated position of the bronchoscope by using a preset algorithm to minimize the first reprojection error until an iteration termination condition is reached; The current position of the bronchoscope is obtained according to the position of the bronchoscope when the first projection error is minimized.

6. The bronchoscope automatic navigation method according to claim 5, characterized in that: The step of obtaining the current posture of the bronchoscope according to the posture of the bronchoscope when the first re-projection error is minimized includes: Determining a second initial estimated pose of the bronchoscope according to the pose of the bronchoscope when the first re-projection error is minimized; Using a projection function, according to the second initial estimated pose, project each data point of the point cloud onto a two-dimensional plane corresponding to the real scene image to obtain a second projection point corresponding to each data point; For all the projection data pairs corresponding to the data points of the point cloud, multiple groups of projection data pairs that meet preset refinement estimation conditions are selected; wherein a group of the projection data pairs is composed of a second projection point corresponding to one of the data points and a pixel point in the real-scene image corresponding thereto, and the preset refinement estimation conditions include: the pixel point in each of the projection data pairs is located within a preset range of the second projection point of the projection data pair, and the distance between the second projection point of the projection data pair and a third projection point of the corresponding data point on the real-scene image of the bronchus of the target patient taken by the bronchoscope at the previous position is less than a preset distance threshold; Using the loss function, according to the second distance between the second projection point and the pixel point of each group of the projection data pair, a second reprojection error is obtained; Iteratively adjusting the second initial estimated position of the bronchoscope by using a preset algorithm to minimize the second re-projection error until an iteration termination condition is reached; The current position of the bronchoscope is obtained according to the position of the bronchoscope when the second projection error is minimized.

7. A bronchoscope automatic navigation device, characterized in that: include: A first acquisition module is used to acquire an established virtual bronchoscope model of a target patient in response to the initiation of a bronchoscopic examination operation; wherein the virtual bronchoscope model is a three-dimensional structural model of a bronchial tree established based on a lung airway map of the target patient; A second acquisition module is used to acquire a navigation path for the bronchoscopic examination according to the virtual bronchoscope model, so that the operator drives the bronchoscope to move toward the region of interest in the bronchus of the target patient according to the navigation path; wherein the navigation path takes the upper part of the main bronchus of the target patient as a starting point, and the region of interest is a target point; A third acquisition module is used for extracting depth information from the real-scene image in response to receiving the real-scene image of the bronchus of the target patient taken by the bronchoscope at the current position, so as to obtain a real-scene depth map corresponding to the current position; A fourth acquisition module is used to extract point cloud information from the real scene depth map to obtain the point cloud of the current position; The fifth acquisition module is used to obtain the current posture of the bronchoscope according to the real-scene depth map and the point cloud of the current position using a projection function, so that the operator can adjust the moving path of the bronchoscope at the next moment based on the current posture of the bronchoscope.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the automatic bronchoscope navigation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the automatic bronchoscope navigation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the automatic bronchoscope navigation method according to any one of claims 1 to 6 is implemented.

Images