Intrabronchial navigation methods, devices, equipment and media

Abstract

The present invention provides a method, device, equipment and medium for intrabronchial navigation, including: obtaining navigation data collected in real time during the operation, the navigation data including the position coordinates of the first sensor, the bronchoscope image and the respiratory curve; determining the current bronchial tree model and the current planned path point cloud data corresponding to the current respiratory phase according to the respiratory curve; obtaining the current virtual coordinates of the first sensor in the medical image coordinate system according to the first conversion relationship and the position coordinates of the first sensor; determining whether the current intervention end is located on the planned path according to the bronchoscope image, the current bronchial tree model, the current planned path point cloud data and the current virtual coordinates. The present invention eliminates the influence of respiratory movement on the airway structure and improves the registration accuracy. The virtual planned path is mapped to the corresponding cavity of the bronchoscope image through path matching recognition, indicating the route of travel, and adaptively fitting movement and deformation to improve the accuracy of registration navigation.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a navigation method, device, equipment and medium during bronchial surgery. Background Art

[0002] Currently, intraoperative bronchial navigation typically generates a 3D airway model from a preoperative CT (Computed Tomography) image dataset. The patient's specific airway is measured, generating a point cloud of the measurements. The point cloud data is then aligned with all potential matches (possible paths) in the 3D airway model, and the potential match with the highest alignment score is extracted. Based on the point cloud data and the best potential match, deformable modeling of the local lumen is performed. The deformable model is then used to register (transform) the preoperative CT image dataset and the generated airway model to the patient's actual measured data. This approach first compares the point cloud data with the 3D airway model to determine the matching relationship between the local point cloud data and the corresponding airway branch. A suitable deformable model is then selected based on the matched data. However, a drawback of this approach is that, before data matching is performed, the actual airway position may shift to another branch in the 3D airway model due to significant deformation. This can result in an incorrect potential match with the highest alignment score, and subsequent deformable modeling cannot be performed correctly. Therefore, using similarity metrics cannot guarantee matching accuracy.

[0003] Therefore, it is necessary to propose a bronchial navigation method, device, equipment and medium to solve the above problems. Summary of the Invention

[0004] The object of the present invention is to provide a method, apparatus, device and medium for intrabronchial navigation, so as to improve the problem that the method using similarity measurement cannot guarantee the matching accuracy.

[0005] In a first aspect, the present invention provides a method for intrabronchial navigation, the method comprising:

[0006] S101: Acquire navigation data collected in real time during surgery, the navigation data including position coordinates of a first sensor, a bronchoscope image, and a respiratory curve, the first sensor and the bronchoscope being disposed at an interventional end of an interventional instrument;

[0007] S102: Determine the current bronchial tree model and the current planned path point cloud data corresponding to the current respiratory phase according to the respiratory curve;

[0008] S103: Obtaining current virtual coordinates of the first sensor in a medical image coordinate system according to a first conversion relationship and position coordinates of the first sensor, where the first conversion relationship is a conversion relationship between the first sensor coordinate system and the medical image coordinate system;

[0009] S104: Determine whether the intervention end is currently located on the planned path based on the bronchoscope image, the current bronchial tree model, the current planned path point cloud data, and the current virtual coordinates.

[0010] In one possible embodiment, before surgery, main airway skeleton point cloud data is extracted from the bronchial tree model, the interventional instrument is used to traverse the main airway, and trajectory point cloud data of the first sensor is collected. Based on the trajectory point cloud data and the main airway skeleton point cloud data, a rigid transformation matrix is ​​calculated for converting the first sensor coordinate system to the medical image coordinate system, and an initial global rigid transformation matrix is ​​obtained through iterative registration processing.

[0011] Obtaining the current virtual coordinates of the first sensor in the medical image coordinate system according to the first conversion relationship and the position coordinates of the first sensor includes:

[0012] According to the position coordinates of the first sensor and the initial global rigid transformation matrix, the current virtual coordinates of the first sensor in the medical image coordinate system are obtained.

[0013] In a possible embodiment, after step S104, the method further includes step S105: finding the nearest neighbor point of the current virtual coordinate in the current planned path point cloud data, and if the current virtual coordinate does not meet the accuracy requirement, then

[0014] When a first correction condition is met, an updated global rigid transformation matrix is ​​obtained by iteratively registering the current global rigid transformation matrix based on the historical trajectory point cloud data of the first sensor during the operation and the portion from the main trachea to the current branch in the currently planned path point cloud data. The first correction condition is that the current navigation depth is within a first depth threshold range or the inner diameter of the current branch is within a first inner diameter threshold range.

[0015] When the second correction condition is met, the planned path point cloud on the current branch is elastically registered and transformed according to the historical trajectory point cloud of the first sensor on the current branch to obtain the registered bronchial tree model and the registered branch planned path point cloud. The second correction condition is that the current navigation depth is within the second depth threshold range or the inner diameter of the current branch is within the second inner diameter threshold range.

[0016] In a possible embodiment, performing elastic registration transformation on the planned path point cloud on the current branch based on the historical trajectory point cloud of the first sensor on the current branch to obtain a registered bronchial tree model and a registered branch planned path point cloud includes:

[0017] Performing line segment fitting on the planned path point cloud on the current branch and the historical trajectory point cloud of the current branch respectively to obtain a first fitting line segment and a second fitting line segment;

[0018] performing a stretching transformation on the historical trajectory point cloud of the current branch to obtain a stretched historical trajectory point cloud according to a length deviation between the first fitting line segment and the second fitting line segment, and performing line segment fitting on the stretched historical trajectory point cloud to obtain a third fitting line segment;

[0019] According to the direction deviation between the first fitting line segment and the third fitting line segment, a rotation transformation is performed on the planned path point cloud on the current branch to obtain a registered bronchial tree model and a registered branch planned path point cloud.

[0020] In a possible embodiment, determining whether the interventional end is currently located on the planned path according to the bronchoscope image, the current bronchial tree model, the current planned path point cloud data, and the current virtual coordinates includes:

[0021] Segment the cavity area in the bronchoscopic image at the current moment and the previous moment, perform connected domain analysis and relative position matching on the segmentation results, and obtain the relative position relationship of the cavity at the current moment and the previous moment;

[0022] Obtaining a second transformation relationship between a bronchoscope coordinate system and a medical image coordinate system according to the first transformation relationship and the posture rotation information of the first sensor, wherein the navigation data further includes the posture rotation information of the first sensor;

[0023] intercepting the current bronchial airway three-dimensional model and the current bronchial tree model at the plane where the bronchoscope is located according to the second conversion relationship to obtain a virtual tracheal image at the current moment;

[0024] According to the virtual trachea image at the current moment, the relative position relationship of the cavities, and the point cloud data of the current planned path, it is determined whether the intervention end is currently located on the planned path.

[0025] In a possible embodiment, before the operation, medical images of the bronchi in the inspiratory and expiratory phases are collected, voxel segmentation of the bronchial airways is performed on the medical images in the inspiratory phase to obtain a three-dimensional model of the bronchial airways, and skeletonization is performed on the three-dimensional model of the bronchial airways to obtain a bronchial tree model and a planning path point cloud in the inspiratory phase.

[0026] In a possible embodiment, after skeletonizing the three-dimensional model of the bronchial airway to obtain the bronchial tree model and the planning path point cloud of the inspiratory phase, the method further includes:

[0027] Perform deformation registration on the medical images of the bronchial inspiratory and expiratory phases to obtain the expiratory phase deformation field;

[0028] Applying the expiratory phase deformation field to the bronchial tree model of the inspiratory phase to obtain the bronchial tree model of the expiratory phase and the planning path point cloud;

[0029] According to the expiratory phase deformation field, a second phase deformation field and a third phase deformation field are obtained by a linear interpolation method, where the second phase and the third phase are respiratory phases between the inhalation phase and the exhalation phase;

[0030] The second phase deformation field and the third phase deformation field are respectively applied to the bronchial tree model in the inspiratory phase to obtain the bronchial tree model and the planning path point cloud in the second phase and the bronchial tree model and the planning path point cloud in the third phase.

[0031] In a second aspect, an embodiment of the present invention further provides a bronchial intraoperative navigation device, comprising modules / units for executing any one of the possible design methods of the first aspect. These modules / units may be implemented in hardware, or in hardware executing corresponding software implementations.

[0032] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a processor and a memory. The memory is configured to store one or more computer programs; when the processor executes the one or more computer programs stored in the memory, the electronic device is capable of implementing any of the possible design methods of the first aspect.

[0033] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the electronic device executes any possible design method of the first aspect above.

[0034] In a fifth aspect, an embodiment of the present invention further provides a method comprising a computer program product, which, when the computer program product is run on an electronic device, enables the electronic device to execute any possible design of any of the above aspects.

[0035] The purpose of the present invention is to provide an improved bronchoscopic-assisted intraoperative navigation solution for bronchial surgery, which mainly solves the problem that traditional bronchial navigation methods cannot adaptively predict the deformation of the lungs and bronchial airways, the registration accuracy changes due to respiratory movement, and the registration error is large when navigating to the end of the bronchus. The present invention develops a respiratory motion fitting model to predict the tracheal deviation caused by respiratory movement; based on the sensor system of the interventional end of the interventional instrument and the bronchoscope image, a real-time path matching recognition algorithm is developed, and when obvious deformation of the bronchus is detected, local corresponding data is extracted to implement local elastic registration. In addition, the virtual planned path is mapped to the corresponding cavity of the bronchoscope image through path matching recognition, indicating the route of travel, and adaptively fitting the movement and deformation to improve the accuracy of registration navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the flow of the intrabronchial navigation method of the present invention.

[0037] Figure 2 Schematic diagram of the bronchial airway three-dimensional model and bronchial tree model in the intrabronchial navigation method of the present invention.

[0038] Figure 3a This is a schematic diagram of the intrabronchial navigation method of the present invention after rigid registration based on three corresponding points.

[0039] Figure 3b This is a schematic diagram of the initial registration fine-tuning based on the ICP method in the intrabronchial navigation method of the present invention.

[0040] Figure 4 This is a schematic diagram of the navigation planning path in the intrabronchial navigation method of the present invention on the bronchial tree model.

[0041] Figure 5a Schematic diagram of a bronchoscopic image at a previous moment in the intrabronchial navigation method of the present invention.

[0042] Figure 5b Schematic diagram of the bronchoscopic image at the current moment in the intrabronchial navigation method of the present invention.

[0043] Figure 5c Schematic diagram of a virtual tracheal image at the current moment in the intrabronchial navigation method of the present invention.

[0044] Figure 5d This is a schematic diagram of the current moment virtual tracheal image interception plane on the current bronchial tree model in the intrabronchial navigation method of the present invention.

[0045] Figure 6a This is a schematic diagram of the global rigid registration before dynamic correction in the intrabronchial navigation method of the present invention.

[0046] Figure 6b This is a schematic diagram of the global rigid registration after dynamic correction in the intrabronchial navigation method of the present invention.

[0047] Figure 7 Schematic diagram of the dynamic correction process of local elastic registration in the intrabronchial navigation method of the present invention.

[0048] Figure 8 A schematic diagram of the process for preparing respiratory cycle bronchial tree model data in the intrabronchial navigation method of the present invention.

[0049] Figure 9 Schematic diagram of the process of main airway coarse registration in the intrabronchial navigation method of the present invention.

[0050] Figure 10 The figure is a flowchart of a specific implementation of the intrabronchial navigation method of the present invention.

[0051] Figure 11 Schematic diagram of the intrabronchial navigation device of the present invention.

[0052] Figure 12 Schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0054] In view of the problems existing in the above technologies, an embodiment of the present invention provides a bronchial intraoperative navigation method, see Figure 1 , the method comprising:

[0055] S101: Acquire real-time navigation data during surgery. The navigation data includes the position coordinates of a first sensor, a bronchoscope image, and a respiratory curve. The first sensor and bronchoscope are located at the interventional end of an interventional instrument. Specifically, the interventional instrument is a slender, flexible catheter. The first sensor and bronchoscope are integrated into the catheter tip. The first sensor is used to acquire positioning data, and the bronchoscope is used to acquire images.

[0056] S102: Determine the current bronchial tree model and the current planned path point cloud data corresponding to the current respiratory phase according to the respiratory curve.

[0057] S103: Obtaining current virtual coordinates of the first sensor in the medical image coordinate system according to a first conversion relationship and position coordinates of the first sensor, where the first conversion relationship is a conversion relationship between the first sensor coordinate system and the medical image coordinate system.

[0058] S104: Determine whether the current intervention end is located on the planned path based on the bronchoscope image, the current bronchial tree model, the current planned path point cloud data, and the current virtual coordinates.

[0059] In this embodiment, the current bronchial tree model and the currently planned path point cloud data corresponding to the current respiratory phase are determined based on the respiratory curve. The current virtual coordinates are then mapped to the current bronchial tree model at the corresponding phase, eliminating the effects of respiratory motion on airway structure and improving registration accuracy. The bronchoscope image is used to determine the relative position of the cavities at the current and previous moments. The current virtual coordinates represent the current position of the interventional device. The current bronchial tree model is used to determine the current cavitary position of the interventional device. Combined with the currently planned path point cloud data, it is determined whether the bronchial branches corresponding to the cavities within the current lens range are on the planned path.

[0060] In one embodiment, the corresponding data collection and data processing preparations are made before the operation. Before the operation, medical images of the bronchus in the inspiratory phase and the expiratory phase are collected. Before performing the coarse registration operation, the skeleton point cloud of the trachea and the left and right main bronchi is extracted. The trachea and the left and right main bronchi are defined as the main airways. The two are equivalent in the following text. The voxel segmentation of the bronchial airway is performed on the medical image of the inspiratory phase to obtain a 3D model of the bronchial airway, such as Figure 2 As shown in the 3D model of the bronchial airway on the left, the 3D model of the bronchial airway is skeletonized to obtain the bronchial tree model and the planning path point cloud S1 of the inspiratory phase, as shown in Figure 2 As shown in the bronchial tree model on the right, the bronchial tree in the inspiratory phase determines the tree structure of the airway in the order from the airway entrance to the bifurcation end. Figure 2 The thick line part in the bronchial tree model is the branch skeleton of the main airway. Mark the target lesion location on the medical image of the inspiratory phase, find the target bronchial tree end (leaf node) closest to the target lesion location, and trace back from the target bronchial tree end along the bronchial tree structure to the main airway to obtain the navigation planning path. Specifically, the medical image can be a CT image or a magnetic resonance image (MRI) or other medical image that can feedback the bronchial anatomical structure, and the corresponding medical image coordinate system can be a CT image coordinate system, a magnetic resonance image coordinate system, etc. During the coarse registration data acquisition process, only the main airway is traversed, which can reduce patient discomfort.

[0061] In yet another embodiment, see Figure 8, respiratory cycle data preparation. The respiratory curve is obtained by setting a second sensor on the body surface or monitoring the blood oxygen concentration. The first sensor and the second sensor are electromagnetic (EM) sensors. For example, the external manifestation of breathing is the expansion and contraction of the chest. The second sensor is fixed at several positions on the patient's body surface, and the respiratory movement is monitored by using the change in relative position relationship. The corresponding respiratory curve shows periodic fluctuations similar to a sine curve, and the peaks and troughs of the curve correspond to inspiration and expiration respectively. Let the first phase (i.e., the inspiratory phase) be P1, the fourth phase (i.e., the expiratory phase) be P4, the respiratory phases in between be the second phase P2, and the third phase P3. An example of a complete respiratory cycle is P1->P2->P3->P4->P3->P2->P1. The inspiratory phase P1 corresponds to the inspiratory phase bronchial tree model S1.

[0062] In a preferred embodiment, see Figure 8 After skeletonizing the three-dimensional model of the bronchial airway to obtain the inspiratory phase bronchial tree model and the planned path point cloud, the method further includes: performing deformation registration on the inspiratory and expiratory phase medical images of the bronchus to obtain the expiratory phase deformation field D4. The expiratory phase deformation field D4 is applied to the inspiratory phase bronchial tree model S1 to perform a deformation operation to obtain the expiratory phase bronchial tree model S4 and the planned path point cloud. Based on the expiratory phase deformation field D4, a second phase deformation field D2 and a third phase deformation field D3 are obtained by linear interpolation. The second phase P2 and the third phase P3 are the respiratory phases between the inspiratory phase and the expiratory phase. The second phase deformation field D2 and the third phase deformation field D3 are applied to the inspiratory phase bronchial tree model S1 to perform a deformation operation to obtain the second phase bronchial tree model S2 and the planned path point cloud, and the third phase bronchial tree model S3 and the planned path point cloud. Therefore, the bronchial tree model corresponding to different respiratory phases can be used to predict the tracheal deviation caused by respiratory movement. Considering the morphological changes caused by lung respiratory movement during intraoperative navigation can improve navigation accuracy.

[0063] In a specific embodiment, see Figure 8, deformable registration is performed on the inspiratory and expiratory medical images of the bronchi to obtain the expiratory phase deformation field D4, including: using the inspiratory phase medical image CT_I as a fixed image and the expiratory phase medical image CT_O as a floating image, performing rigid transformation registration to obtain the rigid transformation registered image CT_Oa. Using the rigid transformation registered image CT_Oa as a fixed image and the inspiratory phase medical image CT_I as a floating image, performing affine transformation registration to obtain the affine transformation registered image CT_Ia. Using the rigid transformation registered image CT_Oa as a fixed image and the affine transformation registered image CT_Ia as a floating image, performing elastic transformation registration to obtain the elastic transformation registered image CT_Ib. Calculating the expiratory phase deformation field D4 from the inspiratory phase medical image CT_I to the elastic transformation registered image CT_Ib. After the registration transformation, the elastically transformed registered image CT_Ib has an anatomical structure similar to that of the expiratory phase medical image CT_O. By applying the expiratory phase deformation field D4 to the inspiratory phase bronchial tree model S1 corresponding to the inspiratory phase medical image CT_I, an expiratory phase bronchial tree model S4 corresponding to the expiratory phase medical image CT_O can be obtained. In this embodiment, rigid body transformation registration is used to eliminate the differences in the spatial origin and rotation angle of the two medical image acquisitions to ensure their alignment in the same spatial coordinate system. Affine transformation registration optimizes image alignment, making the images of the two respiratory phases more consistent. Elastic transformation registration can better handle nonlinear deformations between images, further improving the accuracy and stability of registration.

[0064] In a specific embodiment, see Figure 9 Before surgery, the main airway skeleton point cloud data is extracted from the bronchial tree model. The main airway is traversed by an interventional device, and the trajectory point cloud data of the first sensor is collected. Based on the trajectory point cloud data and the main airway skeleton point cloud data, the rigid transformation matrix of the first sensor coordinate system converted to the medical image coordinate system is calculated, and the initial global rigid transformation matrix is ​​obtained through iterative registration processing. Figure 3a As shown, point cloud 301 is the main airway skeleton point cloud in the medical image coordinate system, point cloud 302 is the trajectory point cloud collected by the first sensor, and point cloud 303 is the trajectory point cloud after rigid transformation, as shown in FIG. Figure 3b As shown, point cloud 304 is the trajectory point cloud after initial iterative registration. The purpose of iterative registration is to reduce the registration error so as to rotate and translate the collected trajectory point cloud to be close to the main airway skeleton point cloud.

[0065] For further information, see Figure 9, the initial global rigid transformation matrix is ​​obtained through iterative registration processing, including: using the iterative closest point (ICP) algorithm to perform iterative registration processing to obtain the initial global rigid transformation matrix. The main airway diameter is relatively thick, and there is an error between the coordinate information collected during the catheter traversal and the actual airway skeleton endpoint. The rigid transformation matrix obtained by the rigid registration method corresponding to only a few points still needs iterative fine-tuning. In this solution, based on all the main airway skeleton point clouds and the trajectory point clouds collected by the first sensor, the ICP algorithm is used for further iterative registration, combined with Figure 3a and Figure 3b , compared with the point cloud 303 before fine-tuning, the point cloud 304 after fine-tuning has a higher similarity matching degree with the point cloud 301, and the initial global rigid transformation matrix H0 is obtained.

[0066] For example, the rigid transformation matrix H is defined as Where R is the rotation matrix, T is the translation vector, and O = [0, 0, 0]. Calculate R and T to obtain the coordinate information of the three endpoints in the main airway skeleton point cloud. And the points collected by the first sensor corresponding to the three endpoints q1, q2, q3 represent the position coordinates of the three endpoints in the main airway skeleton point cloud, p1, p2, p3 represent the position coordinates of the three endpoints in the trajectory point cloud collected by the first sensor, and the rotation and translation effects are as follows: Figure 3a The arrow in the middle points to the endpoint of the trajectory point cloud of the first sensor, and the lower end of the arrow points to the endpoint corresponding to the trajectory point cloud after rigid transformation. The solution process is equivalent to Taking the derivative of this equation, we get Will Substituting into the above formula, we can get R = argmin ‖ Rx i +y i ‖ 2 ,in, Do SVD decomposition, S = UΣV T , where S = XY T , The solution is, Among them, p i is the position coordinate of the i-th endpoint in the trajectory point cloud, q i is the position coordinate of the i-th endpoint of the main airway skeleton point cloud, are the center position coordinates of the three endpoints in the trajectory point cloud, The center position coordinates of the three endpoints in the main airway skeleton point cloud, x i is the offset vector of the i-th endpoint in the trajectory point cloud relative to the center position of the trajectory point cloud, y iThe offset vector of the i-th endpoint of the main airway skeleton point cloud relative to the center position of the main airway skeleton point cloud, SVD is the singular value decomposition, U is an orthogonal matrix, Σ is a diagonal matrix, V T is the transpose of another orthogonal matrix.

[0067] Obtaining the current virtual coordinates of the first sensor in the medical image coordinate system according to the first transformation relationship and the position coordinates of the first sensor, including: obtaining the current virtual coordinates of the first sensor in the medical image coordinate system according to the position coordinates of the first sensor and an initial global rigid transformation matrix.

[0068] Based on the global rigid transformation matrix H0 obtained above, the position coordinates obtained by the first sensor can be converted to the medical imaging coordinate system to obtain the current virtual coordinates. The respiratory phase is determined by combining the respiratory movement feedback from the second sensor on the patient's body surface, and the current virtual coordinates are mapped to the current bronchial tree model of the corresponding phase to eliminate the influence of respiratory movement. The initial global rigid transformation matrix is ​​obtained by alignment based only on the main airway data. Further, during the real-time path navigation process, it is adjusted in combination with the real-time collected intraoperative historical trajectory point cloud data. This solution performs real-time navigation based on the planned path of the lesion. The planned path is as follows: Figure 4 As shown in the figure, the dot represents the lesion location, and the bold line represents the planned path from the target to the lesion. During real-time navigation, the operator inserts the interventional instrument from the patient's mouth along the bronchus to the lesion. The historical trajectory point cloud of the route is converted to the medical imaging coordinate system to match it with the planned path.

[0069] During real-time navigation, navigation data is collected at a fixed frequency. The navigation data at time t includes the data E0 of the first sensor, the data of n second sensors {E i}(i=1,2,…,n) and bronchoscope images. The sensor data is composed of E=[x,y,z,s,u,v,w], which contains position coordinate information [x,y,z] and position rotation information [s,u,v,w]. x, y, z represent the coordinate information of the x-axis, y-axis, and z-axis respectively; s represents the real part of the quaternion, which is related to the rotation angle; u, v, w represent the imaginary part of the quaternion, which is related to the rotation axis. Based on the position coordinate information of the n second sensors, fit it to the periodic respiratory curve to determine the respiratory phase P. t ∈{P1,P2,P3,P4}, select the corresponding bronchial tree model. The position coordinates of the first sensor [x0,y0,z0] are transformed into the medical imaging coordinate system by the global rigid transformation matrix H0 to obtain [x,y,z,1] T =H0[x0,y0,z0,1] T Find the nearest neighbor point of the current virtual coordinate in the current planned path point cloud data and calculate the Euclidean distance.

[0070] In one embodiment, a determination is made as to whether the current interventional end is on the planned path based on the bronchoscope image, the current bronchial tree model, the current planned path point cloud data, and the current virtual coordinates. This includes segmenting the cavity region in the bronchoscope image at the current moment and the previous moment, performing connected domain analysis and relative position matching on the segmentation results, and obtaining the relative position relationship of the cavity at the current moment and the previous moment. Based on the first transformation relationship and the position rotation information of the first sensor, a second transformation relationship is obtained between the bronchoscope coordinate system and the medical image coordinate system, and the navigation data also includes the position rotation information of the first sensor. Based on the second transformation relationship, the current bronchial airway three-dimensional model and the current bronchial tree model at the plane where the bronchoscope is located are intercepted to obtain a virtual tracheal image at the current moment. The current bronchial airway three-dimensional model is the bronchial airway three-dimensional model corresponding to the current respiratory phase. The current virtual tracheal image at the current moment includes the cavity region at the plane where the bronchoscope is located and the bronchial tree branches located within the cavity region. Based on the current virtual tracheal image, the relative position relationship of the cavity, and the current planned path point cloud data, a determination is made as to whether the current interventional end is on the planned path.

[0071] Based on the two frames of bronchoscopic images at the previous moment and the current moment, combined with the coordinate information, it is determined whether the current intervention end is still on the planned path. Figure 5a and Figure 5b As shown in the figure, two frames of bronchial images show that as the interventional device advances, one of the cavities representing different branches at the tracheal bifurcation disappears in the bronchial lens. The purpose of path recognition is to infer whether the bronchial tree branch corresponding to the cavity remaining in the lens range is on the planned path based on the position relationship. The specific steps are:

[0072] a) If Figure 5a and Figure 5b As shown in the figure, the pore area in the bronchoscopic image is segmented. The segmentation algorithm can use other traditional segmentation methods such as threshold segmentation, simple linear iterative clustering (SLIC), or segmentation methods based on deep learning. Connected domain analysis and relative position matching are performed on the segmentation results to determine the relative position of the pore in the current image (as shown in the figure, the pore at the current moment is located to the left of the image at the previous moment).

[0073] b) Based on the rigid transformation matrix H0 and the position rotation information [s,u,v,w] of the first sensor, the conversion relationship between the bronchoscope coordinate system and the medical image coordinate system is derived, and the current bronchial airway three-dimensional model and the current bronchial tree model are intercepted to obtain a virtual tracheal image corresponding to the bronchoscope image. Figure 5d As shown, the arrow direction is the direction of interventional equipment (also the shooting direction of bronchoscope), and the frame in front of the direction of travel is the plane for intercepting the virtual tracheal image. The virtual tracheal image at the current moment is as follows: Figure 5cAs shown in the figure, the white area is the airway segmentation pixel, and the pixel points in the two white areas are the intersection points of the corresponding airway bronchial tree branches and the plane. According to the position correspondence, it can be judged whether the current intervention end is still on the planned path. If the path is judged correctly, the position coordinates [x0, y0, z0] of the first sensor are stored in the position corresponding to the current respiratory phase P t The corresponding historical trajectory point cloud L t If the path judgment is incorrect, a path error prompt message will be sent to prompt the operator to return to the previous fork node.

[0074] In a preferred embodiment, see Figure 10 After step S104, the method further includes step S105: finding the nearest neighbor point of the current virtual coordinate in the current planned path point cloud data; if the current virtual coordinate does not meet the accuracy requirement, then when a first correction condition is met, iterative registration processing is performed on the current global rigid transformation matrix based on the historical trajectory point cloud data of the first sensor during the operation and the portion from the main trachea to the current branch in the current planned path point cloud data to obtain an updated global rigid transformation matrix; the first correction condition is that the current navigation depth is within a first depth threshold range or the inner diameter of the current branch is within a first inner diameter threshold range, that is, the depth of the current branch in the bronchial tree is shallow or the inner diameter of the branch is large. When the second correction situation is met, the path point cloud planned on the current branch is elastically registered and transformed based on the historical trajectory point cloud of the first sensor on the current branch to obtain the registered bronchial tree model and the registered branch path point cloud. The second correction situation is that the current navigation depth is in the second depth threshold range or the inner diameter of the current branch is in the second inner diameter threshold range, that is, the depth of the current branch in the bronchial tree is deeper or the inner diameter of the branch is smaller, the minimum value of the second depth threshold range is greater than the maximum value of the first depth threshold range, and the maximum value of the second inner diameter threshold range is less than the minimum value of the first inner diameter threshold range. Specifically, the current virtual coordinates do not meet the accuracy requirements, including: the Euclidean distance between the nearest neighbor point and the current virtual coordinates is greater than the set distance threshold. In addition, for the second correction situation, the current global rigid transformation matrix is ​​iteratively registered to obtain an updated global rigid transformation matrix. By updating the global rigid transformation matrix in real time, the calculated current virtual coordinates are made more accurate.

[0075] In a specific embodiment, based on the historical trajectory point cloud of the first sensor on the current branch, an elastic registration transformation is performed on the planned path point cloud on the current branch to obtain a registered bronchial tree model and a registered branch planned path point cloud, including: performing line segment fitting on the planned path point cloud on the current branch and the historical trajectory point cloud of the current branch, respectively, to obtain a first fitted line segment and a second fitted line segment. Based on the length deviation of the first fitted line segment and the second fitted line segment, the historical trajectory point cloud of the current branch is stretched to obtain a stretched historical trajectory point cloud, and line segment fitting is performed on the stretched historical trajectory point cloud to obtain a third fitted line segment. Based on the directional deviation of the first fitted line segment and the third fitted line segment, a rotation transformation is performed on the planned path point cloud on the current branch to obtain a registered bronchial tree model and a registered branch planned path point cloud.

[0076] In another embodiment, before performing line segment fitting on the planned path point cloud on the current branch and the historical trajectory point cloud on the current branch to obtain the first fitted line segment and the second fitted line segment, the method further includes: finding the nearest neighbor point of the current virtual coordinate in the current planned path point cloud, and extracting the planned path point cloud on the current branch from the current planned path point cloud based on the nearest neighbor point and the starting point of the planned path point cloud on the current branch. Extract the portion of the historical trajectory point cloud that matches the planned path point cloud on the current branch to obtain the historical trajectory point cloud of the current branch.

[0077] Path recognition is used to assess whether the current virtual coordinates meet the accuracy requirements along the planned path. If not, dynamic registration correction is performed. Because bronchial diameters vary significantly across different branches of the bronchial tree, and the catheter exerts varying stresses on the bronchial wall, this solution utilizes global rigid registration and local elastic registration to achieve dynamic registration correction during real-time navigation. Dynamic registration compensates for errors caused by offset deformation and improves navigation accuracy.

[0078] Dynamic correction using global rigid registration. Based on the calculated nearest neighbor point, if the branch is shallow in the bronchial tree or has a large inner diameter, it is defined as a tracheal branch. The effect of catheter stress on the bronchial wall can be considered negligible, and global rigid registration is used for correction. For detailed methods, see Figure 6a and Figure 6b , set the current respiratory phase P t The corresponding historical trajectory point cloud L t It is a floating point cloud, and the bronchial point cloud from the main trachea to the current branch on the planned path point cloud is a fixed point cloud. The ICP algorithm is used to iteratively update the global rigid transformation matrix H0.

[0079] See also Figure 7, local elastic registration dynamic correction. If the current branch is deeper in the bronchial tree or the inner diameter of the branch is smaller, the branch is defined as a small bronchial branch, and the catheter stress has a greater impact on the bronchial wall. By modeling the tracheal displacement and deformation caused by this stress into a rotation transformation with the branch starting point as the rotation center and a stretching transformation with a length difference. The position coordinates of the first sensor are converted to the current virtual coordinates in the medical imaging coordinate system through the global rigid transformation matrix H0, such as Figure 7 For point e1, find the nearest neighbor point e2 on the currently planned path point cloud. Extract the path point cloud p2e2 of the branch where the nearest neighbor e2 lies (the starting point of p2e2 is the branch starting point p2, and the end point is e2). Extract the portion p1e1 of the historical trajectory point cloud that matches p2e2. Fit line segments to point clouds p1e1 and p2e2 to obtain two fitted line segments. Based on the length deviation of the two fitted line segments, stretch point cloud p1e1 and perform a rotation transformation based on the directional deviation of the two fitted line segments. Set the starting point p2 of point cloud p2e2 as the rotation point, and perform an elastic registration transformation on point cloud p2e2 to improve the registration accuracy of the current branch of the bronchial tree and the historical trajectory point cloud of the current branch after the transformation.

[0080] The technical effects of the intrabronchial navigation method of the present invention are explained in detail below.

[0081] 1. To address changes in bronchial tree structure caused by respiratory motion, preoperative CT images of the patient's inspiratory and expiratory phases are collected. A bronchial tree model for the inspiratory phase is obtained based on the inspiratory phase images. Medical image registration is used to obtain the deformation field from the inspiratory phase to the expiratory phase. The deformation field from the inspiratory phase to other phases in the respiratory cycle is then interpolated. Based on this deformation field, bronchial tree models corresponding to all phases in the respiratory cycle are obtained, thereby suppressing the impact of respiratory motion during navigation.

[0082] 2. Use only the data collected during main airway navigation to perform rigid registration with the main airway point cloud of the bronchial tree, obtaining the transformation relationship from the first sensor coordinate system to the medical image coordinate system. This method is simple to use, can quickly obtain the initial global rigid transformation matrix, and does not cause significant discomfort to the patient.

[0083] 3. During real-time navigation, bronchoscope images are used to assist in determining whether the navigation position is on the planned path, thereby increasing the accuracy of path navigation. When the navigation accuracy is insufficient, different registration correction schemes are used for dynamic correction. When the navigation depth is shallow or the inner diameter of the tracheal branch is large, the stress of the interventional device has little effect on the bronchial wall. At this time, the bronchial tree is still regarded as a rigid body, and the possibility of elastic deformation is ignored. At this time, the global rigid matrix is ​​corrected based on the historical trajectory point cloud; when the navigation depth is deep or the inner diameter of the tracheal branch is small, the stress of the interventional device cannot be ignored. At this time, the elastic deformation of the current branch is modeled as a rotation and stretching transformation with the branch vertex as the rotation point, so that the current branch of the bronchial tree after the transformation is more matched with the historical trajectory point cloud of the current branch.

[0084] In addition, the present invention also proposes a bronchial navigation device, see Figure 11 The device includes: an acquisition unit 1101, which is used to acquire navigation data collected in real time during the operation, and the navigation data includes the position coordinates of the first sensor, the bronchoscope image and the respiratory curve. The first sensor and the bronchoscope are arranged at the intervention end of the interventional instrument; a determination unit 1102, which is used to determine the current bronchial tree model and the current planned path point cloud data corresponding to the current respiratory phase according to the respiratory curve; a conversion unit 1103, which is used to obtain the current virtual coordinates of the first sensor in the medical image coordinate system according to the first conversion relationship and the position coordinates of the first sensor. The first conversion relationship is the conversion relationship between the first sensor coordinate system and the medical image coordinate system; a judgment unit 1104, which is used to determine whether the current intervention end is located on the planned path according to the bronchoscope image, the current bronchial tree model, the current planned path point cloud data, and the current virtual coordinates. All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0085] Preferably, the device also includes: a correction unit, which is used to find the nearest neighbor point of the current virtual coordinate in the current planned path point cloud data. If the current virtual coordinate does not meet the accuracy requirement, then when the first correction situation is met, the current global rigid transformation matrix is ​​iteratively aligned according to the historical trajectory point cloud data of the first sensor during the operation and the main trachea to the current branch part in the current planned path point cloud data to obtain an updated global rigid transformation matrix. The first correction situation is that the current navigation depth is within the first depth threshold range or the inner diameter of the current branch is within the first inner diameter threshold range; when the second correction situation is met, the planned path point cloud on the current branch is elastically aligned and transformed according to the historical trajectory point cloud of the first sensor on the current branch to obtain a registered bronchial tree model and a registered branch planned path point cloud. The second correction situation is that the current navigation depth is within the second depth threshold range or the inner diameter of the current branch is within the second inner diameter threshold range, the minimum value of the second depth threshold range is greater than the maximum value of the first depth threshold range, and the maximum value of the second inner diameter threshold range is less than the minimum value of the first inner diameter threshold range.

[0086] In other embodiments of the present invention, an electronic device is disclosed. Figure 12 The electronic device may include: one or more processors 1201; a memory 1202; a display 1203; one or more applications (not shown); and one or more computer programs 1204. The above components may be connected via one or more communication buses 1205. The one or more computer programs 1204 are stored in the memory 1202 and configured to be executed by the one or more processors 1201. The one or more computer programs 1204 include instructions, which may be used to execute the following instructions: Figure 1 and Figure 11 and each step in the corresponding embodiment.

[0087] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0088] Each functional unit in each embodiment of the present invention may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0089] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, etc., various media that can store program code.

[0090] Although the embodiments of the present invention have been described in detail above, it is obvious to those skilled in the art that various modifications and changes can be made to these embodiments. However, it should be understood that such modifications and changes are within the scope and spirit of the invention as described in the claims. Moreover, the invention described herein may have other embodiments and may be implemented or realized in a variety of ways. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with ordinary skills in the field to which the invention belongs. The words "including" and the like used herein mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

Claims

1. A bronchial intraoperative navigation device, which can execute a bronchial intraoperative navigation method, characterized in that: The method comprises: S101: Acquire navigation data collected in real time during surgery, the navigation data including position coordinates of a first sensor, a bronchoscope image, and a respiratory curve, the first sensor and the bronchoscope being disposed at an interventional end of an interventional instrument; S102: Determine the current bronchial tree model and the current planned path point cloud data corresponding to the current respiratory phase according to the respiratory curve; S103: Obtaining current virtual coordinates of the first sensor in a medical image coordinate system according to a first conversion relationship and position coordinates of the first sensor, where the first conversion relationship is a conversion relationship between the first sensor coordinate system and the medical image coordinate system; S104: determining whether the interventional end is currently located on the planned path based on the bronchoscope image, the current bronchial tree model, the current planned path point cloud data, and the current virtual coordinates; S105: Find the nearest neighbor point of the current virtual coordinate in the current planned path point cloud data. If the current virtual coordinate does not meet the accuracy requirements, When a first correction condition is met, an updated global rigid transformation matrix is ​​obtained by iteratively registering the current global rigid transformation matrix based on the historical trajectory point cloud data of the first sensor during the operation and the portion from the main trachea to the current branch in the currently planned path point cloud data. The first correction condition is that the current navigation depth is within a first depth threshold range or the inner diameter of the current branch is within a first inner diameter threshold range. When the second correction condition is met, the planned path point cloud on the current branch is elastically registered and transformed according to the historical trajectory point cloud of the first sensor on the current branch to obtain the registered bronchial tree model and the registered branch planning path point cloud. The second correction condition is that the current navigation depth is in the second depth threshold range or the inner diameter of the current branch is in the second inner diameter threshold range, the minimum value of the second depth threshold range is greater than the maximum value of the first depth threshold range, and the maximum value of the second inner diameter threshold range is less than the minimum value of the first inner diameter threshold range.

2. The device according to claim 1, characterized in that Before surgery, main airway skeleton point cloud data is extracted from the bronchial tree model, the interventional instrument is used to traverse the main airway, and trajectory point cloud data of the first sensor is collected. Based on the trajectory point cloud data and the main airway skeleton point cloud data, a rigid transformation matrix is ​​calculated to transform the first sensor coordinate system into the medical image coordinate system, and an initial global rigid transformation matrix is ​​obtained through iterative registration processing; Obtaining the current virtual coordinates of the first sensor in the medical image coordinate system according to the first conversion relationship and the position coordinates of the first sensor includes: According to the position coordinates of the first sensor and the initial global rigid transformation matrix, the current virtual coordinates of the first sensor in the medical image coordinate system are obtained.

3. The device according to claim 1, characterized in that According to the historical trajectory point cloud of the first sensor on the current branch, an elastic registration transformation is performed on the planned path point cloud on the current branch to obtain a registered bronchial tree model and a registered branch planned path point cloud, including: Performing line segment fitting on the planned path point cloud on the current branch and the historical trajectory point cloud of the current branch respectively to obtain a first fitting line segment and a second fitting line segment; performing a stretching transformation on the historical trajectory point cloud of the current branch to obtain a stretched historical trajectory point cloud according to a length deviation between the first fitting line segment and the second fitting line segment, and performing line segment fitting on the stretched historical trajectory point cloud to obtain a third fitting line segment; According to the direction deviation between the first fitting line segment and the third fitting line segment, a rotation transformation is performed on the planned path point cloud on the current branch to obtain a registered bronchial tree model and a registered branch planned path point cloud.

4. The device according to claim 1, characterized in that Determining whether the interventional end is currently located on the planned path according to the bronchoscope image, the current bronchial tree model, the current planned path point cloud data, and the current virtual coordinates includes: Segment the cavity area in the bronchoscopic image at the current moment and the previous moment, perform connected domain analysis and relative position matching on the segmentation results, and obtain the relative position relationship of the cavity at the current moment and the previous moment; Obtaining a second transformation relationship between a bronchoscope coordinate system and a medical image coordinate system according to the first transformation relationship and the posture rotation information of the first sensor, wherein the navigation data further includes the posture rotation information of the first sensor; intercepting the current bronchial airway three-dimensional model and the current bronchial tree model at the plane where the bronchoscope is located according to the second conversion relationship to obtain a virtual tracheal image at the current moment; According to the virtual trachea image at the current moment, the relative position relationship of the cavities, and the point cloud data of the current planned path, it is determined whether the intervention end is currently located on the planned path.

5. The device according to claim 1, characterized in that Before the operation, medical images of the bronchi in the inspiratory and expiratory phases are collected, and voxel segmentation of the bronchial airways is performed on the medical images in the inspiratory phase to obtain a three-dimensional model of the bronchial airways. The three-dimensional model of the bronchial airways is skeletonized to obtain a bronchial tree model and a planning path point cloud in the inspiratory phase.

6. The device according to claim 5, characterized in that After skeletonizing the three-dimensional model of the bronchial airway to obtain the bronchial tree model and the planned path point cloud of the inspiratory phase, the method further includes: Perform deformation registration on the medical images of the bronchial inspiratory and expiratory phases to obtain the expiratory phase deformation field; Applying the expiratory phase deformation field to the bronchial tree model of the inspiratory phase to obtain the bronchial tree model of the expiratory phase and the planning path point cloud; According to the expiratory phase deformation field, a second phase deformation field and a third phase deformation field are obtained by a linear interpolation method, where the second phase and the third phase are respiratory phases between the inhalation phase and the exhalation phase; The second phase deformation field and the third phase deformation field are respectively applied to the bronchial tree model in the inspiratory phase to obtain the bronchial tree model and the planning path point cloud in the second phase and the bronchial tree model and the planning path point cloud in the third phase.

7. The device according to any one of claims 1 to 6, characterized in that The device comprises: an acquisition unit, configured to acquire navigation data collected in real time during surgery, the navigation data including the position coordinates of a first sensor, a bronchoscope image, and a respiratory curve, the first sensor and the bronchoscope being disposed at an interventional end of the interventional instrument; a determination unit, configured to determine a current bronchial tree model and current planned path point cloud data corresponding to a current respiratory phase according to the respiratory curve; a conversion unit, configured to obtain the current virtual coordinates of the first sensor in the medical image coordinate system according to a first conversion relationship and the position coordinates of the first sensor, wherein the first conversion relationship is a conversion relationship between the first sensor coordinate system and the medical image coordinate system; A judgment unit is used to determine whether the current intervention end is located on the planned path based on the bronchoscope image, the current bronchial tree model, the current planned path point cloud data, and the current virtual coordinates.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so as to enable the electronic device to execute the method executed by the apparatus according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method performed by the apparatus according to any one of claims 1 to 6 is implemented.

Images