Bronchoscope navigation method, navigation device and storage medium

Abstract

The invention relates to a navigation method and device of a bronchoscope and a storage medium. The navigation method relates to the technical field of medical information processing. The mapping library is constructed based on the dynamic tomographic image, and the target bifurcation point identifier corresponding to the target bifurcation point is determined based on matching of the multi-modal data of the target bifurcation point and the mapping library in the process that the bronchoscope advances along the initial path. And then, determining a target AR mark corresponding to the target bifurcation point identifier based on the mapping library, and superposing the target AR mark in the visual field of the bronchoscope through AR display equipment, so as to intuitively present the position mark, the feature mark, the risk mark, the path mark and other multi-modal parameters of the bifurcation point. And finally, on the basis of a position deviation value between the mark position of the target AR mark monitored in real time and the positioning data, updating the target multi-modal feature corresponding to the target bifurcation point identifier in the mapping library in real time. In conclusion, the accuracy and safety of bronchoscope navigation are improved.

Description

Technical Field

[0001] This application relates to the field of medical information processing technology, specifically to a navigation method, navigation device, and storage medium for a bronchoscope. Background Technology

[0002] Bronchoscopy is one of the core methods for the diagnosis and treatment of lung tumors, infectious diseases, and diffuse lung diseases. It involves inserting a slender bronchoscope into the lung through the airway to achieve visual observation and precise manipulation of the branches of the bronchial tree at all levels.

[0003] The bronchial tree, a complex tree-like anatomical structure in the human body, exhibits significant hierarchical characteristics. It branches progressively from the main bronchus to lobar bronchus, segmental bronchus, subsegmental bronchus, and bronchioles, with the number of bifurcation points at each level increasing exponentially with each level. Within this complex structure, bifurcation points with similar geometric shapes but significantly different mechanical or positional characteristics are common. For example, the bifurcation angles of the anterior subsegmental bronchus in the right upper lobe and the lateral subsegmental bronchus in the middle lobe may both be acute angles ranging from 30° to 45°, and the diameter ratio of the main bronchus to its branch may be close to 2:1. Based solely on geometric characteristics, the two can easily be confused.

[0004] Traditional bronchoscopy navigation technology relies solely on the anatomical grading of the bronchial tree and coarse spatial coordinates to construct a navigation model in the bifurcation point identification and matching stage. It does not incorporate quantitative parameters of geometric features, resulting in a matching error rate of over 30% for bifurcation points of similar grades. This can cause the bronchoscope to enter the wrong branch, which not only prolongs the operation time but may also cause damage to the bronchial wall mucosa and bleeding due to repeated instrument adjustments. In addition, it may lead to the omission of target lesions or accidental damage to normal lung tissue during biopsy or ablation operations. Summary of the Invention

[0005] The purpose of this application is to provide a bronchoscope navigation method, navigation device, and storage medium to solve the problem of low accuracy in traditional bronchoscope navigation technology.

[0006] To achieve the above objectives, the first aspect of this application provides a navigation method for a bronchoscope, comprising: A mapping library is constructed based on dynamic tomographic images of the bronchial region of the patient. The mapping library includes the mapping relationship between the bifurcation points of the bronchial tree, multimodal features, and AR markers. The multimodal features include geometric features, mechanical features, and location features. The AR markers include location markers, feature markers, risk markers, and path markers. During the bronchoscopy's movement along the initial path, multimodal data of the target bifurcation point is acquired in real time, and the target multimodal features corresponding to the multimodal data are extracted. The target multimodal features are matched with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point. The multimodal data includes optical image data, ultrasonic echo data, and positioning data. Based on the mapping library, the target AR marker corresponding to the target bifurcation point is determined, and the target AR marker is superimposed on the field of view of the bronchoscope through an AR display device; The system monitors the positional deviation between the target AR marker's location and the positioning data in real time. If the positional deviation exceeds a set value, the system updates the target multimodal features corresponding to the target bifurcation point identifier in the mapping library in real time.

[0007] A second aspect of this application provides a navigation device for a bronchoscope, comprising: The construction module is used to build a mapping library based on dynamic tomographic images of the patient's bronchial region. The mapping library includes the mapping relationship between the bifurcation points of the bronchial tree, multimodal features, and AR markers. The multimodal features include geometric features, mechanical features, and location features. The AR markers include location markers, feature markers, risk markers, and path markers. The matching module is used to acquire multimodal data of the target bifurcation point in real time during the bronchoscopy's movement along the initial path and extract the target multimodal features corresponding to the multimodal data. The target multimodal features are matched with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point. The multimodal data includes optical image data, ultrasonic echo data and positioning data. The display module is used to determine the target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and to overlay the target AR marker onto the field of view of the bronchoscope through an AR display device; The correction module is used to monitor the position deviation between the target AR marker's position and the positioning data in real time, and in response to the position deviation being greater than a set position deviation, to update the target multimodal features corresponding to the target bifurcation point identifier in the mapping library in real time.

[0008] A third aspect of this application provides a computer-readable storage medium storing a program that can be loaded by a processor and executed by the above-described bronchoscope navigation method.

[0009] The beneficial effects of this application are: This application integrates geometric, mechanical, and positional features to create a multimodal, unique identifier for bifurcation points in the bronchial region. A mapping library built based on dynamic tomographic images can adapt to dynamic anatomical environments. During bronchoscopy along the initial path, the multimodal data of the target bifurcation point is matched with the mapping library to determine the target bifurcation point identifier, accurately distinguishing geometrically similar but mechanically different bifurcation points. Then, based on the mapping library, a target AR marker corresponding to the target bifurcation point identifier is determined and overlaid on the bronchoscope's field of view using an AR display device. This visually presents the bifurcation point's location markers, feature markers, risk markers, and path markers, reducing the physician's reliance on experience and improving operational efficiency. Finally, based on the real-time monitoring of the target AR marker's position and the positional deviation value between the marker and the positioning data, the target multimodal features corresponding to the target bifurcation point identifier in the mapping library are updated in real time. This ensures the navigation model remains synchronized with the actual anatomical state, improving the stability of navigation accuracy. In summary, this application improves the accuracy and safety of bronchoscopy navigation through multimodal feature fusion, dynamic model construction, AR labeling, and closed-loop update mechanism, making it particularly suitable for operating environments in complex subsegmental bronchial regions.

[0010] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of a bronchoscope navigation method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating a bronchoscope navigation method provided in an embodiment of this application. Figure 3 This is a schematic diagram of a bronchial tree model structure provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the left upper lobe of the bronchial tree provided in an embodiment of this application.

[0012] Figure 5 This is a schematic diagram of the structure of a bronchoscope navigation device provided in an embodiment of this application.

[0013] Explanation of reference numerals in the attached figures 110. Controller; 120. Bronchoscope; 130. AR display device. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified. In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use this application. In the following description, details are set forth for illustrative purposes. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid unnecessary detail that would obscure the description of this application. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0016] The bronchoscope navigation method in this embodiment is applied to, for example... Figure 1 The controller 110 shown. Figure 1 This is a schematic diagram illustrating an application scenario of a control method for a bronchoscope 120 provided in this application embodiment. The application scenario may include a controller 110, a bronchoscope 120, and an AR display device 130. The controller 110 communicates with both the bronchoscope 120 and the AR display device 130.

[0017] The front end of the bronchoscope 120 can integrate a multi-sensor module, while the rear end communicates with the controller 110 via a dedicated interface to provide data support for navigation. As an example, the multi-sensor module may include optical imaging sensors, miniature ultrasonic probes, and positioning sensors (such as electromagnetic positioning sensors).

[0018] Optical imaging sensors can include miniature light sources and image sensors. They can acquire optical image data. For example, a miniature light source can illuminate the internal bronchial cavity, ensuring a clear outline of the bronchial tree within the field of view. The image sensor can be a miniature camera with a resolution adapted to the operation of a bronchoscope, capturing optical images of the bronchial cavity in real time. Based on these optical images, the geometric features of target bifurcation points can be obtained.

[0019] Miniature ultrasound probes can be integrated into the endoscopic ultrasound (EBUS) module embedded in the working channel of a bronchoscope. The ultrasound signal transceiver circuitry can be integrated into the bronchoscope handpiece or controller to drive the miniature ultrasound probe to emit ultrasound waves and receive the reflected ultrasound echo data. Based on the ultrasound echo data, the mechanical characteristics of the target bifurcation point can be calculated.

[0020] The positioning sensor typically employs an electromagnetic positioning sensor to adapt to non-metallic environments and eliminate radiation risks. It is usually integrated into the bronchoscope near the imaging components. In another example, optical positioning can also be used. The positioning sensor acquires positioning data from the bronchoscope, which can then be combined with an external respiratory phase acquisition device to obtain the real-time positional characteristics of the target bifurcation point.

[0021] Augmented Reality (AR) display device 130 is an intelligent reality terminal that overlays virtual information onto the real environment in real time using optical perspective technology. It retains the user's intuitive perception of the real scene while also supplementing the visual environment with computer-generated virtual content, enabling interaction between the displayed world and digital information. For example, in the bronchoscopy scenario of this application embodiment, AR display device 130 may include a head-mounted AR display device 130. The head-mounted AR display device 130 can display the field of view of the bronchoscope 120, which refers to the area of ​​the bronchus and surrounding tissues captured in real time by the optical imaging sensor at the front end of the bronchoscope and transmitted to the AR display device. Then, real-time anatomical information, navigation paths, operation instructions, risk warnings, and deviation alerts can be overlaid onto the field of view of the bronchoscope 120 using AR markers.

[0022] In this application embodiment, the navigation method of the bronchoscope 120 includes a controller 110 for the navigation method. The controller 110 can run a computer-readable storage medium corresponding to the bronchoscope 120 to execute the steps of the navigation method of the bronchoscope 120.

[0023] Understandable Figure 1The various electronic devices in the application scenario of the navigation method of the bronchoscope 120 shown do not constitute a limitation on the embodiments of this application. That is, the number and types of devices included in the application scenario of the navigation method of the bronchoscope 120, or the number and types of devices included in each electronic device, do not affect the overall implementation of the technical solution in the embodiments of this application, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of this application.

[0024] In this application embodiment, the controller 110 can be a standalone device, or a network of devices or a cluster of devices. For example, the controller 110 described in this application embodiment includes, but is not limited to, a computer, a network host, a single network device, a set of multiple network devices, or a cloud device composed of multiple devices. Among them, the cloud device is composed of a large number of computers or network devices based on cloud computing.

[0025] Those skilled in the art will understand that Figure 1 The application scenarios shown are merely one application scenario corresponding to the technical solution of this application, and do not constitute a limitation on the application scenarios of the technical solution of this application. Other application scenarios may include more than one application scenario. Figure 1 The number of more or fewer electronic devices shown, or the network connections of electronic devices, for example Figure 1 Only one electronic device is shown in the diagram. It is understood that the navigation method of the bronchoscope 120 may also include one or more other electronic devices, which are not limited here.

[0026] It should be noted that, Figure 1 The application scenario of the navigation method of the bronchoscope 120 shown is merely an example. The application scenario of the navigation method of the bronchoscope 120 described in the embodiments of this application is to more clearly illustrate the technical solution of the embodiments of this application, and does not constitute a limitation on the technical solution provided in the embodiments of this application.

[0027] Based on the application scenarios of the above-mentioned navigation method for bronchoscope 120, an embodiment of the navigation method for bronchoscope 120 is proposed. A detailed description is provided below with reference to the accompanying drawings.

[0028] Figure 2 This is a flowchart illustrating a bronchoscope navigation method provided in an embodiment of this application. Figure 2 As shown, this navigation method can execute steps 201-204 through the controller 110 described above, which will be described in detail below.

[0029] Step 201: Construct a mapping library based on dynamic tomographic images of the patient's bronchial region. The mapping library includes the bifurcation point identifiers of the bronchial tree, multimodal features, and mapping relationships between AR markers. Multimodal features may include geometric features, mechanical features, and location features, while AR markers may include location markers, feature markers, risk markers, and path markers.

[0030] Dynamic tomographic imaging refers to sequential tomographic images that capture patient movement, providing a more accurate representation of the patient's actual physiological state compared to static tomographic images. For example, dynamic tomographic imaging can refer to computed tomography (CT) at different respiratory phases, i.e., 4D-CT, or magnetic resonance imaging (MRI). Respiratory phases can include end-inspiratory, mid-inspiratory, quiescent, mid-exhalation, and end-exhalation. In one example, a dynamic model of the bronchial tree can be constructed using 3D reconstruction techniques to capture changes in the bronchial tree's position caused by respiratory motion.

[0031] The bronchial tree contains numerous bifurcation points that are geometrically similar but differ significantly in their mechanical or positional characteristics. These bifurcation points are specific anatomical locations within the bronchial tree structure where different levels of bronchial branches branch and diverge. They are nodes in the branching process from the trunk to the apex of the bronchial tree and are also key positioning nodes for navigation. This application constructs a mapping library that overcomes the reliance on single features for bronchoscopic navigation. This mapping library includes bifurcation point identifiers, multimodal features, and mapping relationships between AR markers.

[0032] The bifurcation point identifier is a unique numerical code for each bifurcation point, which can be generated by combining anatomical grading, three-dimensional coordinates, and feature types. Multimodal features can include geometric, mechanical, and positional features of the bifurcation point. Geometric features reflect the morphology of the bifurcation point and can include the bifurcation angle, the ratio of the main branch to the branch diameter, and the number of bifurcations. Mechanical features reflect tissue stiffness and can include the elastic modulus of the tube wall and stiffness coefficient calculated based on images. Positional features reflect the location information of the bifurcation point, such as its three-dimensional coordinates. By using a set of parameters describing the attributes of the bifurcation point from different dimensions, the limitations of a single feature of the bifurcation point can be overcome.

[0033] AR markers are virtual visualization elements associated with bifurcation points, and can include location markers, feature markers, risk markers, and path markers. Location markers are 3D frames indicating the spatial location of the bifurcation point. Feature markers are text labels annotating parameters such as geometric and mechanical features. Risk markers are color-coded indicators based on vascular distance, for example, red represents high risk. Path markers are arrowed lines that guide the bronchoscopy.

[0034] In one example, all bifurcation points of the bronchial tree can be identified using an image segmentation algorithm. Then, multimodal features of each bifurcation point are extracted and assigned a unique identifier. Finally, corresponding AR (Archived Resonance) markers are generated based on the feature parameters. This establishes a mapping relationship between bifurcation point identifiers, multimodal features, and AR markers. Storing this mapping relationship allows for the construction of a mapping library, serving as the data foundation for subsequent navigation methods. Accurate differentiation of similar bifurcation points can be achieved through multidimensional feature combinations. Pre-associating AR markers provides a data foundation for real-time display during bronchoscopy, reducing computational latency.

[0035] Step 202: During the bronchoscopy's movement along the initial path, acquire multimodal data of the target bifurcation point in real time and extract the target multimodal features corresponding to the multimodal data. Match the target multimodal features with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point.

[0036] The initial path is a pre-planned route before the bronchoscopy procedure. The target bifurcation point is the point where the bronchoscope passes along the initial path. For the target bifurcation point, multimodal data can be acquired in real time, including optical imaging data, ultrasonic echo data, and positioning data. Optical imaging data is real-time footage acquired by an optical imaging sensor. Ultrasonic echo data is the echo signal of the bronchial wall and surrounding tissues acquired through a miniature ultrasonic probe. Positioning data is the three-dimensional coordinates of the bronchoscope tip output by a positioning sensor.

[0037] During the bronchoscopy's movement along the initial path, multimodal data of the target bifurcation point can be acquired in real time, and then the corresponding target multimodal features can be extracted. These target multimodal features can include target geometric features, target mechanical features, and target positional features. The concepts of target geometric features, target mechanical features, and target positional features can be referenced in the description of multimodal features in the mapping library described above. Matching the target multimodal features with the mapping library determines the target bifurcation point identifier. For example, a weighted Euclidean distance algorithm can be used to compare the real-time extracted target multimodal features with pre-stored features in the mapping library; when the matching degree exceeds a set matching degree, it can be identified as the corresponding target bifurcation point. As an example, for bifurcation points with similar geometric features, the weights of mechanical and positional features can be increased to improve the distinguishability between different bifurcation points.

[0038] Multimodal data fusion can solve the problem of confusion caused by single features, especially in the subsegmental bronchial region, and can improve the recognition of bifurcation points.

[0039] Step 203: Determine the target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and overlay the target AR marker onto the field of view of the bronchoscope through an AR display device.

[0040] Based on the target bifurcation point identifier, AR markers corresponding to that identifier can be determined from the mapping library, namely location markers, feature markers, risk markers, and path markers. AR display devices, through optical perspective technology, can spatially register virtual AR markers with the real-time field of view of the bronchoscope, achieving an overlay display of real anatomical structures and virtual navigation information. In one example, as the bronchoscope moves, the AR markers can update their position and angle in real time with changes in the field of view, maintaining spatial consistency with the actual bifurcation point. In another example, when multiple AR markers overlap, the display hierarchy can be automatically adjusted; for example, risk markers can be prioritized and displayed at the highest level.

[0041] By visually presenting abstract navigation information, doctors can intuitively observe the three-dimensional structure of the bronchial tree. Through the coordinated use of multiple types of AR markers, comprehensive information can be provided to help doctors quickly assess the characteristics of target bifurcation points and operational risks.

[0042] Step 204: Monitor the position deviation between the target AR marker location data in real time. If the position deviation is greater than the set position deviation, update the target multimodal features corresponding to the target bifurcation point identifier in the mapping library in real time.

[0043] In this embodiment, bronchoscopy navigation uses AR markers to guide the physician to the target bifurcation point. Compared to bronchoscopy navigation that strictly follows a preset path, there is a problem of positional deviation between the AR markers and the actual bifurcation. Even if the path perfectly matches the plan, the AR markers may be misaligned with the actual bifurcation due to insufficient adaptation of the mapping library to respiratory movements, bronchial wall deformation, registration errors, etc., which can easily lead to operational errors. Focusing only on the deviation between the bronchoscopy and the planned path cannot solve problems such as inaccurate control of the bronchoscopy path and angle due to physician hand tremors during the operation. Therefore, this embodiment dynamically corrects the misalignment between the set data and the actual anatomical results by focusing on the positional deviation value between the AR markers and the actual bifurcation.

[0044] The positional deviation value refers to the three-dimensional distance between the theoretical position of the target AR marker calculated in real time and the bronchoscopic positioning data. The set positional deviation value is a threshold used to determine whether the mapping library needs to be updated. When the real-time monitored positional deviation value exceeds the set value, the target multimodal features corresponding to the target bifurcation point in the mapping library need to be updated, and the corresponding target AR marker needs to be updated synchronously. In one example, the feature parameters of the corresponding bifurcation point in the mapping library can be weighted and corrected according to the priority of positional features, geometric features, and mechanical features. Through the closed loop of deviation monitoring and dynamic updating of the mapping library, it is ensured that the AR marker is anchored to the true bifurcation point, reducing the fatal risk of the path being correct but the marker being incorrect.

[0045] This application's embodiments form a multimodal unique identifier for bifurcation points in the bronchial region by fusing geometric, mechanical, and positional features. A mapping library built based on dynamic tomographic images can adapt to dynamic anatomical environments. During the bronchoscopy's movement along the initial path, the multimodal data of the target bifurcation point is matched with the mapping library to determine the target bifurcation point identifier, accurately distinguishing geometrically similar but mechanically different bifurcation points. Then, based on the mapping library, a target AR marker corresponding to the target bifurcation point identifier is determined and overlaid on the bronchoscope's field of view using an AR display device. This visually presents the bifurcation point's location markers, feature markers, risk markers, and path markers, reducing the physician's reliance on experience and improving operational efficiency. Finally, based on the real-time monitoring of the target AR marker's position and the positional deviation value between the marker and the positioning data, the target multimodal features corresponding to the target bifurcation point identifier in the mapping library are updated in real time. This ensures the navigation model remains synchronized with the actual anatomical state, improving the stability of navigation accuracy. In summary, this application improves the accuracy and safety of bronchoscopy navigation through multimodal feature fusion, dynamic model construction, AR labeling, and closed-loop update mechanism, making it particularly suitable for operating environments in complex subsegmental bronchial regions.

[0046] In step 201, dynamic tomographic images of the bronchial region at multiple respiratory phases can be acquired to construct a three-dimensional dynamic model of the bronchial tree. Multiple respiratory phases cover the patient's complete respiratory cycle. For example, this can include five typical respiratory phases: end-inspiratory, mid-inspiratory, quiet breathing, mid-exhalation, and end-exhalation, reducing the limitation of a single phase failing to reflect anatomical dynamic changes. Dynamic tomographic images can clearly capture features such as the bronchial tree's positional shift and diameter changes during respiration. To improve accuracy, respiratory motion correction can be used to eliminate respiratory artifacts, and then image segmentation algorithms can be used to separate the bronchial tree structure from the dynamic tomographic images, removing interfering tissues such as blood vessels and lungs. Based on the segmented bronchial tree images from multiple phases, three-dimensional reconstruction can be performed separately to generate a corresponding number of three-dimensional models, each corresponding to a respiratory phase. Then, a three-dimensional dynamic model of the complete respiratory cycle is constructed using dynamic interpolation to simulate the continuous deformation process of the bronchial tree from end-inspiratory to end-exhalation.

[0047] After constructing a three-dimensional dynamic model of the bronchial tree, bifurcation points can be identified, and their geometric, mechanical, and positional features can be extracted. Bifurcation points can be identified using branch detection algorithms, such as those based on the bronchial tree skeleton, to determine the total number of bifurcation points in the bronchial tree. Then, the geometric, mechanical, and positional features of each bifurcation point can be extracted. Combining the parameters describing each bifurcation point from multiple dimensions can reduce the difficulty of distinguishing similar bifurcation points using a single feature, providing data for subsequent coding.

[0048] Secondly, each bifurcation point can be encoded based on its multimodal features to obtain a unique identifier for each bifurcation point, which is then used as the bifurcation point identifier. The unique identifier is generated from the multimodal features of each bifurcation point and can distinguish the unique numerical code of each bifurcation point in the tracheal tree. To ensure the uniqueness of each bifurcation point, in one example, this embodiment selects three non-repeatable feature dimensions for encoding. For example, the three feature dimensions may include anatomical grading, three-dimensional coordinates, and geometric feature type.

[0049] Specifically, the anatomical structure corresponding to the bifurcation point in the bronchial tree can be determined based on the location characteristics of the bifurcation point, and the anatomical grade of the bifurcation point can be determined based on the anatomical structure. The anatomical grade is then encoded to obtain the first sub-identifier.

[0050] Anatomical grading is based on the hierarchical relationship between the main trunk and branches of the bronchial tree, dividing the bronchial tree structure at bifurcation points into multiple levels to reflect the macroscopic location of the bifurcation point within the entire bronchial tree. This macroscopic location is determined based on anatomical structures, which may include the main bronchus, lobar bronchus, segmental bronchus, subsegmental bronchus, and bronchioles. The location characteristics of the bifurcation point allow for the identification of the corresponding anatomical structure. Each anatomical structure corresponds to an anatomical grade, which can be identified by numbers 1-5, making it concise and easy to recognize. The first sub-identifier is a numerical code for the anatomical grade and can be associated with abbreviations of its location. This allows for rapid location of the large anatomical area of ​​the bifurcation point, and establishing a macroscopic index of the first sub-identifier reduces the amount of searching required.

[0051] Figure 3 This is a schematic diagram of a bronchial tree model structure provided in an embodiment of this application. Figure 3 As shown, the trachea is divided into the left main bronchus and the right main bronchus, corresponding to the left and right lungs respectively. These two main bronchuses can be classified as level 1. The bifurcation point between the left and right main bronchuses is the carina. The right main bronchus bifurcates from the trachea at an angle of 20-30 degrees, while the left main bronchus bifurcates at 45-55 degrees. The lobar bronchus branching off from the main bronchus can be classified as level 2. The right lung includes three lobar bronchuses, and the left lung includes two lobar bronchuses. Further, each lobar bronchus can be subdivided into segmental bronchuses, which are classified as level 3, branching off from the lobar bronchus and corresponding to lung segments. For example, the upper lobe of the right lung can include the apical, anterior, and posterior segmental bronchuses. Subsegmental bronchuses (not shown in the figure) can be classified as level 4, branching off from the segmental bronchuses, located within lung segments, and with more dense branching. The bronchioles (not shown in the diagram) can correspond to level 5, branching off from the subsegmental bronchi. They are typically less than 3mm in diameter, located close to the alveoli, and have no cartilage in their walls. This allows for a rapid narrowing of the matching range for the bifurcation points and aligns with the doctor's understanding of the bronchial tree.

[0052] Secondly, the three-dimensional coordinates of the bifurcation points in the three-dimensional dynamic model are determined based on their location characteristics, and these coordinates are encoded to obtain a second sub-identifier. The three-dimensional coordinates are a spatial coordinate system established in the three-dimensional dynamic model of the bronchial tree. The real-time three-dimensional coordinates of the bifurcation point in this spatial coordinate system reflect its microscopic spatial location. To reduce coordinate fluctuations caused by extreme positions at the end of inspiration or expiration, the three-dimensional coordinates of the relatively stable quiet breathing phase of the three-dimensional dynamic model can be selected as the encoding basis. For example, this spatial coordinate system can be established with the carina as the origin, the X-axis representing the anterior-posterior direction of the three-dimensional dynamic model, the Y-axis representing the lateral direction, and the Z-axis representing the vertical direction. The second sub-identifier is a spatial index of a unique identifier formed after encoding the three-dimensional coordinates, which can solve the problem of distinguishing multiple bifurcation points under the same anatomical grade. For example, if there are three grade 4 subsegment bifurcation points in the right upper lobe, they need to be distinguished by coordinates. Assuming the three-dimensional coordinates of a bifurcation point are X=100.6mm, Y=80.2mm, and Z=60.5, the second sub-identifier can be represented as 100.6_80.2_60.5. In actual bronchoscopy, phase changes occur, which can be quickly corrected using coordinate offsets to maintain positioning accuracy. This solves the problem of distinguishing bifurcation points of the same anatomical grade and also provides positional data for subsequent spatial registration of AR markers.

[0053] Next, the feature type of the bifurcation point is determined based on its geometric characteristics, and the feature type is encoded to obtain a third sub-identifier. The geometric characteristics of the bifurcation point can include the bifurcation angle, the ratio of the main branch to the branch diameter, and the number of bifurcations. From these geometric characteristics, the most discriminative features can be selected as the basis for encoding the third sub-identifier. The third sub-identifier is an identifier obtained by encoding the feature type, which can solve the problem of distinguishing bifurcation points of the same grade and similar coordinates but with significant geometric differences. Since the diameter ratio and the number of bifurcations are the main factors affecting the difficulty of bronchoscope advancement and branch selection during bronchoscopy, the feature type can be determined by selecting the diameter ratio and the number of bifurcations. Feature types can be classified according to clinical risk and operational difficulty. For example, based on the diameter ratio, they can be divided into high-risk and low-risk types; based on the number of bifurcations, they can be divided into two-bifurcation and multi-bifurcation types. For example, the diameter ratio can be encoded as β + a specific numerical value, and the number of bifurcations can be marked as a numerical value + bifurcation. For example, for a two-bifurcation type with a diameter ratio of 2.5, it can be β2.5 - two-bifurcation. Associating feature types with operational risks can provide data support for subsequent navigation strategies and displays, reducing decision-making time during operations.

[0054] Finally, the first, second, and third sub-identifiers are combined to obtain a unique identifier for the bifurcation point. These three sub-identifiers can be combined according to macroscopic, microscopic, and attribute logic. The combination format can be set according to requirements; for example, they can be connected using "-" to obtain first sub-identifier-second sub-identifier-third sub-identifier. Generating a unique identifier through three levels of sub-identifiers solves the problem of difficulty in distinguishing similar bronchial tree bifurcation points, reducing data confusion. Furthermore, associating the encoding with macroscopic, microscopic, and attribute aspects allows for rapid determination of the bifurcation point's attributes without needing to consult additional data, improving the efficiency of bronchoscopy.

[0055] Figure 4 This is a schematic diagram of the structure of the left upper lobe of the bronchial tree provided in an embodiment of this application. Figure 4 Taking the bifurcation point A of the anterior segment of the left upper lobe as an example, the anterior segment of the left upper lobe belongs to the segmental bronchus, which branches off from the lobar bronchus of the left upper lobe, corresponding to an anatomical grade of 3. Therefore, the first sub-identifier can be B3-L. B represents the bronchus, 3 represents the anatomical grade, and L represents the left lung. Assuming the three-dimensional coordinates are 26, 55, 25, the second sub-identifier can be (26, 55, 25). Assuming the geometric features are a bifurcation angle θ = 45° and a diameter ratio β = 1.2, the third sub-identifier can be θ45β1.2. Finally, the first, second, and third sub-identifiers are combined and connected by "-", resulting in the unique identifier for the anterior segment of the left upper lobe: B3-L-(26, 55, 25)-θ45β1.2, which is used as the bifurcation point identifier. In this way, the location of the bifurcation point can be quickly obtained from the bifurcation point marker, the anatomical grade of the operation can be clearly defined, and the coordinates can be quickly determined, reducing the positioning time. At the same time, geometric features can also provide auxiliary prompts on the current operation difficulty, so that doctors can make timely decisions.

[0056] Additionally, location markers and risk markers can be generated based on location features associated with the bifurcation point identifier. Feature markers are generated based on geometric and mechanical features associated with the bifurcation point identifier. A navigation strategy is generated based on the geometric, mechanical, and location features associated with the bifurcation point identifier, and path markers are generated based on the navigation strategy. The path markers may include suggested operational parameters.

[0057] Specifically, location markers can be generated based on the three-dimensional coordinates of the bifurcation point during quiet breathing and displayed through a three-dimensional semi-transparent frame. Similarly, risk markers can be generated based on the location characteristics of the bifurcation point, such as the shortest distance to the pleura or blood vessels, to create a warning indication. For example, risk levels can be differentiated by color, with red indicating the highest risk, yellow indicating medium risk, and green indicating low risk. Feature markers can be generated based on the geometric and mechanical features of the bifurcation point, for example, by combining geometric and mechanical features, and displayed next to the location markers to identify the bifurcation point's attributes. A navigation strategy can be calculated based on the geometric, mechanical, and location features, and based on this strategy, a directional guide line, i.e., a path marker, can be generated. The path marker can also include suggested operating parameters to provide doctors with operational guidance and suggestions. For example, at the bifurcation point where the arrow points to the opening of the next branch, the suggested advancement speed and minimum turning radius can be marked. In this way, the data needed for the operation can be visually presented in the bronchoscopic field of view through AR markers, improving the efficiency of doctors' decision-making and the safety of the operation.

[0058] By identifying the bifurcation points and multimodal features of the bronchial tree through dynamic tomographic imaging, each bifurcation point is uniquely identified and associated with the parameters required for AR labeling, forming a mapping library. This provides a data foundation for subsequent AR guidance and correction.

[0059] In step 202, the outline of the bronchial tree in the optical image data can be extracted first by using an edge detection algorithm. The axis of the main branch and branches in the bronchial tree can be detected by Hough transform. The bifurcation angle of the target bifurcation point, the diameter ratio of the main branch and the branch where the target bifurcation point is located, and the number of bifurcations can be determined to obtain the geometric features of the target bifurcation point.

[0060] Edge detection algorithms are used to identify pixels with abrupt changes in grayscale values ​​in an image to determine the contour boundaries of the bronchial tree. For example, the Canny edge detection algorithm can be used, setting dual thresholds for pixels to determine the contour boundaries of the bronchial tree. A higher threshold is used to identify strong edges, and a lower threshold is used to preserve weak edges, such as the walls of the bronchioles. For example, assuming a high threshold of 150 and a low threshold of 50, pixels with a value higher than 150 are considered strong edges and are directly retained without further verification. Pixels with a value higher than 50 but less than or equal to 150 are considered candidate edges, requiring further verification of edge connectivity. If the edges are continuous, they are considered extensions of true edges; if isolated, they are considered false edges and require filtering. For example, candidate edges generated by localized reflections of secretions are false edges. Pixels with a value less than or equal to 50 indicate weak signals, such as background noise or blurred secretions, and are considered non-edges requiring thorough filtering. In this way, the loss of strong edges is reduced, ensuring the integrity of the bronchial tree trunk, while also filtering out false edges and improving the accuracy of bronchial tree edge extraction.

[0061] The Hough transform converts line detection in image space into peak detection in parameter space, effectively identifying intermittent lines affected by interference. Since the axes of the main branches and branches of the bronchial tree are the centerlines of continuous straight edges, they form significant voting peaks in parameter space. The main branch axis is selected based on the line with the highest voting peak and longest length, while the branch axes are selected based on lines that form a certain angle with the main branch axis and have a voting peak greater than a set number of votes. The set number of votes can be set according to the characteristics of the branch axes. This effectively identifies intermittent lines affected by noise, such as airway edges. Then, based on the identified axes of the main branches and branches, geometric features such as the bifurcation angle of the target bifurcation point, the diameter ratio of the main branch to the branch, and the number of bifurcations can be determined. The bifurcation angle is the angle between the main branch axis and the branch axis in the image plane; if there are multiple branches, the angle between each branch and the main branch is calculated separately. The diameter ratio is obtained by measuring the diameters of the main branch and branches in the direction perpendicular to the axes. The number of bifurcations is the number of branch axes that form an effective angle with the main branch axis and have a sufficient number of votes.

[0062] Geometric features are the fundamental dimension for bifurcation point identification. Precise bifurcation angles and diameter ratios can narrow the matching range of the mapping library and improve matching efficiency.

[0063] Since the propagation speed of ultrasound echoes in soft tissue is constant, the wall thickness at the target bifurcation point can be calculated based on ultrasound echo data. For example, the wall thickness can be obtained by dividing the result of the echo time difference and the speed of sound by 2.

[0064] During bronchoscopy, it's necessary to determine the wall stiffness, thus requiring the calculation of the elastic modulus. Furthermore, the bronchoscopic resistance to deformation during closure needs to be known, necessitating the calculation of the stiffness coefficient. However, the elastic modulus cannot be directly measured and must be indirectly calculated using ultrasound echo data. The stiffer the tissue, the higher the elastic modulus; greater density results in stronger reflection of ultrasound waves and a higher peak echo intensity. In one example, the peak intensity of ultrasound echo data can be converted into the elastic modulus. For instance, a correlation model can be established based on ultrasound echo intensity and elastic modulus calibration tests. Using simulated tissue tests of the elastic modulus, a regression formula for elastic modulus conversion can be obtained. The elastic modulus can then be calculated based on this regression formula.

[0065] The stiffness coefficient is a sum of material hardness and structural dimensions. Based on the pipe wall thickness and elastic modulus, the stiffness coefficient of the target bifurcation point can be calculated. The stiffness coefficient reflects the pipe wall's resistance to compressive deformation. For example, the stiffness coefficient can be calculated as the product of the elastic modulus and the cube of the pipe wall thickness, thus yielding the mechanical characteristics of the target bifurcation point. These mechanical characteristics can include the wall thickness, elastic modulus, and stiffness coefficient.

[0066] Optical imaging data can only observe the surface of the tube wall, while ultrasonic echo data can penetrate deep into the tube wall, thus solving the problem of distinguishing bifurcation points that are geometrically similar but have significant mechanical differences. Furthermore, bifurcation points with low elastic modulus and low stiffness coefficients are thin-walled, requiring deceleration during operation to reduce the risk of perforation. Conversely, high elastic modulus may indicate diseased tissue. Elastic modulus can also be used for risk warning, providing a data foundation for risk marking in AR (Anaerobic Angiography) labeling.

[0067] The lungs are organs that move dynamically with respiration, and the position of the bronchial tree shifts periodically with the respiratory phase. Directly using the raw location data can distort the determination of bifurcation points, affecting the accuracy of anatomical grading and risk assessment. Therefore, for location characteristics, respiration compensation can be applied to the location data based on the current respiratory phase to obtain compensated location data. For example, a respiratory motion data can be collected in real time using an impedance respiratory monitor on the patient's chest to determine the respiratory phase. Based on the phase-offset mapping relationship, the offset pattern of the target bifurcation point relative to the baseline anatomical point in the current respiratory phase is determined, thus obtaining compensated location data. The baseline anatomical point refers to an anatomical structure less affected by the respiratory phase and usually has a stable position. Examples include the tracheal carina and the bifurcation of pulmonary vessels. Then, based on the compensated location data, the anatomical grade of the bifurcation point can be determined, and the corresponding risk distance parameter can be calculated. The risk distance parameter is the shortest distance from the bifurcation point to a nearby important blood vessel, thus obtaining the location characteristics of the target bifurcation point. This ensures that the bifurcation point seen intraoperatively is the same structure as the preoperative bifurcation point, improving the accuracy of bifurcation point determination in the bronchial tree.

[0068] In one example, a respiratory deformation correlation model is used to obtain compensated positioning data. Therefore, in step 201, anatomical reference points associated with bifurcation point markers can be marked in the dynamic tomographic images, and the positional information of these reference points can be linked to the bifurcation point markers. The coefficient of variation of the anatomical reference points across multiple respiratory phases is less than a set coefficient. The set coefficient is a threshold for determining whether a bifurcation point of the bronchial tree is a stable anatomical structure during respiratory motion. A coefficient less than the set coefficient indicates that the bifurcation point is a stable anatomical structure with a small coefficient of variation, and can be used as an anatomical reference point. Then, the positional offset of the bifurcation point relative to the anatomical reference point is recorded across multiple respiratory phases, establishing a respiratory deformation correlation model. The respiratory deformation correlation model is a mathematical model that quantifies the positional offset between phases and bifurcation points, allowing for real-time calculation of offsets at any phase, achieving dynamic compensation. Specifically, based on the respiratory deformation correlation model, the coordinate offset of the positioning data relative to the anatomical reference point in the current respiratory phase can be found. Then, respiratory compensation is performed on the positioning data based on the coordinate offset to obtain compensated positioning data. The compensated positioning data can be the difference between the original coordinates and the coordinate offset.

[0069] Finally, the geometric, mechanical, and positional features of the target bifurcation point are cross-validated, and the target multimodal features of the target bifurcation point are determined based on the cross-validation results. Cross-validation is a logical consistency verification of features across different modalities; that is, geometric, mechanical, and positional features must conform to the anatomical patterns of the bronchial tree. For example, consistency between geometric and positional features is reflected in the fact that the higher the anatomical grade, the smaller the bifurcation angle and the larger the diameter ratio of the main branch to the branch. Consistency between mechanical and geometric features is reflected in the fact that the larger the diameter of the main branch, the greater the wall thickness and the higher the elastic modulus. Consistency between mechanical and positional features is reflected in the fact that bifurcation points closer to blood vessels, due to long-term blood flow pulsation, generally have a higher elastic modulus than bifurcation points farther from blood vessels. Based on the above consistency rules, different validation conditions can be set to perform consistency verification between features of different modalities. This can reduce the unreasonable data caused by noise and algorithm errors in a single modality and better conform to the anatomical patterns of the bronchial tree.

[0070] The overlay of raw data on geometric and mechanical features in AR markers can easily increase the information acquisition burden on physicians. Therefore, embodiments of this application can distinguish various AR markers using different colors or symbols, displaying them in the bronchoscope's field of view to reduce information interpretation time and improve decision-making efficiency.

[0071] Because small bronchi have a small diameter, they occupy a small portion of the bronchoscopic field of view and exhibit significant respiratory deviation. Therefore, during target tracking, it is easy to miss the target. Based on this, this application's embodiments address the problem of poor positioning accuracy for small moving targets by using dynamic and static navigation frames. For bronchi with larger diameters, such as main and lobar bronchi, they occupy a larger portion of the bronchoscopic field of view, exhibit less respiratory deviation, and have relatively stable positions. Therefore, static frames can meet the requirements for target positioning, reduce distractions for the physician, and decrease the system's computational load.

[0072] Specifically, in step 203, the geometric characteristics of the target bifurcation point can be determined first based on the target bifurcation point identifier. Bronchioles with smaller diameters or larger diameters are distinguished according to the diameter of the target bifurcation point and the respiratory offset. For example, a preset diameter can be set to distinguish diameters: those less than 3mm without cartilage support are considered small bronchioles, while those greater than or equal to 3mm have cartilage support and are relatively stable, thus considered non-small bronchioles. Regarding the respiratory offset, a first preset offset and a second preset offset can be set as thresholds for determining the magnitude of the respiratory offset. A value greater than the first preset offset indicates a large offset, while a value less than the second preset offset indicates a small respiratory offset. The first preset offset must be greater than or equal to the second preset offset.

[0073] When the diameter of the target bifurcation point is smaller than the set diameter, and the respiratory offset is greater than the first set offset, it can be identified as a small bronchus. Therefore, a dynamic navigation frame needs to be generated. The size of the dynamic navigation frame changes with the actual diameter, and a high-risk label is displayed. The high-risk label can be displayed around the dynamic frame to enhance the risk warning. When the diameter of the target bifurcation point is greater than or equal to the set diameter, and the respiratory offset is less than the second set offset, a static navigation frame is generated. The size of the static navigation frame is the set size. The set size is a fixed size to match the target detection scenario of non-small bronchus.

[0074] Then, feature markers are generated based on geometric and mechanical characteristics. Feature markers convert quantified feature parameters into symbols, colors, and text that the human eye can quickly recognize. The purpose is to compress information, reduce the doctor's interpretation time, and allow the corresponding attribute to be determined simply by color and symbol. For example, a red elastic modulus marker might indicate diseased tissue. This improves the guidance efficiency of AR displays.

[0075] Finally, risk markers and path markers can be generated based on location features. These markers have different visual characteristics. Both are generated based on location features; risk markers are used to indicate potential risks, while path markers are used to plan the forward path. Based on anatomical grading and compensatory positioning data, the direction of bronchoscope movement can be marked with arrows from the current location of the bronchoscope to the target bifurcation point, and risk markers and path markers can be distinguished by different colors and shapes. Risk markers can address the problem of blood vessels not being clearly visible in optical imaging data, while path markers can reduce the possibility of misoperation in situations with dense bronchial tree branches.

[0076] During bronchoscopy, the positional deviation between the target AR marker's location and the positioning data can be monitored in real time. In step 204, in response to a positional deviation value exceeding a set value, the current actual multimodal features are extracted, and the feature deviation value between the actual multimodal features and the target multimodal features associated with the target bifurcation point marker is calculated. The feature deviation value is the difference between the actual multimodal features and the target multimodal features in the mapping library, and can be calculated based on positional features, geometric features, and mechanical features. The larger the deviation value, the greater the discrepancy between the pre-stored data and the actual anatomy, thus requiring correction.

[0077] Then, based on the feature deviation values, the target multimodal features are corrected in the order of positional features, geometric features, and mechanical features, and the target AR markers are updated synchronously according to the corrected target multimodal features. The correction order can follow the principle of prioritizing basic features, because positional features are the reference for geometric and mechanical features; therefore, geometric features are the morphological basis of mechanical features. Reversing the order can easily lead to correction errors. Based on this, in one example, positional features can be corrected first, then geometric features, and finally mechanical features. This ensures that the corrected feature data conforms to the actual structure of the bronchial tree. If the feature deviation values ​​are not corrected in a timely manner, the accumulated operation time will lead to a large positioning error, affecting AR navigation and marking, thus making bronchoscopy operations dangerous.

[0078] In this embodiment, AR devices can also be used for tiered warning alerts. These tiered warning alerts differentiate based on the severity of the deviation. This application attempts to achieve multimodal tiered warnings through a combination of mechanical, visual, and auditory feedback to ensure timely correction of risks. For example, a first deviation value and a second deviation value can be set as tiered thresholds for determining the degree of deviation, with the first deviation value being greater than the second deviation value.

[0079] Specifically, if the positional deviation value is less than the first deviation value, it indicates an acceptable slight deviation, i.e., a mild risk, and the final trajectory of the path is fine-tuned. Fine-tuning does not affect the branch selection. If the positional deviation value is greater than or equal to the first deviation value but less than the second deviation value, it indicates a moderate risk, and a mechanical feedback prompt is generated and a deceleration prompt is displayed. For example, the bronchoscope handle vibrates, and a deceleration prompt is displayed on the AR display device. If the positional deviation value is greater than or equal to the second deviation value, it is identified as a severe risk, and both mechanical and audible feedback prompts are generated. For example, the bronchoscope handle vibrates, the AR device not only displays a danger warning but also provides audible prompts via a buzzer. Simultaneously, the bronchoscope's locking mechanism can be activated, for example, allowing only backward movement of the bronchoscope and preventing it from advancing, to reduce the risk of bronchoscope operation.

[0080] Through tiered early warning systems, differentiated alerts can be provided based on the severity of the risk, allowing for appropriate intervention. Furthermore, combining tactile, visual, and auditory multimodal sensory cues can address situations where single feedback is overlooked by doctors.

[0081] The embodiments of this application form a dual guarantee of active optimization and passive protection through feature correction and graded early warning, which can improve the safety and accuracy of bronchoscopy operations, and is especially suitable for operation scenarios of subsegment and smaller bronchi.

[0082] Figure 5 This is a schematic diagram of the structure of a bronchoscope navigation device 500 provided in an embodiment of this application. Figure 5 As shown, the bronchoscope navigation device 500 may include a construction module 501, a matching module 502, a display module 503, and a correction module 504.

[0083] The construction module 501 is used to construct a mapping library based on dynamic tomographic images of the patient's bronchial region. The mapping library may include the bifurcation point identifiers of the bronchial tree, multimodal features, and mapping relationships between AR markers. Multimodal features may include geometric features, mechanical features, and location features, and AR markers may include location markers, feature markers, risk markers, and path markers.

[0084] The matching module 502 is used to acquire multimodal data of the target bifurcation point in real time during the bronchoscope's movement along the initial path, extract the target multimodal features corresponding to the multimodal data, match the target multimodal features with a mapping library, and determine the target bifurcation point identifier corresponding to the target bifurcation point. The multimodal data may include optical image data, ultrasound echo data, and positioning data.

[0085] The display module 503 is used to determine the target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and to overlay the target AR marker onto the field of view of the bronchoscope through the AR display device.

[0086] The correction module 504 is used to monitor the position deviation between the target AR marker position and the positioning data in real time. In response to the position deviation value being greater than the set position deviation value, the module updates the target multimodal features corresponding to the target bifurcation point identifier in the mapping library in real time.

[0087] The construction module 501 may include an acquisition unit, an encoding unit, and a marking unit.

[0088] The acquisition unit is used to acquire dynamic tomographic images of the bronchial region of the patient at multiple respiratory phases, construct a three-dimensional dynamic model of the bronchial tree, identify the bifurcation points in the three-dimensional dynamic model, and extract the geometric, mechanical and positional features of the bifurcation points.

[0089] The encoding unit is used to encode the bifurcation point based on the multimodal features of each bifurcation point, and obtain a unique identifier for each bifurcation point. The unique identifier is used as the bifurcation point identifier of the bifurcation point.

[0090] The generation unit is used to generate location markers and risk markers based on location features associated with the bifurcation point identifier, generate feature markers based on geometric and mechanical features associated with the bifurcation point identifier, generate navigation strategies based on geometric, mechanical, and location features associated with the bifurcation point identifier, and generate path markers based on the navigation strategies. The path markers include suggested operating parameters.

[0091] The encoding unit is also used to determine the anatomical structure corresponding to the bifurcation point in the bronchial tree based on the location features of the bifurcation point, and to determine the anatomical grade of the bifurcation point based on the anatomical structure. The anatomical grade is encoded to obtain the first sub-identifier. The anatomical structure includes the main bronchus, lobar bronchus, segmental bronchus, subsegmental bronchus, and bronchioles. The three-dimensional coordinates of the bifurcation point in the three-dimensional dynamic model are determined based on the location features of the bifurcation point, and the three-dimensional coordinates are encoded to obtain the second sub-identifier. The feature type of the bifurcation point is determined based on the geometric features of the bifurcation point, and the feature type is encoded to obtain the third sub-identifier. The first sub-identifier, the second sub-identifier, and the third sub-identifier are combined to obtain the unique identifier of the bifurcation point.

[0092] The matching module 502 may include a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, and a cross-validation unit.

[0093] The first feature extraction unit is used to extract the outline of the bronchial tree in the optical image data through the edge detection algorithm, detect the axis of the main branch and branches in the bronchial tree through Hough transform, and determine the bifurcation angle of the target bifurcation point, the diameter ratio of the main branch and the branch where the target bifurcation point is located, and the number of bifurcations, so as to obtain the geometric features of the target bifurcation point.

[0094] The second feature extraction unit is used to calculate the pipe wall thickness at the target bifurcation point based on ultrasonic echo data, convert the peak intensity of the ultrasonic echo data into the elastic modulus, and calculate the stiffness coefficient of the target bifurcation point based on the pipe wall thickness and the elastic modulus to obtain the mechanical characteristics of the target bifurcation point.

[0095] The third feature extraction unit is used to perform respiratory compensation on the positioning data based on the current respiratory phase to obtain compensated positioning data. Based on the compensated positioning data, the anatomical grade of the bifurcation point is determined, and the risk distance parameter corresponding to the compensated positioning data is calculated to obtain the location features of the target bifurcation point.

[0096] The cross-validation unit is used to cross-validate the geometric, mechanical and positional features of the target bifurcation point, and to determine the target multimodal features of the target bifurcation point based on the cross-validation results.

[0097] The building module 501 also includes a benchmark determination unit and a modeling unit.

[0098] The reference point determination unit is used to mark anatomical reference points associated with bifurcation point identifiers in dynamic tomographic images and associate the location information of the anatomical reference points with the bifurcation point identifiers. The position variation coefficient of the anatomical reference points in multiple respiratory phases is less than a set coefficient.

[0099] The modeling unit is used to record the positional offset of the bifurcation point relative to the anatomical reference point under multiple respiratory phases, and to establish a respiratory deformation correlation model.

[0100] The third feature extraction unit is also used to find the coordinate offset of the positioning data relative to the anatomical reference point in the current respiratory phase based on the respiratory deformation correlation model; and to perform respiratory compensation on the positioning data based on the coordinate offset to obtain compensated positioning data.

[0101] The display module 503 includes a feature determination unit, a dynamic navigation marker unit, a static navigation marker unit, a feature marker unit, a risk marker, and a path marker unit.

[0102] The feature determination unit is used to determine the geometric features of the target bifurcation point based on the target bifurcation point identifier.

[0103] The dynamic navigation marker unit is used to generate a dynamic navigation box when the diameter of the target bifurcation point is smaller than the set diameter and the breathing offset is greater than the first set offset. The size of the dynamic navigation box changes with the actual diameter and displays a high-risk label.

[0104] The static navigation marker unit is used to generate a static navigation frame when the diameter of the target bifurcation point is greater than or equal to a set diameter and the breathing offset is less than a second set offset. The frame size of the static navigation frame is a set size.

[0105] The feature labeling unit is used to generate feature labels based on geometric and mechanical features; The risk marker and path marker units generate risk markers and path markers based on location features. The risk markers and path markers have different visual features.

[0106] The correction module 504 may include a monitoring unit and an update unit.

[0107] The detection unit is used to extract the current actual multimodal features and calculate the feature deviation value between the actual multimodal features and the target multimodal features associated with the target bifurcation point identifier in response to the position deviation value being greater than the set position deviation value.

[0108] The update unit is used to correct the target multimodal features based on the feature deviation value in the order of positional features, geometric features, and mechanical features, and to update the target AR tag synchronously according to the corrected target multimodal features.

[0109] The bronchoscope navigation device 500 of this application embodiment may further include a warning module. The warning module may include a first warning unit, a second warning unit, and a third warning unit.

[0110] The first prompting unit is used to fine-tune the end trajectory of the path if the position deviation value is less than the first deviation value.

[0111] The second prompting unit is used to generate a mechanical feedback prompt and display a deceleration prompt if the position deviation value is greater than or equal to the first deviation value but less than the second deviation value.

[0112] The third prompting unit is used to generate mechanical and audible feedback prompts if the positional deviation value is greater than or equal to the second deviation value, while simultaneously activating the bronchoscope's locking mechanism.

[0113] This application also provides a computer-readable storage medium storing a program that can be loaded by a processor and executed by any of the bronchoscope navigation methods in this application.

[0114] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0115] The above examples illustrate this application only to aid understanding and are not intended to limit its scope. Those skilled in the art to which this application pertains can make various simple deductions, modifications, or substitutions based on the ideas presented.

Claims

1. A navigation method for a bronchoscope, characterized in that, include: A mapping library is constructed based on dynamic tomographic images of the bronchial region of the patient. The mapping library includes the mapping relationship between the bifurcation points of the bronchial tree, multimodal features, and AR markers. The multimodal features include geometric features, mechanical features, and location features. The AR markers include location markers, feature markers, risk markers, and path markers. During the bronchoscopy's movement along the initial path, multimodal data of the target bifurcation point is acquired in real time, and the target multimodal features corresponding to the multimodal data are extracted. The target multimodal features are matched with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point. The multimodal data includes optical image data, ultrasonic echo data, and positioning data. Based on the mapping library, the target AR marker corresponding to the target bifurcation point is determined, and the target AR marker is superimposed on the field of view of the bronchoscope through an AR display device; The system monitors the positional deviation between the target AR marker's location and the positioning data in real time. If the positional deviation exceeds a set value, the system updates the target multimodal features corresponding to the target bifurcation point identifier in the mapping library in real time.

2. The bronchoscope navigation method according to claim 1, characterized in that, The mapping library constructed based on dynamic tomographic images of the patient's bronchial region includes: The patient's bronchial region dynamic tomographic images at multiple respiratory phases were acquired, a three-dimensional dynamic model of the bronchial tree was constructed, the bifurcation points in the three-dimensional dynamic model were identified, and the geometric, mechanical and positional features of the bifurcation points were extracted. The bifurcation points are encoded based on the multimodal features of each bifurcation point to obtain a unique identifier for each bifurcation point, and the unique identifier is used as the bifurcation point identifier of the bifurcation point. The location marker and the risk marker are generated based on the location features associated with the bifurcation point identifier. The feature marker is generated based on the geometric features and the mechanical features associated with the bifurcation point identifier. A navigation strategy is generated based on the geometric features, the mechanical features, and the location features associated with the bifurcation point identifier. A path marker is generated based on the navigation strategy. The path marker includes suggested operating parameters.

3. The bronchoscope navigation method according to claim 2, characterized in that, The process of encoding the bifurcation points based on the multimodal features of each bifurcation point to obtain a unique identifier for each bifurcation point includes: Based on the location features of the bifurcation point, the anatomical structure corresponding to the bifurcation point in the bronchial tree is determined, and the anatomical grade of the bifurcation point is determined based on the anatomical structure. The anatomical grade is encoded to obtain a first sub-identifier. The anatomical structure includes the main bronchus, lobar bronchus, segmental bronchus, subsegmental bronchus, and bronchioles. Based on the positional features of the bifurcation point, the three-dimensional coordinates of the bifurcation point in the three-dimensional dynamic model are determined, and the three-dimensional coordinates are encoded to obtain a second sub-identifier; Based on the geometric features of the bifurcation point, the feature type of the bifurcation point is determined, and the feature type is encoded to obtain a third sub-identifier; The first sub-identifier, the second sub-identifier, and the third sub-identifier are combined to obtain the unique identifier of the bifurcation point.

4. The bronchoscope navigation method according to claim 1, characterized in that, The process of acquiring multimodal data of the target bifurcation point in real time during the bronchoscope's movement along the initial path and extracting the target multimodal features corresponding to the multimodal data includes: The contour of the bronchial tree in the optical image data is extracted by an edge detection algorithm. The axis of the main branch and the branch in the bronchial tree is detected by Hough transform. The bifurcation angle of the target bifurcation point, the diameter ratio of the main branch and the branch where the target bifurcation point is located, and the number of bifurcations are determined to obtain the geometric features of the target bifurcation point. The wall thickness of the pipe at the target bifurcation point is calculated based on the ultrasonic echo data. The peak intensity of the ultrasonic echo data is converted into the elastic modulus. The stiffness coefficient of the target bifurcation point is calculated based on the wall thickness and the elastic modulus to obtain the mechanical characteristics of the target bifurcation point. Based on the current respiratory phase, the positioning data is compensated for to obtain compensated positioning data. The anatomical grade of the bifurcation point is determined according to the compensated positioning data, and the risk distance parameter corresponding to the compensated positioning data is calculated to obtain the location characteristics of the target bifurcation point. The geometric features, mechanical features, and positional features of the target bifurcation point are cross-validated, and the target multimodal features of the target bifurcation point are determined based on the cross-validation results.

5. The bronchoscope navigation method according to claim 4, characterized in that, The mapping library constructed based on dynamic tomographic images of the patient's bronchial region also includes: In the dynamic tomographic images, anatomical reference points associated with the bifurcation point identifier are marked, and the position information of the anatomical reference points is associated with the bifurcation point identifier. The position variation coefficient of the anatomical reference points in multiple respiratory phases is less than a set coefficient. Record the positional offset of the bifurcation point relative to the anatomical reference point under multiple respiratory phases to establish a respiratory deformation correlation model; The process of obtaining compensated positioning data by performing respiratory compensation on the positioning data based on the current respiratory phase includes: Based on the respiratory deformation correlation model, find the coordinate offset of the positioning data relative to the anatomical reference point in the current respiratory phase; Breathing compensation is performed on the positioning data based on the coordinate offset to obtain the compensated positioning data.

6. The bronchoscope navigation method according to claim 1, characterized in that, The step of determining the target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and overlaying the target AR marker onto the field of view of the bronchoscope through an AR display device, includes: The geometric features of the target bifurcation point are determined based on the target bifurcation point identifier; When the diameter of the target bifurcation point is smaller than the set diameter and the breathing offset is greater than the first set offset, a dynamic navigation box is generated. The size of the dynamic navigation box changes with the actual diameter and a high-risk label is displayed. When the diameter of the target bifurcation point is greater than or equal to the set diameter, and the breathing offset is less than the second set offset, a static navigation frame is generated, and the frame size of the static navigation frame is the set size; The feature markers are generated based on the geometric and mechanical features; The risk marker and the path marker are generated based on the location features, and the risk marker and the path marker have different visual features.

7. The bronchoscope navigation method according to claim 1, characterized in that, The step of responding to the position deviation value being greater than a set position deviation value by real-time updating the target multimodal features corresponding to the target bifurcation point identifier in the mapping library includes: In response to the position deviation value being greater than the set position deviation value, the current actual multimodal features are extracted, and the feature deviation value of the target multimodal features associated with the actual multimodal features and the target bifurcation point identifier is calculated; Based on the feature deviation value, the target multimodal features are corrected in the order of positional features, geometric features, and mechanical features, and the target AR tag is updated synchronously according to the corrected target multimodal features.

8. The bronchoscope navigation method according to claim 7, characterized in that, Also includes: If the position deviation value is less than the first deviation value, then the end trajectory of the fine-tuning path is adjusted. If the position deviation value is greater than or equal to the first deviation value but less than the second deviation value, a mechanical feedback prompt is generated and a deceleration prompt is displayed; If the positional deviation value is greater than or equal to the second deviation value, a mechanical feedback prompt and an audible feedback prompt are generated, and the locking mechanism of the bronchoscope is activated at the same time.

9. A navigation device for a bronchoscope, characterized in that, include: The construction module is used to build a mapping library based on dynamic tomographic images of the patient's bronchial region. The mapping library includes the mapping relationship between the bifurcation points of the bronchial tree, multimodal features, and AR markers. The multimodal features include geometric features, mechanical features, and location features. The AR markers include location markers, feature markers, risk markers, and path markers. The matching module is used to acquire multimodal data of the target bifurcation point in real time during the bronchoscopy's movement along the initial path and extract the target multimodal features corresponding to the multimodal data. The target multimodal features are matched with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point. The multimodal data includes optical image data, ultrasonic echo data and positioning data. The display module is used to determine the target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and to overlay the target AR marker onto the field of view of the bronchoscope through an AR display device; The correction module is used to monitor the position deviation between the target AR marker's position and the positioning data in real time, and in response to the position deviation being greater than a set position deviation, to update the target multimodal features corresponding to the target bifurcation point identifier in the mapping library in real time.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that can be loaded by a processor and executed as described in any one of claims 1 to 8 for the navigation method of a bronchoscope.

Images