Navigation device for bronchoscopes

Abstract

The application relates to a navigation device of a bronchoscope, and relates to the technical field of medical information processing. A mapping library is constructed based on dynamic tomographic images. In the process of the bronchoscope advancing along an initial path, the target bifurcation point identifier corresponding to a target bifurcation point is determined based on the matching of the multi-modal data of the target bifurcation point and the mapping library. Then, the target AR mark corresponding to the target bifurcation point identifier is determined based on the mapping library, and is superimposed in the field of view of the bronchoscope through an AR display device, so that the position mark, characteristic mark, risk mark and path mark of the bifurcation point and other multi-modal parameters are intuitively presented. Finally, the target multi-modal characteristics corresponding to the target bifurcation point identifier in the mapping library are updated in real time based on the position deviation value between the mark position and the positioning data of the target AR mark monitored in real time. In conclusion, the application improves the accuracy and safety of bronchoscope navigation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical information processing, in particular to a bronchoscope navigation device. BACKGROUND

[0002] Bronchoscope technology is one of the core means for the diagnosis and treatment of lung tumors, infectious diseases, diffuse lung diseases, etc. It realizes the visualization observation and accurate operation of each branch of the bronchial tree by inserting a slender bronchoscope into the lung through the airway.

[0003] The bronchial tree is a complex tree-shaped anatomical structure of the human body, which has a significant hierarchical characteristic. From the main bronchus, it gradually branches into the segmental bronchus, the sub-segmental bronchus, and the fine bronchus. The number of bifurcation points at each level increases exponentially with the increase of the hierarchy. In this complex structure, there are bifurcation points with similar geometric characteristics but significantly different mechanical characteristics or location characteristics. For example, the bifurcation of the right upper lobe anterior segmental bronchus and the middle lobe lateral segmental bronchus may both have an acute angle of 30° to 45°, and the diameter ratio of the main branch to the branch may also be close to 2:1. From the geometric characteristics alone, the two are easily confused.

[0004] The traditional bronchoscope navigation technology only relies on the anatomical hierarchy and rough spatial coordinates of the bronchial tree to construct a navigation model in the bifurcation point identification and matching link, without incorporating quantitative parameters of geometric characteristics, resulting in a bifurcation point matching error rate of more than 30% for similar hierarchical bifurcation points. This leads to the bronchoscope entering the wrong branch, which not only prolongs the operation time, but also may cause mucosal damage, bleeding due to repeated adjustment of the instrument, and even miss the target lesion or damage normal lung tissue during biopsy or ablation operation. SUMMARY

[0005] The purpose of the present application is to provide a navigation device for a bronchoscope to solve the problem of low precision of traditional bronchoscope navigation technology.

[0006] In order to achieve the above-mentioned purpose, the present application provides a navigation device for a bronchoscope, comprising:

[0007] A construction module is configured to construct a mapping library based on dynamic tomographic images of the bronchial region of a patient. The mapping library includes a mapping relationship between the bifurcation point identifiers, multi-modal characteristics, and AR markers of the bifurcation points of the bronchial tree. The multi-modal characteristics include geometric characteristics, mechanical characteristics, and location characteristics. The AR markers include location markers, feature markers, risk markers, and path markers.

[0008] The matching module is configured to acquire multi-modal data of a target bifurcation point in real time during the bronchoscope traveling along the initial path, extract target multi-modal features corresponding to the multi-modal data, match the target multi-modal features with the mapping library, and determine a target bifurcation point identifier corresponding to the target bifurcation point, wherein the multi-modal data includes optical image data, ultrasonic echo data, and positioning data;

[0009] The display module is configured to determine a target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and superimpose the target AR marker in the field of view of the bronchoscope through an AR display device.

[0010] The correction module is configured to monitor a position deviation value between a marker position of the target AR marker and the positioning data in real time, and update target multi-modal features corresponding to the target bifurcation point identifier in the mapping library in real time in response to the position deviation value being greater than a set position deviation value.

[0011] The matching module includes:

[0012] The first feature extraction unit is configured to extract a contour of a bronchial tree in the optical image data through an edge detection algorithm, detect axes of a main branch and a branch in the bronchial tree through a Hough transform, determine a bifurcation angle of the target bifurcation point, a diameter ratio of the main branch and the branch where the target bifurcation point is located, and a bifurcation number, and obtain geometric features of the target bifurcation point.

[0013] The second feature extraction unit is configured to calculate a tube wall thickness where the target bifurcation point is located based on the ultrasonic echo data, convert an intensity peak value of the ultrasonic echo data into an elastic modulus, calculate a stiffness coefficient of the target bifurcation point based on the tube wall thickness and the elastic modulus, and obtain mechanical features of the target bifurcation point.

[0014] The third feature extraction unit is configured to perform respiratory compensation on the positioning data based on a current respiratory phase to obtain compensated positioning data, determine an anatomical classification of the target bifurcation point according to the compensated positioning data, and calculate a risk distance parameter corresponding to the compensated positioning data, and obtain position features of the target bifurcation point.

[0015] The cross-validation unit is configured to cross-check the geometric features, the mechanical features, and the position features of the target bifurcation point, and determine the target multi-modal features of the target bifurcation point based on a cross-checking result.

[0016] The application has the following beneficial effects:

[0017] The application fuses geometric features, mechanical features and position features to form a multi-modal unique identifier of the bifurcation point of the bronchial region. A mapping library is constructed based on dynamic tomographic images, which can adapt to the dynamic anatomical environment. During the advancement of the bronchoscope along the initial path, the multi-modal data of the target bifurcation point is matched with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point, which can accurately distinguish the bifurcation points that are geometrically similar but different in mechanical properties. Then, the target AR marker corresponding to the target bifurcation point identifier is determined based on the mapping library and is superimposed in the field of view of the bronchoscope through the AR display device, so as to intuitively present the multi-modal parameters such as the position marker, feature marker, risk marker and path marker of the bifurcation point, reduce the dependence of the doctor on experience and improve the operation efficiency. Finally, based on the position deviation value between the marker position and the positioning data of the target AR marker monitored in real time, the target multi-modal features corresponding to the target bifurcation point identifier in the mapping library are updated in real time, which can ensure that the navigation model is always synchronized with the actual anatomical state and improve the stability of the navigation accuracy. In summary, the application improves the accuracy and safety of bronchoscope navigation through multi-modal feature fusion, dynamic model construction, AR marker and closed-loop updating mechanism, and is especially suitable for complex sub-segment bronchial regions.

[0018] Other features and advantages of the present application will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 An application scenario diagram of a bronchoscope navigation method provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of a bronchoscope navigation method provided in an embodiment of the present application;

[0021] Figure 3 A model structure diagram of a bronchial tree provided in an embodiment of the present application;

[0022] Figure 4 A structure diagram of the upper lobe of the left lung of a bronchial tree provided in an embodiment of the present application.

[0023] Figure 5 A structure diagram of a bronchoscope navigation device provided in an embodiment of the present application.

[0024] BRIEF DESCRIPTION OF DRAWINGS

[0025] 110, controller; 120, bronchoscope; 130, AR display device. DETAILED DESCRIPTION

[0026] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0027] In the description of the present application, it should be understood that the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited. In the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration". Any embodiment described as "exemplary" in the present application is not necessarily to be construed as preferred or advantageous over other embodiments. In order to enable any person skilled in the art to implement and use the present application, the following description is given. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can realize the present application without using these specific details. In other examples, well-known structures and processes will not be described in detail to avoid unnecessary details making the description of the present application obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope consistent with the principles and characteristics disclosed in the present application.

[0028] The navigation method of the bronchoscope in the embodiments of the present application is applied to the controller 110 as shown. Figure 1 Figure 1 The application scenario of the control method of the bronchoscope 120 provided in the embodiments of the present application is shown. The application scenario can include a controller 110, a bronchoscope 120, and an AR display device 130. The controller 110 communicates with the bronchoscope 120 and the AR display device 130, respectively.

[0029] The front end of the bronchoscope 120 can integrate a multi-sensor module, and the rear end communicates with the controller 110 through a special interface to provide data support for navigation. As an example, the multi-sensor module can include an optical imaging sensor, a miniature ultrasonic probe, and a positioning sensor (such as an electromagnetic positioning sensor), etc.

[0030] ​The optical imaging sensor can include a miniature light source and an image sensor, etc. The optical imaging sensor can collect optical image data. For example, the miniature light source illuminates the inside of the bronchial cavity to ensure that the bronchial tree in the field of view is clearly outlined. The image sensor can be a miniature camera with a resolution suitable for the operating scene of the bronchoscope, which can capture real-time optical images inside the bronchial cavity. Based on the optical images, the geometric characteristics of the target bifurcation point can be obtained.

[0031] The miniature ultrasonic probe can be integrated into an endoscopic ultrasound (EBUS) module embedded in the working channel of the bronchoscope. The ultrasonic signal transmitting and receiving circuit can be integrated into the handle or controller of the bronchoscope to drive the miniature ultrasonic probe to emit ultrasonic waves and receive reflected ultrasonic echo data. Based on the ultrasonic echo data, the mechanical characteristics of the target bifurcation point can be calculated.

[0032] The positioning sensor is usually an electromagnetic positioning sensor to adapt to the non-metallic environment without radiation risk. The positioning sensor can be integrated into the bronchoscope near the imaging assembly. In another example, optical positioning can also be used. The positioning sensor collects the positioning data of the bronchoscope, which can then be combined with an external respiratory phase acquisition device to obtain the position characteristics of the target bifurcation point in real time.

[0033] The augmented reality (AR) display device 130 is a kind of intelligent reality terminal that superimposes virtual information on the real environment in real time through optical see-through technology, which not only retains the user's intuitive perception of the real scene, but also supplements the invisible information in the field of view through computer-generated virtual content, and realizes the interaction between the display world and digital information. For example, in the scene of the bronchoscope in the embodiments of the present application, the AR display device 130 can include a head-mounted AR display device 130. The field of view of the bronchoscope 120 can be displayed in the head-mounted AR display device 130, which refers to the image range of the bronchus and the surrounding tissue captured by the optical imaging sensor at the front end of the bronchoscope 120 and transmitted to the AR display device in real time. Then, through AR markers, real-time anatomical information, navigation paths, operation instructions, risk warnings, and deviation prompts can be superimposed into the field of view of the bronchoscope 120.

[0034] The application scenario of the navigation method of the bronchoscope 120 in the embodiments of the present application includes a controller 110 for the navigation method, which can run the corresponding computer readable storage medium of the bronchoscope 120 to execute the steps of the navigation method of the bronchoscope 120.

[0035] It can be understood that, Figure 1The various electronic devices in the application scenario of the navigation method of the bronchoscope 120 shown do not constitute a limitation on the embodiments of this application. That is, the number and types of devices included in the application scenario of the navigation method of the bronchoscope 120, or the number and types of devices included in each electronic device, do not affect the overall implementation of the technical solution in the embodiments of this application, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of this application.

[0036] In this application embodiment, the controller 110 can be a standalone device, or a network of devices or a cluster of devices. For example, the controller 110 described in this application embodiment includes, but is not limited to, a computer, a network host, a single network device, a set of multiple network devices, or a cloud device composed of multiple devices. Among them, the cloud device is composed of a large number of computers or network devices based on cloud computing.

[0037] Those skilled in the art will understand that Figure 1 The application scenarios shown are merely one application scenario corresponding to the technical solution of this application, and do not constitute a limitation on the application scenarios of the technical solution of this application. Other application scenarios may include more than one application scenario. Figure 1 The number of more or fewer electronic devices shown, or the network connections of electronic devices, for example Figure 1 Only one electronic device is shown in the diagram. It is understood that the navigation method of the bronchoscope 120 may also include one or more other electronic devices, which are not limited here.

[0038] It should be noted that, Figure 1 The application scenario of the navigation method of the bronchoscope 120 shown is merely an example. The application scenario of the navigation method of the bronchoscope 120 described in the embodiments of this application is to more clearly illustrate the technical solution of the embodiments of this application, and does not constitute a limitation on the technical solution provided in the embodiments of this application.

[0039] Based on the application scenarios of the above-mentioned navigation method for bronchoscope 120, an embodiment of the navigation method for bronchoscope 120 is proposed. A detailed description is provided below with reference to the accompanying drawings.

[0040] Figure 2 This is a flowchart illustrating a bronchoscope navigation method provided in an embodiment of this application. Figure 2 As shown, this navigation method can execute steps 201-204 through the controller 110 described above, which will be described in detail below.

[0041] Step 201, constructing a mapping library based on dynamic tomographic images of the bronchial region of the patient, the mapping library including a mapping relationship between a bifurcation point identifier, multi-modal features, and AR markers of the bifurcation points of the bronchial tree. The multi-modal features can include geometric features, mechanical features, and position features, and the AR markers can include position markers, feature markers, risk markers, and path markers.

[0042] The dynamic tomographic image refers to a sequence of tomographic images that can capture the movement of the patient, which is closer to the real physiological state of the patient than the static tomographic image. For example, the dynamic tomographic image can refer to X-ray computed tomography (CT) at different breathing phases, i.e., 4D-CT or magnetic resonance imaging (MRI). The breathing phases can include end-inspiration, mid-inspiration, quiet phase, mid-expiration, and end-expiration. In one example, a dynamic model of the bronchial tree can be constructed by three-dimensional reconstruction technology to capture the position changes of the bronchial tree caused by respiratory movement.

[0043] There are a large number of bifurcation points in the bronchial tree that are geometrically similar but have significant differences in mechanical features or position features. The bifurcation point of the bronchial tree is a specific anatomical position where each level of bronchial branch of the bronchial tree structure branches and bifurcates, is a node in the branching process of the bronchial tree from the trunk to the tip, and is also a key positioning node for navigation. The embodiments of the present application construct a mapping library that can break through the single feature dependence of bronchoscopic navigation, which includes a mapping relationship between a bifurcation point identifier, multi-modal features, and AR markers of the bifurcation points of the bronchial tree.

[0044] The bifurcation point identifier is a unique numerical code for each bifurcation point, which can be generated by combining, for example, anatomical classification, three-dimensional coordinates, and feature types. The multi-modal features can include geometric features, mechanical features, and position features of the bifurcation points. The geometric features are features reflecting the morphology of the bifurcation point, which can include bifurcation angle, main branch and branch diameter ratio, and bifurcation number, etc. The mechanical features are features reflecting the hardness of the tissue, which can include wall elastic modulus, stiffness coefficient, etc. calculated based on the image. The position features are features reflecting the position information such as three-dimensional coordinates of the bifurcation point. By describing the parameters of the bifurcation point attributes from different dimensions, the limitations of single features of the bifurcation point can be broken through.

[0045] The AR marker is a virtual visualization element associated with the bifurcation point, which can include a position marker, a feature marker, a risk marker, and a path marker. The position marker is a three-dimensional box indicating the spatial position of the bifurcation point. The feature marker is a text label labeling the parameters of the geometric features, mechanical features, etc. The risk marker is a marker for risk warning based on the color of the blood vessel distance, for example, red represents high risk. The path marker is an arrow line guiding the bronchoscope to travel.

[0046] In one example, all the bifurcation points of the bronchial tree can be identified by an image segmentation algorithm. Then the multi-modal features of each bifurcation point are extracted and assigned with a unique identifier as the bifurcation point identifier. Finally, the corresponding AR marker is generated according to the feature parameters. In this way, a mapping relationship between the bifurcation point identifier, the multi-modal feature and the AR marker can be formed, and the mapping relationship stored can construct a mapping library as the data basis for subsequent navigation methods. Through multi-dimensional feature combination, similar bifurcation points can be accurately distinguished, and the AR marker can be associated in advance to provide data basis for real-time display in bronchoscopy operation and reduce calculation delay.

[0047] Step 202, real-time multi-modal data of the target bifurcation point is acquired during the bronchoscope advancing along the initial path, and the target multi-modal feature corresponding to the multi-modal data is extracted. The target multi-modal feature is matched with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point.

[0048] The initial path is a path planned in advance before bronchoscopy operation. The target bifurcation point is a bifurcation point passed through during the bronchoscope advancing along the initial path. For the target bifurcation point, multi-modal data can be acquired in real time, which can include optical image data, ultrasonic echo data and positioning data. The optical image data is a real-time picture collected by an optical imaging sensor. The ultrasonic echo data is an echo signal of the tube wall and surrounding tissue obtained by a miniature ultrasonic probe. The positioning data is a three-dimensional coordinate of the front end of the bronchoscope output by a positioning sensor.

[0049] During the bronchoscope advancing along the initial path, the multi-modal data of the target bifurcation point can be acquired in real time, and then the target multi-modal feature corresponding to the multi-modal data is extracted. The target multi-modal feature can include target geometric feature, target mechanical feature and target position feature. The concepts of target geometric feature, target mechanical feature and target position feature can refer to the description of multi-modal feature in the above mapping library construction. Matching the target multi-modal feature with the mapping library can determine the target bifurcation point identifier corresponding to the target bifurcation point. For example, a weighted Euclidean distance algorithm can be used to compare the target multi-modal feature extracted in real time with the pre-stored feature in the mapping library, and when the matching degree exceeds the set matching degree, the corresponding target bifurcation point can be determined. As an example, for bifurcation points with similar geometric features, the weights of mechanical features and position features can be increased to improve the discrimination of different bifurcation points.

[0050] Through multi-modal data fusion, the problem of easy confusion of single feature can be solved, especially in the sub-segment bronchial region, which can improve the recognition of bifurcation points.

[0051] Step 203, determine the target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and superimpose the target AR marker in the field of view of the bronchoscope through the AR display device.

[0052] According to the target bifurcation point identifier, the AR marker corresponding to the target bifurcation point identifier, i.e. the position marker, the feature marker, the risk marker and the path marker, can be determined from the mapping library. Through optical see-through technology, the AR display device can spatially register the virtual AR marker with the real-time field of view of the bronchoscope, realizing the superimposed display of real anatomical structure and virtual navigation information. In one example, when the bronchoscope moves, the AR marker can update the position and angle in real time as the field of view changes, maintaining spatial consistency with the real bifurcation point. In another example, when multiple AR markers overlap, the display level can be automatically adjusted, for example, the risk marker is displayed in the highest level.

[0053] By intuitively presenting abstract navigation information, the doctor can intuitively observe the three-dimensional structure of the bronchial tree. Through the cooperation of multiple types of AR markers, comprehensive information can also be provided to help doctors quickly judge the characteristics of the target bifurcation point and operation risks, etc.

[0054] Step 204, real-time monitoring of the position deviation value between the marker position positioning data of the target AR marker, in response to the position deviation value being greater than the set position deviation value, real-time updating of the target multi-modal feature corresponding to the target bifurcation point identifier in the mapping library.

[0055] The bronchoscope navigation in the embodiments of the present application is to guide the doctor to find the operation target bifurcation point through the AR marker. Compared with the bronchoscope navigation that strictly follows the preset path operation, there is a problem of position deviation between the AR marker and the actual bifurcation. Even if the path completely conforms to the planning, the AR marker may be misaligned with the actual bifurcation due to the mapping library not fully adapting to respiratory motion, tube wall deformation, registration error, etc., thereby easily leading to operation failure. Focusing only on the deviation of the bronchoscope and the planned path cannot solve the problem of inaccurate control of the bronchoscope path and angle caused by the doctor's hand shaking during the operation process. Therefore, the embodiments of the present application dynamically correct the misalignment between the set data and the real anatomy by focusing on the position deviation value between the AR marker and the actual bifurcation.

[0056] The position deviation value refers to the three-dimensional distance between the theoretical position of the target AR marker calculated in real time and the bronchoscope positioning data. The set position deviation value is a threshold value for determining whether the mapping library needs to be updated. When the real-time monitored position deviation value is greater than the set position deviation value, the target multi-modal feature corresponding to the target bifurcation point in the mapping library needs to be updated, and the corresponding target AR marker is also updated synchronously. In one example, the feature parameters of the corresponding bifurcation point in the mapping library can be weighted and corrected according to the priority of the position feature, the geometric feature and the mechanical feature. Through the closed loop of deviation monitoring and dynamic updating of the mapping library, it is ensured that the AR marker is anchored to the real bifurcation point, and the fatal risk caused by the correct path but the wrong marker is reduced.

[0057] The bifurcation point of the bronchial region is formed by fusing the geometric feature, the mechanical feature and the position feature. Based on the dynamic tomographic image, the mapping library is constructed, which can adapt to the dynamic anatomical environment. During the advancement of the bronchoscope along the initial path, the multi-modal data of the target bifurcation point is matched with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point, which can accurately distinguish the bifurcation points that are geometrically similar but mechanically different. Then, the target AR marker corresponding to the target bifurcation point identifier is determined based on the mapping library, and is superimposed in the field of view of the bronchoscope through the AR display device, so as to intuitively present the multi-modal parameters such as the position marker, the feature marker, the risk marker and the path marker of the bifurcation point, reduce the dependence of the doctor on experience, and improve the operation efficiency. Finally, based on the position deviation value between the marker position of the target AR marker and the positioning data, the target multi-modal feature corresponding to the target bifurcation point identifier in the mapping library is updated in real time, which can ensure that the navigation model is always synchronized with the actual anatomical state, and improve the stability of the navigation accuracy. In summary, the present application improves the accuracy and safety of bronchoscope navigation through multi-modal feature fusion, dynamic model construction, AR marker and closed loop updating mechanism, and is especially suitable for complex sub-segment bronchial regions.

[0058] In step 201, dynamic tomographic images of the bronchial region of the patient at multiple respiratory phases can be acquired first to construct a three-dimensional dynamic model of the bronchial tree. The multiple respiratory phases cover the complete respiratory cycle of the patient. For example, five typical respiratory phases, including end-inspiration, mid-inspiration, resting respiration, mid-expiration, and end-expiration, can be included to reduce the defect that a single phase cannot reflect the dynamic changes of the anatomy. The dynamic tomographic images can clearly capture the features of the position offset and the tube diameter change of the bronchial tree with respiration. In order to improve the accuracy, respiratory artifacts can also be eliminated by respiratory motion correction, and then the bronchial tree structure can be separated from the dynamic tomographic images by an image segmentation algorithm to remove interfering tissues such as blood vessels and lungs. Based on the segmented bronchial tree images at multiple phases, a corresponding number of three-dimensional models can be generated by three-dimensional reconstruction respectively, and each three-dimensional model corresponds to a respiratory phase. Then, a three-dimensional dynamic model of the complete respiratory cycle can be constructed by a dynamic interpolation method to simulate the continuous deformation process of the bronchial tree from end-inspiration to end-expiration.

[0059] After the three-dimensional dynamic model of the bronchial tree is constructed, the bifurcation points in the three-dimensional dynamic model can be identified, and the geometric features, mechanical features, and position features of the bifurcation points can be extracted. The bifurcation points can be extracted by a branch detection algorithm, such as a bifurcation point based on the skeleton of the bronchial tree, to determine the total number of bifurcation points of the bronchial tree. Then, the geometric features, mechanical features, and position features of the bifurcation points can be extracted, and the combination of parameters describing each bifurcation point from multiple dimensions can reduce the problem that a single feature is difficult to distinguish similar bifurcation points, thereby providing data basis for subsequent coding.

[0060] Secondly, each bifurcation point can be coded based on the multi-modal features of each bifurcation point to obtain a unique identifier of each bifurcation point, and the unique identifier is taken as a bifurcation point identifier of the bifurcation point. The unique identifier is generated based on the multi-modal features of each bifurcation point, and can distinguish the unique digital code of each bifurcation point of the bronchial tree. In order to make each bifurcation point unique, in an example, three feature dimensions of non-repeatability are selected for coding by the embodiments of the present application. For example, the three feature dimensions can include an anatomical level, a three-dimensional coordinate, and a geometric feature type.

[0061] Specifically, the position feature of the bifurcation point can be used to determine the corresponding anatomical structure of the bifurcation point in the bronchial tree, and the anatomical level of the bifurcation point can be determined based on the anatomical structure. The anatomical level is coded to obtain a first sub-identifier.

[0062] Anatomical classification is to divide the bronchial tree structure at the bifurcation point into multiple levels according to the hierarchical relationship of the main stem and branches of the bronchial tree, so as to reflect the macro position of the bifurcation point in the whole bronchial tree. The macro position is determined based on the anatomical structure. The anatomical structure can include main bronchus, lobar bronchus, segmental bronchus, subsegmental bronchus and fine bronchus. The position characteristics of the bifurcation point can be used to determine the anatomical structure corresponding to the bifurcation point. Each anatomical structure corresponds to an anatomical classification, which can be identified by numbers 1-5, which is simple and easy to identify. The first sub-identifier is a digital code for anatomical classification, and can be associated with the abbreviation of the location, etc. In this way, the macro index of the first sub-identifier can quickly locate the bifurcation point of the large anatomical range, and can reduce the search amount.

[0063] Figure 3 A model structure of a bronchial tree is provided in the embodiments of the present application. As shown in Figure 3 , the trachea is divided into left main bronchus and right main bronchus, which correspond to left lung and right lung respectively. The two main bronchi can correspond to level 1, and the bifurcation point between the left main bronchus and the right main bronchus is the carina. The right main bronchus diverges from the trachea at an angle of 20-30 degrees, and the left main bronchus diverges at an angle of 45-55 degrees. The lobar bronchus diverging from the main bronchus can correspond to level 2. The right lung includes three lobar bronchi, and the left lung includes two lobar bronchi. Further, each lobar bronchus can be divided into segmental bronchi, which correspond to level 3, and correspond to lung segments. For example, the right upper lobe of the lung can include apical segmental bronchus, anterior segmental bronchus and posterior segmental bronchus. Subsegmental bronchi (not shown in the figure) can correspond to level 4, diverging from segmental bronchi, located inside the lung segment, and the branches are relatively dense. Fine bronchi (not shown in the figure) can correspond to level 5, diverging from subsegmental bronchi, with a tube diameter usually less than 3mm, close to the alveoli, and the tube wall has no cartilage. In this way, the matching range of the bifurcation point can be quickly narrowed down, and it conforms to the cognitive logic of the bronchial tree by doctors.

[0064] Secondly, the three-dimensional coordinates of the bifurcation point in the three-dimensional dynamic model are determined based on the position characteristics of the bifurcation point, and a second sub-identifier is obtained by encoding the three-dimensional coordinates. The three-dimensional coordinates are a spatial coordinate system established by the bronchial tree in the three-dimensional dynamic model, and the real-time three-dimensional coordinates of the bifurcation point in the spatial coordinate system can reflect the micro spatial position of the bifurcation point. In order to reduce the coordinate fluctuation caused by the extreme position of the end of inspiration or expiration, the three-dimensional coordinates of the three-dimensional dynamic model in the stable breathing period can be selected as the basis for encoding. For example, the spatial coordinate system can be established with the carina as the origin, the X-axis as the front-back direction of the three-dimensional dynamic model, the Y-axis as the left-right direction, and the Z-axis as the up-down direction. The second sub-identifier is a spatial index of the unique identifier formed by encoding the three-dimensional coordinates, which can solve the problem of distinguishing multiple bifurcation points at the same anatomical level. For example, there are 3 bifurcation points of 4-level sub-segments in the right upper lobe, which need to be distinguished by coordinates. Assuming that the three-dimensional coordinates of a bifurcation point are X=100.6mm, Y=80.2mm and Z=60.5, the second sub-identifier can be represented as 100.6_80.2_60.5. There is a phase change in the actual operation of the bronchoscope, which can be quickly corrected by the coordinate offset to maintain the accuracy of the positioning. In this way, the problem of distinguishing bifurcation points at the same anatomical level can be solved, and the position data basis for subsequent spatial registration of AR markers is also provided.

[0065] Then, the feature type of the bifurcation point is determined based on the geometric characteristics of the bifurcation point, and a third sub-identifier is obtained by encoding the feature type. The geometric characteristics of the bifurcation point can include bifurcation angle, main branch and branch diameter ratio, and bifurcation number, etc. The most distinguishing features can be selected from these geometric characteristics as the encoding basis of the third sub-identifier. The third sub-identifier is an identifier obtained by encoding the feature type, which can solve the problem of distinguishing bifurcation points with the same level, similar coordinates and significant geometric differences. In the operation of the bronchoscope, the diameter ratio and the number of bifurcations are the main factors affecting the difficulty of advancing the bronchoscope and the difficulty of selecting branches, so the diameter ratio and the number of bifurcations can be selected to determine the feature type. The feature type can be divided according to the clinical risk and operation difficulty. For example, according to the diameter ratio, it can be divided into high risk and low risk, and according to the number of bifurcations, it can be divided into binary and multi-branch types. For example, the diameter ratio can be encoded as β+specific value, and the number of bifurcations can be marked as value+bifurcation. For example, for a binary type with a diameter ratio of 2.5, it can be β2.5-binary. Associating the feature type with the operation risk can provide data basis for the strategy and display of subsequent navigation, and reduce the decision-making time in operation.

[0066] Finally, the first, second, and third sub-identifiers are combined to obtain a unique identifier for the bifurcation point. These three sub-identifiers can be combined according to macroscopic, microscopic, and attribute logic. The combination format can be set according to requirements; for example, they can be connected using "-" to obtain first sub-identifier-second sub-identifier-third sub-identifier. Generating a unique identifier through three levels of sub-identifiers solves the problem of difficulty in distinguishing similar bronchial tree bifurcation points, reducing data confusion. Furthermore, associating the encoding with macroscopic, microscopic, and attribute aspects allows for rapid determination of the bifurcation point's attributes without needing to consult additional data, improving the efficiency of bronchoscopy.

[0067] Figure 4 This is a schematic diagram of the structure of the left upper lobe of the bronchial tree provided in an embodiment of this application. Figure 4 Taking the bifurcation point A of the anterior segment of the left upper lobe as an example, the anterior segment of the left upper lobe belongs to the segmental bronchus, which branches off from the lobar bronchus of the left upper lobe, corresponding to an anatomical grade of 3. Therefore, the first sub-identifier can be B3-L. B represents the bronchus, 3 represents the anatomical grade, and L represents the left lung. Assuming the three-dimensional coordinates are 26, 55, 25, the second sub-identifier can be (26, 55, 25). Assuming the geometric features are a bifurcation angle θ = 45° and a diameter ratio β = 1.2, the third sub-identifier can be θ45β1.2. Finally, the first, second, and third sub-identifiers are combined and connected by "-", resulting in the unique identifier for the anterior segment of the left upper lobe: B3-L-(26, 55, 25)-θ45β1.2, which is used as the bifurcation point identifier. In this way, the location of the bifurcation point can be quickly obtained from the bifurcation point marker, the anatomical grade of the operation can be clearly defined, and the coordinates can be quickly determined, reducing the positioning time. At the same time, geometric features can also provide auxiliary prompts on the current operation difficulty, so that doctors can make timely decisions.

[0068] Additionally, location markers and risk markers can be generated based on location features associated with the bifurcation point identifier. Feature markers are generated based on geometric and mechanical features associated with the bifurcation point identifier. A navigation strategy is generated based on the geometric, mechanical, and location features associated with the bifurcation point identifier, and path markers are generated based on the navigation strategy. The path markers may include suggested operational parameters.

[0069] Specifically, the position marker can be generated according to the three-dimensional coordinates of the bifurcation point in the quiet breathing period, and displayed by a three-dimensional translucent frame. Similarly, the risk marker can also be formed according to the position characteristics of the bifurcation point, such as the shortest distance of the pleura or blood vessels, to form a warning indication. For example, the risk level can be distinguished by color, with red indicating the highest risk level, yellow indicating medium risk, and green indicating low risk. The feature marker can be generated based on the geometric and mechanical characteristics of the bifurcation point, for example, by generating a feature marker through geometric and mechanical characteristics, and displaying it next to the position marker to identify the properties of the bifurcation point. According to the calculation of geometric characteristics, mechanical characteristics and position characteristics, a navigation strategy can be obtained, based on which a directional guide line, i.e. a path marker, can be formed. The path marker can also include recommended operation parameters to provide operation guidance and recommendations for the doctor. For example, an arrow points to the opening of the next branch at the bifurcation point, and the recommended pushing speed and minimum turning radius are marked. In this way, the data needed in the operation can be intuitively presented in the bronchoscope field through AR markers, improving the decision-making efficiency and operation safety of the doctor.

[0070] The bifurcation points of the bronchial tree and the multi-modal characteristics of the bifurcation points are identified by dynamic tomographic images, and then each bifurcation point is uniquely identified and associated with the parameters required for AR markers to form a mapping library. This can provide a data basis for subsequent AR guidance and correction, etc.

[0071] In step 202, the contour of the bronchial tree in the optical image data can be extracted by an edge detection algorithm, the axis of the main branch and the branch in the bronchial tree can be detected by Hough transform, and the bifurcation angle of the target bifurcation point, the diameter ratio of the main branch and the branch where the target bifurcation point is located, and the number of bifurcations can be determined to obtain the geometric characteristics of the target bifurcation point.

[0072] The edge detection algorithm is an algorithm for identifying pixels with abrupt changes in gray scale values in an image to determine the contour boundaries of the bronchial tree. For example, the Canny edge detection algorithm can be used to set double thresholds for pixels to determine the contour boundaries of the bronchial tree. The high threshold is used to determine strong edges, and the low threshold is used to retain weak edges, such as thin bronchial walls. For example, assume that the high threshold is 150 and the low threshold is 50. Pixels higher than 150 can be determined as strong edges, which are true edges, and are directly retained without subsequent verification. Pixels higher than 50 but less than or equal to 150 can be determined as candidate edges, which need to be verified for edge connectivity in the subsequent step. If the edges are continuous, they can be determined as extensions of true edges, and if the edges are isolated, they can be determined as false edges and filtered out. For example, candidate edges caused by local reflection of secretions are false edges. Pixels less than or equal to 50 are weak signals, such as background noise or blurred secretions, which can be determined as non-edges and need to be thoroughly filtered out. In this way, the loss of strong edges is reduced, the main stem of the bronchial tree is ensured to be complete, false edges are filtered out, and the accuracy of edge extraction of the bronchial tree is improved.

[0073] The Hough transform is a method of converting line detection in the image space to peak detection in the parameter space, which can effectively identify interrupted lines disturbed by noise. Since the axes of the main branches and the branches of the bronchial tree are center lines of continuous straight edge lines, they will form obvious voting peaks in the parameter space. The main branch axis is selected as the line with the highest voting peak and the longest length, and the branch axis is selected as the line with a certain angle to the main branch axis and a voting number greater than a set voting number. The set voting number can be set according to the characteristics of the branch axis. Thus, interrupted lines disturbed by noise, such as airway edges, can be effectively identified. Then, based on the identified main branch and branch axes, geometric features of the target bifurcation point, such as the bifurcation angle, the diameter ratio of the main branch and the branch, and the number of bifurcations, can be determined. The bifurcation angle is the angle between the main branch axis and the branch axis in the image plane, and if there are multiple branches, the angle between each branch and the main branch is calculated. The diameter ratio is measured in the vertical direction of the axis. The number of bifurcations is the number of branch axes that form an effective angle with the main branch axis and have a voting number that meets the standard.

[0074] The geometric features are the basic dimensions for bifurcation point identification, and accurate bifurcation angle and diameter ratio can reduce the matching range of the mapping library and improve the matching efficiency.

[0075] The propagation speed of ultrasound echoes in soft tissue is constant, so the thickness of the tube wall where the target bifurcation point is located can be calculated based on the ultrasound echo data. For example, the tube wall thickness can be obtained by dividing the time difference of the echoes by the speed of sound and then dividing by 2.

[0076] In the operation of bronchoscope, it is necessary to determine the wall hardness, so it is necessary to calculate the elastic modulus, and it is also necessary to know the closing deformation resistance, so it is necessary to calculate the stiffness coefficient. The elastic modulus cannot be directly measured and needs to be indirectly converted through ultrasonic echo data. The harder the tissue, the higher the elastic modulus, the greater the density, the stronger the reflection ability of ultrasonic waves, and the higher the peak of echo intensity. In an example, the intensity peak of the ultrasonic echo data can be converted into the elastic modulus. For example, a correlation model can be established based on the ultrasonic echo intensity and the elastic modulus calibration test, and the elastic modulus simulation tissue test is used to obtain the regression formula of the elastic modulus conversion. The elastic modulus can be calculated based on the regression formula.

[0077] The stiffness coefficient is a comprehensive embodiment of the material hardness and the structure size. Based on the wall thickness and the elastic modulus, the stiffness coefficient of the target bifurcation point can be calculated, and the stiffness coefficient reflects the compression deformation resistance of the tube wall. For example, the stiffness coefficient can be the product of the elastic modulus and the cube of the wall thickness, so the mechanical characteristics of the target bifurcation point can be obtained. The mechanical characteristics can include the closing thickness, the elastic modulus and the stiffness coefficient.

[0078] The optical image data can only observe the surface of the tube wall, while the ultrasonic echo data can penetrate into the interior of the tube wall, thereby solving the problem of distinguishing the bifurcation points with similar geometry but large mechanical difference. In addition, the bifurcation point with low elastic modulus and low stiffness coefficient is thin-walled, and needs to be reduced in speed during operation to reduce the risk of perforation. The high elastic modulus may be a lesion tissue. Based on the elastic modulus, the risk can also be warned, and data basis is provided for the risk marking in the AR marking.

[0079] The lung is an organ that moves dynamically with respiration, and the position of the bronchial tree will shift periodically with the respiratory phase. If the original positioning data is directly used, it will cause the bifurcation point position determination to be distorted, and affect the accuracy of the anatomical classification and risk assessment. Therefore, for the position characteristics, the positioning data can be compensated based on the current respiratory phase to obtain compensated positioning data. For example, the respiratory motion data can be collected in real time through a patient chest impedance respiration detector to determine the respiratory phase. Based on the mapping relationship between the phase and the offset, the offset rule of the target bifurcation point relative to the reference anatomical point at the current respiratory phase is determined, so as to obtain the compensated positioning data. The reference anatomical point refers to an anatomical structure that is less affected by the respiratory phase, and usually has a stable position. For example, the tracheal carina and the bifurcation of the pulmonary vascular. Then, the anatomical classification of the target bifurcation point can be determined according to the compensated positioning data, and the risk distance parameter corresponding to the compensated positioning data is calculated, the risk distance parameter refers to the shortest distance from the bifurcation point to the adjacent important blood vessels, so as to obtain the position characteristics of the target bifurcation point. In this way, it can be ensured that the bifurcation point seen during the operation is the same structure as the bifurcation point before the operation, and the accuracy of the bifurcation point determination in the bronchial tree is improved.

[0080] In one example, the respiratory compensation respiratory deformation correlation model obtains compensated positioning data. Therefore, in step 201, an anatomical reference point associated with the bifurcation point identification can be marked in the dynamic tomography image, and the position information of the anatomical reference point is correlated to the bifurcation point identification, and the position variation coefficient of the anatomical reference point at multiple respiratory phases is less than a set coefficient. The set coefficient is a threshold for determining whether a certain bifurcation point of the bronchial tree in the respiratory motion is a stable anatomical structure. Less than the set coefficient, it means that the bifurcation point is a stable anatomical structure, and the variation coefficient is small, which can be used as an anatomical reference point. Then, the position offset of the bifurcation point relative to the anatomical reference point at multiple respiratory phases is recorded, and a respiratory deformation correlation model is established. The respiratory deformation correlation model is a mathematical model that quantifies the phase and the position offset of the bifurcation point, which can calculate the offset at any phase in real time to achieve dynamic compensation. Specifically, based on the respiratory deformation correlation model, the coordinate offset of the positioning data at the current respiratory phase relative to the anatomical reference point can be found. Then, the respiratory compensation is performed on the positioning data based on the coordinate offset to obtain compensated positioning data. The compensated positioning data can be the difference between the original coordinates and the coordinate offset.

[0081] Finally, the geometric features, mechanical features and position features of the target bifurcation point are cross-verified, and the target multi-modal features of the target bifurcation point are determined based on the cross-verification result. The cross-verification is a logical consistency verification of different modal features, that is, the geometric features, mechanical features and position features all need to conform to the anatomical rules of the bronchial tree. For example, the consistency of the geometric features and the position features is reflected in that the higher the anatomical level is, the smaller the bifurcation angle is, and the larger the diameter ratio of the main branch to the branch is. The consistency of the mechanical features and the geometric features is reflected in that the larger the main branch diameter is, the larger the wall thickness is, and the higher the elastic modulus is. The consistency of the mechanical features and the position features is reflected in that the bifurcation point close to the blood vessel usually has a higher elastic modulus than the bifurcation point far from the blood vessel due to long-term reception of blood flow pulsation. Based on the above consistency rules, different verification conditions can be set for the consistency verification between the features of different modalities, so that the unreasonable data caused by noise and algorithm errors of a single modality can be reduced, and the anatomical rules of the bronchial tree can be more consistent.

[0082] The original data of the geometric features and the mechanical features are superimposed in the AR mark, which is easy to increase the information acquisition burden of the doctor. Therefore, the embodiments of the present application can distinguish multiple AR marks by different colors or symbols, and display them in the field of view of the bronchoscope, so as to reduce the information interpretation time and improve the decision efficiency.

[0083] Since the small bronchus itself has a small diameter, it occupies a small proportion in the bronchoscope field of view, and the respiratory deviation is large. Therefore, in the target tracking process, it is easy to appear the case that the target cannot be boxed. Based on this, the embodiments of the present application divide the dynamic navigation box and the static navigation box, and dynamically navigate the size of the small bronchus according to the diameter, to solve the problem of poor positioning accuracy of small moving targets. For the bronchus with a larger diameter, such as the main bronchus and the leaf bronchus, it occupies a large proportion in the bronchoscope field of view, and the respiratory deviation is small, and the position is relatively stable. Therefore, the static box can meet the requirements of positioning the target, and can reduce the dispersion of the doctor's attention, and reduce the system calculation amount.

[0084] Specifically, in step 203, the geometric features of the target bifurcation point can be determined based on the target bifurcation point identification. The small bronchus or the bronchus with a larger diameter is distinguished according to the diameter of the target bifurcation point and the respiratory deviation. For example, a set diameter can be set in advance to distinguish the diameter. Less than 3mm without cartilage support is a small bronchus, and greater than or equal to 3mm with cartilage support is a non-small bronchus. For the respiratory deviation, a first set deviation and a second set deviation can be set as the judgment threshold of the respiratory deviation. Greater than the first set deviation indicates that the deviation is large, and less than the second set deviation indicates that the respiratory deviation is small. The first set deviation is greater than or equal to the second set deviation.

[0085] When the diameter of the target bifurcation point is less than the set diameter, and the respiratory deviation is greater than the first set deviation, it can be determined that it is a small bronchus. Therefore, a dynamic navigation box needs to be generated, the size of the dynamic navigation box changes with the actual diameter, and a high-risk label is displayed. The high-risk label can be displayed around the dynamic box to strengthen the risk prompt. When the diameter of the target bifurcation point is greater than or equal to the set diameter, and the respiratory deviation is less than the second set deviation, a static navigation box is generated, and the size of the static navigation box is a set size. The set size is a fixed size set to match the target detection scene of the non-small bronchus.

[0086] Then, the feature label is generated according to the geometric features and the mechanical features. The feature label is to convert the quantitative feature parameters into symbols, colors and texts that can be quickly recognized by the human eye, in order to compress information and reduce the interpretation time of the doctor. The corresponding attributes can be judged through the color and the symbol. For example, the red elastic modulus mark may be a diseased tissue. In this way, the guidance efficiency of AR display can be improved.

[0087] Finally, the risk markers and path markers can be generated according to the position features, and the risk markers and the path markers have different visual features. Both the risk markers and the path markers are generated based on the position features, the risk markers are used to prompt potential risks, and the path markers are used to plan the advancing path. Based on the anatomical classification and the compensation positioning data, the advancing direction of the bronchoscope can be marked by an arrow from the position where the bronchoscope is currently located to the target bifurcation point, and the risk markers and the path markers can be distinguished by different colors and shapes. The risk markers can solve the problem that the blood vessels are not obvious in the optical image data, and the path markers can reduce the risk of misoperation in the case of dense branching of the bronchial tree.

[0088] In the bronchoscope operation, the position deviation between the marker position of the target AR marker and the positioning data can be monitored in real time. In step 204, in response to the position deviation value being greater than the set position deviation value, the current actual multi-modal feature is extracted, and a feature deviation value of the actual multi-modal feature and the target multi-modal feature associated with the target bifurcation point identifier is calculated. The feature deviation value is the difference between the actual multi-modal feature and the target multi-modal feature in the mapping library, which can be calculated according to the position feature, the geometric feature, and the mechanical feature. The greater the deviation value, the greater the difference between the pre-stored data and the actual anatomy, and therefore correction is needed.

[0089] Then, the target multi-modal feature is corrected according to the order of the position feature, the geometric feature, and the mechanical feature based on the feature deviation value, and the target AR marker is updated synchronously according to the corrected target multi-modal feature. The correction order can follow the principle of giving priority to basic features, because the position feature is the reference for the geometric feature and the mechanical feature, and therefore the geometric feature is the morphological basis for the mechanical feature. If the order is reversed, it is easy to cause correction errors. Based on this, in one example, the position feature can be corrected first, then the geometric feature, and finally the mechanical feature. In this way, it can be ensured that the corrected feature data conforms to the actual structure of the bronchial tree. If the feature deviation value is not corrected in time, the positioning error will be large due to the cumulative operation time, which will affect the AR navigation and the marker, thereby causing the bronchoscope operation to be dangerous.

[0090] In the embodiments of the present application, hierarchical warning prompts can also be performed through the AR device. The hierarchical warning prompts are differential reminders according to the severity of the deviation. The present application can perform multi-modal hierarchical warning through the combination of mechanical feedback, visual feedback, and auditory feedback to ensure that the risks are corrected in time. For example, a first deviation value and a second deviation value can be set as hierarchical determination thresholds for the deviation amount, and the first deviation value is greater than the second deviation value.

[0091] Specifically, if the position deviation value is less than the first deviation value, indicating a slight deviation that is acceptable, i.e., a mild risk, the end trajectory of the fine-tuning path is adjusted. In the fine-tuning case, the selection of the branch is not affected. If the position deviation value is greater than or equal to the first deviation value but less than the second deviation value, indicating a moderate risk, a mechanical feedback prompt is generated and a deceleration prompt is displayed. For example, the bronchoscope handle vibrates, and a deceleration prompt is displayed on the AR display device. If the position deviation value is greater than or equal to the second deviation value, indicating a serious risk, a mechanical feedback prompt and an audible feedback prompt are generated, for example, the bronchoscope handle vibrates, the AR device not only displays a danger prompt, but also provides an audible prompt through a buzzer or the like. At the same time, a locking mechanism of the bronchoscope can also be activated, for example, only the bronchoscope is allowed to retreat, and the bronchoscope is not allowed to continue advancing, so as to reduce the risk of bronchoscope operation.

[0092] Through hierarchical early warning, differentiated warning measures can be provided according to the severity of the risk, and the existing risk can be intervened moderately. In addition, in combination with the multi-modal perception prompts of touch, vision and hearing, the situation that a single feedback is ignored by a doctor can be solved.

[0093] The embodiments of the present application form a double guarantee of active optimization and passive protection through feature correction and hierarchical early warning, which can improve the safety and accuracy of bronchoscope operation, and is especially suitable for operation scenarios of sub-segment and below small bronchus.

[0094] Figure 5 FIG. 1 is a structural schematic diagram of a navigation device 500 of a bronchoscope provided in an embodiment of the present application. As shown in the figure, the navigation device 500 of the bronchoscope can include a construction module 501, a matching module 502, a display module 503 and a correction module 504. Figure 5

[0095] The construction module 501 is configured to construct a mapping library based on a dynamic tomographic image of a bronchial region of a patient. The mapping library can include a bifurcation point identifier of a bifurcation point of a bronchial tree, a multi-modal feature, and a mapping relationship between AR markers. The multi-modal feature can include a geometric feature, a mechanical feature and a position feature. The AR markers can include a position marker, a feature marker, a risk marker and a path marker.

[0096] The matching module 502 is configured to acquire multi-modal data of a target bifurcation point in real time during the bronchoscope travels along an initial path, and extract target multi-modal features corresponding to the multi-modal data. The target multi-modal features are matched with the mapping library to determine a target bifurcation point identifier corresponding to the target bifurcation point. The multi-modal data can include optical image data, ultrasonic echo data and positioning data.

[0097] ​The display module 503 is configured to determine the target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and superimpose the target AR marker in the field of view of the bronchoscope through the AR display device.

[0098] The correction module 504 is configured to monitor a position deviation value between the marker position of the target AR marker and the positioning data in real time, and update the target multi-modal feature corresponding to the target bifurcation point identifier in the mapping library in real time in response to the position deviation value being greater than a set position deviation value.

[0099] The construction module 501 can include an acquisition unit, an encoding unit, and a marking unit.

[0100] The acquisition unit is configured to acquire dynamic tomographic images of a bronchial region of a patient at a plurality of breathing phases, construct a three-dimensional dynamic model of a bronchial tree, identify bifurcation points in the three-dimensional dynamic model, and extract geometric features, mechanical features, and position features of the bifurcation points.

[0101] The encoding unit is configured to encode each bifurcation point based on the multi-modal features of the bifurcation point to obtain a unique identifier of each bifurcation point, and use the unique identifier as a bifurcation point identifier of the bifurcation point.

[0102] The generation unit is configured to generate a position marker and a risk marker based on the position features associated with the bifurcation point identifier, generate a feature marker based on the geometric features and the mechanical features associated with the bifurcation point identifier, generate a navigation strategy based on the geometric features, the mechanical features, and the position features associated with the bifurcation point identifier, and generate a path marker based on the navigation strategy, the path marker including recommended operation parameters.

[0103] The encoding unit is further configured to determine an anatomical structure corresponding to the bifurcation point in the bronchial tree based on the position features of the bifurcation point, determine an anatomical classification of the bifurcation point based on the anatomical structure, encode the anatomical classification to obtain a first sub-identifier, the anatomical structure including a main bronchus, a lobar bronchus, a segmental bronchus, a sub-segmental bronchus, and a fine bronchus; determine a three-dimensional coordinate of the bifurcation point in the three-dimensional dynamic model based on the position features of the bifurcation point, and encode the three-dimensional coordinate to obtain a second sub-identifier; determine a feature type of the bifurcation point based on the geometric features of the bifurcation point, and encode the feature type to obtain a third sub-identifier; and combine the first sub-identifier, the second sub-identifier, and the third sub-identifier to obtain the unique identifier of the bifurcation point.

[0104] The matching module 502 can include a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, and a cross-validation unit.

[0105] The first feature extraction unit is configured to extract the contour of the bronchial tree in the optical image data by an edge detection algorithm, detect the axis of the main branch and the branch in the bronchial tree by a Hough transform, determine the bifurcation angle of the target bifurcation point, the diameter ratio of the main branch and the branch where the target bifurcation point is located, and the bifurcation number, so as to obtain the geometric feature of the target bifurcation point.

[0106] The second feature extraction unit is configured to calculate the wall thickness of the tube where the target bifurcation point is located based on the ultrasonic echo data, convert the intensity peak value of the ultrasonic echo data into the elastic modulus, and calculate the stiffness coefficient of the target bifurcation point based on the wall thickness and the elastic modulus, so as to obtain the mechanical feature of the target bifurcation point.

[0107] The third feature extraction unit is configured to perform respiratory compensation on the positioning data based on the current respiratory phase to obtain compensated positioning data, determine the anatomical classification of the target bifurcation point according to the compensated positioning data, and calculate the risk distance parameter corresponding to the compensated positioning data, so as to obtain the position feature of the target bifurcation point.

[0108] The cross-validation unit is configured to cross-check the geometric feature, the mechanical feature and the position feature of the target bifurcation point, and determine the target multi-modal feature of the target bifurcation point based on the cross-checking result.

[0109] The construction module 501 further includes a reference point determination unit and a modeling unit.

[0110] The reference point determination unit is configured to mark an anatomical reference point associated with the bifurcation point identification in the dynamic tomographic image, and associate the position information of the anatomical reference point to the bifurcation point identification, the position variation coefficient of the anatomical reference point at multiple respiratory phases being less than a set coefficient.

[0111] The modeling unit is configured to record the position offset of the bifurcation point relative to the anatomical reference point at multiple respiratory phases, and establish a respiratory deformation correlation model.

[0112] The third feature extraction unit is further configured to find the coordinate offset of the positioning data relative to the anatomical reference point at the current respiratory phase based on the respiratory deformation correlation model, and perform respiratory compensation on the positioning data based on the coordinate offset to obtain compensated positioning data.

[0113] The display module 503 includes a feature determination unit, a dynamic navigation marking unit, a static navigation marking unit, a feature marking unit, a risk marking and a path marking unit.

[0114] The feature determination unit is configured to determine the geometric feature of the target bifurcation point based on the target bifurcation point identification.

[0115] The dynamic navigation marking unit is configured to generate a dynamic navigation frame when the diameter of the target bifurcation point is less than a set diameter and the respiratory offset is greater than a first set offset, the frame size of the dynamic navigation frame changes with the actual diameter, and a high-risk label is displayed.

[0116] The static navigation marking unit is configured to generate a static navigation frame when the diameter of the target bifurcation point is greater than or equal to a set diameter and the respiratory offset is less than a second set offset, and the frame size of the static navigation frame is a set size.

[0117] The feature marking unit is configured to generate feature marks according to the geometric features and the mechanical features.

[0118] The risk marking and path marking unit is configured to generate risk marks and path marks according to the position features, and the risk marks and the path marks are different in visual features.

[0119] The correction module 504 can include a monitoring unit and an updating unit.

[0120] The detection unit is configured to, in response to the position deviation value being greater than a set position deviation value, extract a current actual multi-modal feature and calculate a feature deviation value of the actual multi-modal feature and a target multi-modal feature associated with the target bifurcation point identifier.

[0121] The updating unit is configured to correct the target multi-modal feature according to the order of the position features, the geometric features and the mechanical features based on the feature deviation value, and update the target AR mark synchronously according to the corrected target multi-modal feature.

[0122] The navigation device 500 of the bronchoscope according to the embodiments of the present application can further include a pre-warning module. The pre-warning module can include a first prompting unit, a second prompting unit and a third prompting unit.

[0123] The first prompting unit is configured to fine-tune the end trajectory of the path if the position deviation value is less than a first deviation value.

[0124] The second prompting unit is configured to generate a mechanical feedback prompt and display a deceleration prompt if the position deviation value is greater than or equal to the first deviation value but less than a second deviation value.

[0125] The third prompting unit is configured to generate a mechanical feedback prompt and an acoustic feedback prompt, and simultaneously activate a locking mechanism of the bronchoscope if the position deviation value is greater than or equal to the second deviation value.

[0126] The embodiments of the present application also provide a computer readable storage medium, which stores a program capable of being loaded by a processor and executing any one of the bronchoscope navigation methods according to the embodiments of the present application.

[0127] Those skilled in the art can understand that all or part of the functions of various methods in the above embodiments can be realized by hardware or by a computer program. When all or part of the functions in the above embodiments are realized by a computer program, the program can be stored in a computer readable storage medium, which can include a read-only memory, a random access memory, a magnetic disk, an optical disk, a hard disk, and the like. The above functions are realized by executing the program by a computer. For example, the program is stored in a memory of a device, and when the program in the memory is executed by a processor, the above functions are realized. In addition, when all or part of the functions in the above embodiments are realized by a computer program, the program can also be stored in a storage medium such as a server, another computer, a disk, an optical disk, a flash disk, or a mobile hard disk, and is downloaded or copied into a memory of a local device or is updated in version to a system of the local device. When the program in the memory is executed by a processor, all or part of the functions in the above embodiments are realized.

[0128] The above application is described by using specific examples, which is only used to help understand the application and does not limit the application. According to the idea of the application, those skilled in the art can make several simple deductions, modifications, or substitutions.

Claims

1. A navigation device for a bronchoscope, characterized in that, include: The construction module is used to build a mapping library based on dynamic tomographic images of the patient's bronchial region. The mapping library includes the mapping relationship between the bifurcation points of the bronchial tree, multimodal features, and AR markers. The multimodal features include geometric features, mechanical features, and location features. The AR markers include location markers, feature markers, risk markers, and path markers. The matching module is used to acquire multimodal data of the target bifurcation point in real time during the bronchoscopy's movement along the initial path and extract the target multimodal features corresponding to the multimodal data. The target multimodal features are matched with the mapping library to determine the target bifurcation point identifier corresponding to the target bifurcation point. The multimodal data includes optical image data, ultrasonic echo data and positioning data. The display module is used to determine the target AR marker corresponding to the target bifurcation point identifier based on the mapping library, and to overlay the target AR marker onto the field of view of the bronchoscope through an AR display device; The correction module is used to monitor the position deviation between the target AR marker's position and the positioning data in real time, and in response to the position deviation being greater than a set position deviation, to update the target multimodal features corresponding to the target bifurcation point identifier in the mapping library in real time. The matching module includes: The first feature extraction unit is used to extract the outline of the bronchial tree in the optical image data through an edge detection algorithm, detect the axis of the main branch and the branch in the bronchial tree through Hough transform, and determine the bifurcation angle of the target bifurcation point, the diameter ratio of the main branch and the branch where the target bifurcation point is located, and the number of bifurcations, so as to obtain the geometric features of the target bifurcation point. The second feature extraction unit is used to calculate the pipe wall thickness at the target bifurcation point based on the ultrasonic echo data, convert the peak intensity of the ultrasonic echo data into the elastic modulus, and calculate the stiffness coefficient of the target bifurcation point based on the pipe wall thickness and the elastic modulus to obtain the mechanical characteristics of the target bifurcation point. The third feature extraction unit is used to perform respiratory compensation on the positioning data based on the current respiratory phase to obtain compensated positioning data, determine the anatomical grade of the target bifurcation point based on the compensated positioning data, and calculate the risk distance parameter corresponding to the compensated positioning data to obtain the positional features of the target bifurcation point. The cross-validation unit is used to perform cross-validation on the geometric features, mechanical features and positional features of the target bifurcation point, and determine the target multimodal features of the target bifurcation point based on the cross-validation results.

2. The navigation device for a bronchoscope according to claim 1, characterized in that, The building module includes: The acquisition unit is used to acquire dynamic tomographic images of the bronchial region of the patient at multiple respiratory phases, construct a three-dimensional dynamic model of the bronchial tree, identify the bifurcation points in the three-dimensional dynamic model, and extract the geometric, mechanical and positional features of the bifurcation points. The encoding unit is used to encode the bifurcation point based on the multimodal features of each bifurcation point to obtain a unique identifier for each bifurcation point, and to use the unique identifier as the bifurcation point identifier of the bifurcation point. A generation unit is configured to generate the location marker and the risk marker based on the location features associated with the bifurcation point identifier, generate the feature marker based on the geometric features and the mechanical features associated with the bifurcation point identifier, generate a navigation strategy based on the geometric features, the mechanical features and the location features associated with the bifurcation point identifier, and generate a path marker based on the navigation strategy, wherein the path marker includes suggested operating parameters.

3. The navigation device for a bronchoscope according to claim 2, characterized in that, The encoding unit is also used for: Based on the location features of the bifurcation point, the anatomical structure corresponding to the bifurcation point in the bronchial tree is determined, and the anatomical grade of the bifurcation point is determined based on the anatomical structure. The anatomical grade is encoded to obtain a first sub-identifier. The anatomical structure includes the main bronchus, lobar bronchus, segmental bronchus, subsegmental bronchus, and bronchioles. Based on the positional features of the bifurcation point, the three-dimensional coordinates of the bifurcation point in the three-dimensional dynamic model are determined, and the three-dimensional coordinates are encoded to obtain a second sub-identifier; Based on the geometric features of the bifurcation point, the feature type of the bifurcation point is determined, and the feature type is encoded to obtain a third sub-identifier; The first sub-identifier, the second sub-identifier, and the third sub-identifier are combined to obtain the unique identifier of the bifurcation point.

4. The navigation device for a bronchoscope according to claim 1, characterized in that, The building module also includes: A reference point determination unit is used to mark anatomical reference points associated with the bifurcation point identifier in the dynamic tomographic image, and associate the position information of the anatomical reference points with the bifurcation point identifier, wherein the position variation coefficient of the anatomical reference points in multiple respiratory phases is less than a set coefficient. A modeling unit is used to record the positional offset of the bifurcation point relative to the anatomical reference point under multiple respiratory phases, and to establish a respiratory deformation correlation model. The third feature extraction unit is also used for: Based on the respiratory deformation correlation model, find the coordinate offset of the positioning data relative to the anatomical reference point in the current respiratory phase; Breathing compensation is performed on the positioning data based on the coordinate offset to obtain the compensated positioning data.

5. The navigation device for a bronchoscope according to claim 1, characterized in that, The display module includes: A feature determination unit is configured to determine the geometric features of the target bifurcation point based on the target bifurcation point identifier; A dynamic navigation marker unit is used to generate a dynamic navigation box when the diameter of the target bifurcation point is smaller than a set diameter and the breathing offset is greater than a first set offset. The size of the dynamic navigation box changes with the actual diameter and displays a high-risk label. A static navigation marker unit is used to generate a static navigation frame when the diameter of the target bifurcation point is greater than or equal to the set diameter and the breathing offset is less than the second set offset. The frame size of the static navigation frame is a set size. A feature marking unit is used to generate the feature markings based on the geometric features and the mechanical features; A risk marker and a path marker unit is used to generate the risk marker and the path marker based on the location features, wherein the risk marker and the path marker have different visual features.

6. The navigation device for a bronchoscope according to claim 1, characterized in that, The correction module includes: The detection unit is configured to, in response to the position deviation value being greater than the set position deviation value, extract the current actual multimodal features and calculate the feature deviation value of the target multimodal features associated with the actual multimodal features and the target bifurcation point identifier; The update unit is used to correct the target multimodal features based on the feature deviation value in the order of positional features, geometric features, and mechanical features, and to update the target AR tag synchronously according to the corrected target multimodal features.

7. The navigation device for a bronchoscope according to claim 6, characterized in that, It also includes an early warning module, which includes: The first prompting unit is used to fine-tune the end trajectory of the path if the position deviation value is less than the first deviation value. The second prompting unit is used to generate a mechanical feedback prompt and display a deceleration prompt if the position deviation value is greater than or equal to the first deviation value but less than the second deviation value. The third prompting unit is used to generate a mechanical feedback prompt and an audible feedback prompt if the positional deviation value is greater than or equal to the second deviation value, and at the same time activate the locking mechanism of the bronchoscope.

Images