Dual-scale multi-mode bronchial focus navigation method and device

Abstract

The invention provides a dual-scale multi-mode bronchial focus navigation method and device, and belongs to the technical field of medical instrument navigation, and the method comprises the following steps: carrying out fusion processing on electromagnetic pose information and cavity structural features, and carrying out dynamic matching on a fusion result and a three-dimensional airway model, the real-time spatial position of the bronchoscope in the three-dimensional airway model is obtained; guiding the bronchoscope to move to the target area based on the deviation relationship between the real-time spatial position and the planned path; controlling the bronchoscope to extend out of the confocal imaging probe, and receiving microstructure information acquired by the confocal imaging probe; and determining whether the bronchoscope reaches the area where the target focus is located based on the microstructure structure information. Through electromagnetism, vision and model three-source fusion positioning, deviation-driven active navigation guidance and macroscopic-microscopic dual-scale information linkage, the precision, efficiency and credibility of bronchoscope navigation are improved.

Description

Technical Field

[0001] This application relates to the field of medical device navigation technology, and in particular to a dual-scale multimodal bronchial lesion navigation method and device. Background Technology

[0002] Bronchoscopy is a crucial tool for diagnosing lung lesions, especially peripheral lung lesions. With the widespread use of low-dose CT (Computed Tomography) screening, an increasing number of small, deep lung nodules are being detected early, placing higher demands on precise biopsies. However, due to the complex branching of peripheral bronchi, the small size of lesions, and the lack of clear endoscopic landmarks, conventional bronchoscopy often struggles to accurately reach the target area, resulting in a low biopsy positivity rate. Therefore, developing efficient and precise bronchial lesion navigation technology is of great significance for improving the diagnostic capabilities for diseases such as early-stage lung cancer.

[0003] Currently, the mainstream bronchial navigation methods mainly include electromagnetic navigation bronchoscopy (ENB), endoscopic visual navigation, and local microscopic imaging techniques. ENB tracks the instrument position in real time through electromagnetic fields, which improves the reach rate of peripheral lesions, but it is susceptible to electromagnetic interference and lacks intuitive anatomical references. Visual navigation uses endoscopic images for autonomous positioning and path inference, which can reduce the dependence on preoperative images, but its reliability is insufficient due to the limited lighting and field of view in the deep bronchi, and it often relies on high-cost robotic systems. While microscopic imaging techniques such as ultrasound (e.g., r-EBUS) or confocal laser endoscopy (CLE) can provide local evidence of lesions, the former has limited resolution and cannot distinguish cells, while the latter can only observe cellular tissue. Both are difficult to link with macroscopic navigation coordinates, resulting in the final positioning still relying on empirical blind sampling.

[0004] In conclusion, achieving precise global and local navigation of the bronchoscopy system has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a dual-scale multimodal bronchial lesion navigation method and device to address the shortcomings of existing bronchial lesion navigation technologies in that they lack effective integration between macroscopic path guidance and microscopic tissue identification.

[0006] This application provides a dual-scale, multimodal bronchial lesion navigation method, comprising the following steps: Electromagnetic pose information and cavity structure features are fused together, and the fusion result is dynamically matched with a three-dimensional airway model to obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. Based on the deviation between the real-time spatial location and the planned path, the bronchoscope is guided to move to the target area; The bronchoscope is controlled to extend the confocal imaging probe to receive microscopic tissue structure information acquired by the confocal imaging probe; Based on the microscopic tissue structure information, it is determined whether the bronchoscope has reached the area where the target lesion is located.

[0007] According to the dual-scale multimodal bronchial lesion navigation method provided in this application, the three-dimensional airway model is constructed based on the following steps: Using a preset image processing algorithm, the airway cavity is extracted from the CT image to obtain a binary mask of the airway. The binary mask is reconstructed in three dimensions to obtain a three-dimensional airway model.

[0008] According to the dual-scale multimodal bronchial lesion navigation method provided in this application, before guiding the bronchoscope to the target area based on the deviation relationship between the real-time spatial location and the planned path, the method further includes: The centerline of the three-dimensional airway model is extracted to obtain a centerline network, which runs through the main trunk of the airway to the end. Starting from the airway inlet, a preset search algorithm is used to search for a path leading to the area where the target lesion is located, and a planned path is obtained.

[0009] According to the dual-scale multimodal bronchial lesion navigation method provided in this application, determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information includes: If the microscopic tissue structure information matches the microscopic tissue characteristics of the target lesion, then it is determined that the bronchoscope has reached the area where the target lesion is located.

[0010] According to the dual-scale multimodal bronchial lesion navigation method provided in this application, after determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information, the method further includes: If the microstructure information does not match the microstructure characteristics of the target lesion, the bronchoscope will continue to move based on the deviation between the real-time spatial position and the planned path until the microstructure information acquired by the confocal imaging probe matches the microstructure characteristics of the target lesion.

[0011] According to the dual-scale multimodal bronchial lesion navigation method provided in this application, after determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information, the method further includes: If the microstructure information matches the microstructure characteristics of the target lesion, the position of the end of the bronchoscope is locked. Control the bronchoscope to perform biopsy sampling in the locked position.

[0012] This application also provides a dual-scale multimodal bronchial lesion navigation device, comprising the following modules: The multimodal fusion module is used to: fuse electromagnetic pose information with cavity structure features, dynamically match the fusion result with a three-dimensional airway model, and obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. The path navigation module is used to guide the bronchoscope to the target area based on the deviation between the real-time spatial location and the planned path. The probe control module is used to: control the bronchoscope to extend the confocal imaging probe and receive the microscopic tissue structure information collected by the confocal imaging probe; The region determination module is used to determine whether the bronchoscope has reached the region where the target lesion is located, based on the microscopic tissue structure information.

[0013] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dual-scale multimodal bronchial lesion navigation method as described above.

[0014] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the dual-scale multimodal bronchial lesion navigation method as described above.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the dual-scale multimodal bronchial lesion navigation method as described above.

[0016] The dual-scale multimodal bronchial lesion navigation method and device provided in this application fuse electromagnetic pose information with cavity structural features. This fusion process not only uses electromagnetic data to provide global coordinate reference, but also verifies whether the current anatomical branch is consistent with the corresponding position in the three-dimensional airway model through structural features, significantly improving the spatial consistency and anatomical accuracy of macroscopic navigation. Based on the deviation relationship between the real-time spatial position obtained by fusion positioning and the planned path, the geometric deviation between it and the preoperative planned path can be calculated, and navigation guidance instructions can be generated accordingly to continuously optimize the path and drive the bronchoscope to gradually approach the target. When entering the target's vicinity, the confocal imaging probe is automatically triggered to extend and acquire microscopic images of local tissues. Since the confocal imaging probe extends and images under a known precise spatial position (from fusion positioning), the acquired microscopic tissue structure information can be mapped back to specific coordinate points in the three-dimensional airway model. Therefore, it is possible to further compare whether there are nodules, abnormal density, or other imaging features at this location in the preoperative CT, achieving multimodal cross-validation, thereby greatly improving the confidence and diagnostic efficiency of lesion determination. In summary, this application effectively overcomes the core defects of existing bronchial navigation technologies, such as inaccurate positioning, disconnected guidance, and isolated verification, by integrating electromagnetic, visual, and model-based three-source positioning, deviation-driven active navigation guidance, and macro-micro dual-scale information linkage, thereby improving the accuracy, efficiency, and reliability of bronchoscopic navigation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the dual-scale multimodal bronchial lesion navigation method provided in this application; Figure 2 This is a schematic diagram of the bronchoscope structure with the integrated confocal imaging probe provided in this application; Figure 3 This is a schematic diagram of the bronchial lesion examination provided in this application; Figure 4 This is a schematic diagram of the structure of the dual-scale multimodal bronchial lesion navigation device provided in this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that in the description of the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, or integral connections; they can be mechanical connections or electrical connections; they can be direct connections or indirect connections through an intermediate medium; and they can be internal connections between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0021] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0022] The following is combined with Figures 1-5 This application describes the dual-scale multimodal bronchial lesion navigation method and apparatus provided in its embodiments.

[0023] Figure 1 This is a flowchart illustrating the dual-scale multimodal bronchial lesion navigation method provided in this application, as shown below. Figure 1 As shown, the method includes the following: S110, the electromagnetic pose information and cavity structure features are fused together, and the fusion result is dynamically matched with the three-dimensional airway model to obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. S120, based on the deviation between the real-time spatial position and the planned path, guide the bronchoscope to move to the target area; S130, control the bronchoscope to extend the confocal imaging probe and receive the microscopic tissue structure information collected by the confocal imaging probe; S140, Based on the microscopic tissue structure information, determine whether the bronchoscope has reached the area where the target lesion is located.

[0024] In this embodiment, the executing entity of the dual-scale multimodal bronchial lesion navigation method can be a dual-scale multimodal bronchial lesion navigation device. This dual-scale multimodal bronchial lesion navigation device may include, but is not limited to, servers, computer devices such as mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs). The executing entity of the dual-scale multimodal bronchial lesion navigation method can also be a dual-scale multimodal bronchial lesion navigation system, which is subordinate to the dual-scale multimodal bronchial lesion navigation device.

[0025] In this embodiment, electromagnetic pose information refers to the real-time acquisition of the sensor's three-dimensional position (x, y, z) and attitude (pitch angle, yaw angle, roll angle) within a known spatial electromagnetic field generated by an external electromagnetic field generator, by integrating a miniature electromagnetic sensor at the end of the bronchoscope or its working channel. This information reflects the absolute spatial state of the instrument within the patient's body. Cavity structural features refer to the anatomical semantic information extracted from real-time endoscopic videos captured by the bronchoscope's front-end camera. These features may include, but are not limited to, bronchial bifurcation morphology, changes in lumen diameter, mucosal texture, vascular course, and carina angle. These features can be quantified using computer vision algorithms. The three-dimensional airway model is an individualized three-dimensional digital model of the bronchial tree constructed based on the patient's preoperative high-resolution chest CT images using image segmentation, centerline extraction, and surface reconstruction techniques. It includes the spatial orientation, branch topology, and diameter information of each level of bronchus.

[0026] In S110, after the bronchoscope enters the airway, the system acquires the spatial pose information of the bronchoscope tip in real time. At the same time, the bronchoscope endoscope camera collects real-time images of the bronchus. The current cavity structure features are extracted from the real-time images, and the electromagnetic pose information is fused with the cavity structure features and dynamically matched with the three-dimensional airway model. This allows the system to determine the macroscopic spatial position of the bronchoscope in the airway model in real time and output the real-time spatial position and anatomical semantic label of the bronchoscope tip in the three-dimensional airway model.

[0027] Understandably, relying solely on electromagnetic data may lead to drift due to metallic interference or soft tissue deformation; relying solely on visual features may result in inaccuracies in distal small bronchi due to reflections, mucus obstruction, or other reasons. This application fuses both approaches, allowing the system to automatically correct for the correct branch. For example, if the electromagnetic coordinates point to branch B4, but the visual features better match branch B5, the system calculates the geometric consistency score between the theoretical structural features and the actual cavity structural features of the two candidate branches, thereby determining the correct branch.

[0028] In S120, the system calculates the spatial deviation vector between the current real-time position and the planned path, including distance error and direction error, and guides the bronchoscope to advance continuously along the planned path, so that the bronchoscope gradually approaches the target area on a macroscopic scale, that is, the airway branch area where the target lesion is located.

[0029] In S130, when the system determines that the bronchoscope has entered the area adjacent to the target lesion, such as being less than 10mm from the center of the lesion on CT, it automatically triggers a command, such as... Figure 2 As shown, the confocal probe channel of the bronchoscope extends out of the confocal imaging probe, as... Figure 3 As shown, the confocal imaging probe is brought close to the bronchial wall or the surface of the suspected lesion area to perform real-time high-resolution microscopic imaging of the local submucosal tissue, obtain the microscopic tissue structure information of the corresponding area, and thus determine whether the bronchoscope has reached the target area based on the microscopic tissue structure information.

[0030] Optionally, the system synchronously displays a confocal image, showing the precise coordinates of the imaging location in the three-dimensional airway model, as well as the lesion image in the corresponding CT slice, thus facilitating the doctor's determination of whether the end of the bronchoscope has reached the target area.

[0031] In S140, based on microscopic tissue structure information, such as cellular atypia, basement membrane damage, and microvascular proliferation, it is determined whether the current bronchoscope tip accurately hits the target lesion area at the tissue scale.

[0032] The dual-scale multimodal bronchial lesion navigation method provided in this application integrates electromagnetic pose information with cavity structural features. This fusion process not only utilizes electromagnetic data to provide global coordinate references but also verifies whether the current anatomical branch is consistent with the corresponding position in the three-dimensional airway model through structural features, significantly improving the spatial consistency and anatomical accuracy of macroscopic navigation. Based on the deviation relationship between the real-time spatial position obtained from the fusion positioning and the planned path, the geometric deviation from the preoperative planned path can be calculated, and navigation guidance instructions can be generated accordingly to continuously optimize the path and drive the bronchoscope to gradually approach the target. When entering the target's vicinity, the confocal imaging probe is automatically triggered to extend and acquire microscopic images of local tissues. Since the confocal imaging probe extends and images from a known precise spatial position (from the fusion positioning), the acquired microscopic tissue structure information can be mapped back to specific coordinate points in the three-dimensional airway model. Therefore, it is possible to further compare whether there are nodules, abnormal density, or other imaging features at this location in the preoperative CT scan, achieving multimodal cross-validation, thereby significantly improving the confidence and diagnostic efficiency of lesion determination. In summary, this application effectively overcomes the core defects of existing bronchial navigation technologies, such as inaccurate positioning, disconnected guidance, and isolated verification, by integrating electromagnetic, visual, and model-based three-source positioning, deviation-driven active navigation guidance, and macro-micro dual-scale information linkage, thereby improving the accuracy, efficiency, and reliability of bronchoscopic navigation.

[0033] In an optional embodiment, the three-dimensional airway model is constructed based on the following steps: Using a preset image processing algorithm, the airway cavity is extracted from the CT image to obtain a binary mask of the airway. The binary mask is reconstructed in three dimensions to obtain a three-dimensional airway model.

[0034] In this embodiment, the patient's preoperative chest CT image data is acquired, the airway region is segmented and reconstructed in three dimensions, and a three-dimensional airway model is constructed.

[0035] Specifically, patients undergo low-dose or conventional high-resolution chest CT scans, with images covering the entire lung to ensure the inclusion of the target peripheral lesion and adjacent bronchial structures. Further, air-containing regions are coarsely extracted using a thresholding method, and morphological manipulations are used to remove non-airway structures such as blood vessels and the chest wall. Optionally, a deep learning model is used for fine segmentation of the lung parenchyma.

[0036] Furthermore, tubular structures are enhanced by airway enhancement filtering. In the enhanced image, a local adaptive threshold or machine learning classifier is used to distinguish airway cavities from surrounding tissues. Starting from the main trachea (known anatomical location), region growth is performed along the enhancement response direction to ensure that only structures connected to the main airway are preserved, isolated artifacts are eliminated, and a three-dimensional binary mask is output, where 1 represents the airway cavity and 0 represents the non-airway region.

[0037] Furthermore, isosurfaces of the airway lumen are extracted from the binary mask to generate a triangular mesh model. Then, the centralline skeleton of the bronchial tree is extracted from the mask for path planning and branch labeling. "Spices" or false branches caused by partial volume effects or noise, such as terminal protrusions with a length of <2mm, are removed. Based on the centralline bifurcation hierarchy and standard bronchial naming rules, semantic labels such as RB1 and LB8 are automatically assigned to each level of branch. Finally, a complete three-dimensional airway model containing geometry, centralline topology, and anatomical semantic labels is output.

[0038] The dual-scale multimodal bronchial lesion navigation method provided in this application accurately restores the actual direction, bifurcation angle, and diameter changes of the bronchus in the constructed three-dimensional airway model, and retains the small distal branches near the lesion, thereby making the planned path closer to the patient's actual anatomy, providing high-fidelity data for subsequent electromagnetic-visual fusion positioning, and fundamentally reducing the systematic deviation of macroscopic navigation.

[0039] In an optional embodiment, before guiding the bronchoscope to the target area based on the deviation between the real-time spatial location and the planned path, the method further includes: The centerline of the three-dimensional airway model is extracted to obtain a centerline network, which runs through the main trunk of the airway to the end. Starting from the airway inlet, a preset search algorithm is used to search for a path leading to the area where the target lesion is located, and a planned path is obtained.

[0040] In this embodiment, the centerline of the three-dimensional airway model is extracted to obtain a centerline network, thereby transforming the surface mesh-like three-dimensional airway model into a topologically clear and geometrically continuous skeleton representation, which facilitates path planning and navigation control. Furthermore, starting from the airway inlet, a preset search algorithm is used to search for a path leading to the target lesion area, resulting in a planned path. The path is stored in the form of a discrete point series or spline curve for subsequent real-time deviation calculation.

[0041] Preferably, by constraining the tube diameter to eliminate branches that the instrument cannot pass through, and by optimizing the curvature to avoid excessively curved paths, the bronchoscope's passability is improved, so that the final planned path obtained by the search is not only the shortest geometric distance, but also a clinically feasible and instrument-accessible navigation route.

[0042] The specific search algorithm used here is not limited. For example, Dijkstra's algorithm, A* algorithm, and constrained shortest path algorithm can be used.

[0043] The dual-scale multimodal bronchial lesion navigation method provided in this application uses a one-dimensional curve as the centerline path, which has inherent directionality and parameterization properties. During real-time navigation, the system can project the real-time spatial position onto the nearest centerline point to obtain the progress and lateral offset along the path, clearly defining the deviation and providing a high-precision, low-dimensional reference trajectory for real-time navigation, generating intuitive guidance instructions. Furthermore, while the aforementioned electromagnetic and visual corrections rely on enumerating candidate branches, the centerline network provides an explicit branch topology graph. Each node / edge has a unique ID and parent-child relationship, allowing the system to quickly determine whether it is still within a subtree of the planned path. If an entry into a non-planned branch is detected, an alarm can be immediately triggered or replanning can be initiated, improving the topological safety of navigation.

[0044] In an optional embodiment, determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information includes: If the microscopic tissue structure information matches the microscopic tissue characteristics of the target lesion, then it is determined that the bronchoscope has reached the area where the target lesion is located.

[0045] In this embodiment, the microscopic tissue structure information collected in real time by the confocal imaging probe is compared with the expected microscopic tissue characteristics of the target lesion to determine whether the lesion area has been reached.

[0046] Specifically, once the bronchoscope enters a location near the target area based on macroscopic navigation, the system controls the extension of the confocal imaging probe through the working channel. The probe gently touches the bronchial mucosa surface to acquire a real-time video stream. After preprocessing the video frames, a pre-trained feature extraction model is used to extract the microscopic structural features of the current tissue. The similarity between the real-time microscopic structural features and the expected template is calculated. If the similarity meets a set threshold, it is determined that the microscopic tissue structure information matches the microscopic tissue features of the target lesion. The system comprehensively judges that the macroscopic positioning is close to the lesion and the microscopic features highly match the expected lesion pattern, confirming that the bronchoscope has reached the area where the target lesion is located.

[0047] The dual-scale multimodal bronchial lesion navigation method provided in this application directly verifies whether the tissue seen at this moment has the typical pathological manifestations of the target lesion at the cellular level through microscopic tissue feature matching. It further refines the navigation endpoint from geometric proximity to pathological consistency, fundamentally solving the clinical pain point of not being able to obtain lesions and significantly improving the biopsy positive rate. In addition, current peripheral lesion biopsies often require multiple attempts, and each puncture increases the risk of pneumothorax and bleeding. However, this embodiment only determines that the target area has been reached after confirming the microscopic feature matching, and then performs a physical biopsy, avoiding repeated sampling of normal lung tissue and reducing the risk of complications.

[0048] In an optional embodiment, after determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information, the method further includes: If the microstructure information does not match the microstructure characteristics of the target lesion, the bronchoscope will continue to move based on the deviation between the real-time spatial position and the planned path until the microstructure information acquired by the confocal imaging probe matches the microstructure characteristics of the target lesion.

[0049] In this embodiment, the microscopic imaging results are used as the basis for determining the navigation endpoint, and a new round of path correction and approximation is automatically triggered when the target is not reached.

[0050] Specifically, when the bronchoscope determines that it is close to the target lesion area based on macroscopic navigation, the system automatically controls the confocal imaging probe to extend, acquire real-time microscopic images of the current mucosal surface, extract microscopic tissue structure features, and match them with the pre-set microscopic feature template of the target lesion. If the matching degree is less than the set threshold, that is, the microscopic tissue structure information does not match the characteristics of the target lesion, the re-navigation cycle is started. The system obtains the latest real-time spatial position of the current bronchoscope tip, calculates the geometric deviation between the position and the original planned path, and moves according to the deviation. After each fine adjustment or movement, the system extends the confocal imaging probe again to acquire new images, repeating the process of acquisition, feature extraction, and matching judgment. Once a match is successful, navigation is immediately terminated and the position is locked.

[0051] Optionally, to avoid infinite loops, set a maximum number of attempts, such as 3–5 times, or a maximum exploration radius. If the limit is exceeded, indicate that the confocal imaging probe has not confirmed the lesion and suggest combining it with other methods.

[0052] In some embodiments, when the confocal imaging probe is still on the planned path but has not reached the destination, it continues to move along the original planned path.

[0053] In some embodiments, when operational deviations or anatomical variations cause a wrong turn and a deviation from the planned path, the path can be reverted to the previous fork point, the correct branch can be re-identified, and a locally replanned path from the current fork point to the target endpoint can be generated.

[0054] In other embodiments, when the confocal imaging probe has reached the endpoint but the microscopic tissue characteristics do not match, it indicates that although the endpoint of the planned path is the nearest bronchial opening, the lesion is actually located in the adjacent lung parenchyma, and the probe is in contact with the normal bronchial wall. The end of the bronchoscope can be controlled to swing or rotate within a preset range of the endpoint, and the confocal imaging probe image can be acquired multiple times from multiple angles.

[0055] The dual-scale multimodal bronchial lesion navigation method provided in this application defines the navigation endpoint as the consistency between the probe imaging and the microscopic features of the lesion. Even if there is a millimeter-level error in the initial planning, the system can still continuously approach the real lesion through microscopic feedback to achieve precise positioning in a biological sense. Even if the endpoint of the planned path is slightly deviated due to CT respiratory phase and reconstruction errors, the system can use the confocal imaging probe as a local sensor to perform adaptive search near the endpoint, thereby enhancing the robustness of the system.

[0056] In an optional embodiment, after determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information, the method further includes: If the microstructure information matches the microstructure characteristics of the target lesion, the position of the end of the bronchoscope is locked. Control the bronchoscope to perform biopsy sampling in the locked position.

[0057] In this embodiment, when the similarity between the real-time extracted microscopic tissue structure features and the pre-set target lesion microscopic feature template is greater than a set threshold, and the macroscopic positioning confirms that the current bronchoscope tip is located in the vicinity of the lesion, it is comprehensively determined that the bronchoscope tip has reached the area where the target lesion is located. The spatial coordinates and instrument posture of the current optimal sampling point are fixed in the navigation system as the benchmark for biopsy operation.

[0058] Specifically, the current fusion localization result, namely the 6-DOF pose after electromagnetic and visual correction, is obtained, and this pose is bound to the specific anatomical location in the 3D airway model to establish a traceable and reproducible sampling coordinate system.

[0059] In some embodiments, the system integrates a flexible robotic bronchoscope or force feedback catheter, automatically enabling fine-tuning servo control to resist respiratory movements or operational jitters and maintain the end point near the locking point, such as within ±0.5 mm of the locking point.

[0060] In other embodiments, if the operation is manual, the current optimal biopsy point is highlighted on the display.

[0061] Furthermore, the system issues instructions to control the working channel to execute the standard biopsy procedure, retract the confocal imaging probe, clear the working channel, insert biopsy forceps or puncture needles, guide the biopsy instruments to advance to a preset depth along the locked axis, perform opening / closing / aspiration actions, obtain tissue samples, and record the biopsy space log, including but not limited to sampling point coordinates, instrument type, needle insertion depth, etc.

[0062] The dual-scale multimodal bronchial lesion navigation method provided in this application first locks the positive location confirmed by the confocal imaging probe, and then samples are taken under this precise coordinate, avoiding false negatives where optics show malignancy and biopsies retrieve normal results, significantly improving the consistency between pathological diagnosis and probe image interpretation. Each biopsy is associated with unique spatial coordinates and multimodal data, achieving standardization and traceability of the biopsy operation. In addition, in robotic bronchoscopy systems, automated biopsy requires a clearly defined target point as the control endpoint. The locking position provided in this application is the ideal execution point, providing reliable triggering conditions and execution benchmarks for automated biopsy.

[0063] The dual-scale multimodal bronchial lesion navigation device provided in the embodiments of this application will be described below. The dual-scale multimodal bronchial lesion navigation device described below can be referred to in correspondence with the dual-scale multimodal bronchial lesion navigation method described above.

[0064] Figure 4 This is a schematic diagram of the structure of the dual-scale multimodal bronchial lesion navigation device provided in this application, as shown below. Figure 4 As shown, this dual-scale multimodal bronchial lesion navigation device may include, but is not limited to: The multimodal fusion module 410 is used to: fuse electromagnetic pose information with cavity structure features, dynamically match the fusion result with a three-dimensional airway model, and obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. The path navigation module 420 is used to guide the bronchoscope to move to the target area based on the deviation between the real-time spatial position and the planned path. The probe control module 430 is used to: control the bronchoscope to extend the confocal imaging probe and receive the microscopic tissue structure information collected by the confocal imaging probe; The region determination module 440 is used to determine whether the bronchoscope has reached the region where the target lesion is located, based on the microscopic tissue structure information.

[0065] It should be noted that the dual-scale multimodal bronchial lesion navigation device provided in this application embodiment can execute the dual-scale multimodal bronchial lesion navigation method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0066] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a dual-scale multimodal bronchial lesion navigation method, which includes: Electromagnetic pose information and cavity structure features are fused together, and the fusion result is dynamically matched with a three-dimensional airway model to obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. Based on the deviation between the real-time spatial location and the planned path, the bronchoscope is guided to move to the target area; The bronchoscope is controlled to extend the confocal imaging probe to receive microscopic tissue structure information acquired by the confocal imaging probe; Based on the microscopic tissue structure information, it is determined whether the bronchoscope has reached the area where the target lesion is located.

[0067] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the dual-scale multimodal bronchial lesion navigation method provided by the above methods, the method comprising: Electromagnetic pose information and cavity structure features are fused together, and the fusion result is dynamically matched with a three-dimensional airway model to obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. Based on the deviation between the real-time spatial location and the planned path, the bronchoscope is guided to move to the target area; The bronchoscope is controlled to extend the confocal imaging probe to receive microscopic tissue structure information acquired by the confocal imaging probe; Based on the microscopic tissue structure information, it is determined whether the bronchoscope has reached the area where the target lesion is located.

[0069] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the dual-scale multimodal bronchial lesion navigation method provided by the methods described above, the method comprising: Electromagnetic pose information and cavity structure features are fused together, and the fusion result is dynamically matched with a three-dimensional airway model to obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. Based on the deviation between the real-time spatial location and the planned path, the bronchoscope is guided to move to the target area; The bronchoscope is controlled to extend the confocal imaging probe to receive microscopic tissue structure information acquired by the confocal imaging probe; Based on the microscopic tissue structure information, it is determined whether the bronchoscope has reached the area where the target lesion is located.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A dual-scale, multimodal bronchial lesion navigation method, characterized in that, include: Electromagnetic pose information and cavity structure features are fused together, and the fusion result is dynamically matched with a three-dimensional airway model to obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. Based on the deviation between the real-time spatial location and the planned path, the bronchoscope is guided to move to the target area; The bronchoscope is controlled to extend the confocal imaging probe to receive microscopic tissue structure information acquired by the confocal imaging probe; Based on the microscopic tissue structure information, it is determined whether the bronchoscope has reached the area where the target lesion is located.

2. The dual-scale multimodal bronchial lesion navigation method according to claim 1, characterized in that, The three-dimensional airway model was constructed based on the following steps: Using a preset image processing algorithm, the airway cavity is extracted from the CT image to obtain a binary mask of the airway. The binary mask is reconstructed in three dimensions to obtain a three-dimensional airway model.

3. The dual-scale multimodal bronchial lesion navigation method according to claim 2, characterized in that, Before guiding the bronchoscope to the target area based on the deviation between the real-time spatial location and the planned path, the method further includes: The centerline of the three-dimensional airway model is extracted to obtain a centerline network, which runs through the main trunk of the airway to the end. Starting from the airway inlet, a preset search algorithm is used to search for a path leading to the area where the target lesion is located, and a planned path is obtained.

4. The dual-scale multimodal bronchial lesion navigation method according to claim 1, characterized in that, The step of determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information includes: If the microscopic tissue structure information matches the microscopic tissue characteristics of the target lesion, then it is determined that the bronchoscope has reached the area where the target lesion is located.

5. The dual-scale multimodal bronchial lesion navigation method according to claim 1, characterized in that, After determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information, the method further includes: If the microstructure information does not match the microstructure characteristics of the target lesion, the bronchoscope will continue to move based on the deviation between the real-time spatial position and the planned path until the microstructure information acquired by the confocal imaging probe matches the microstructure characteristics of the target lesion.

6. The dual-scale multimodal bronchial lesion navigation method according to claim 1, characterized in that, After determining whether the bronchoscope has reached the target lesion area based on the microscopic tissue structure information, the method further includes: If the microstructure information matches the microstructure characteristics of the target lesion, the position of the end of the bronchoscope is locked. Control the bronchoscope to perform biopsy sampling in the locked position.

7. A dual-scale, multimodal bronchial lesion navigation device, characterized in that, include: The multimodal fusion module is used to: fuse electromagnetic pose information with cavity structure features, dynamically match the fusion result with a three-dimensional airway model, and obtain the real-time spatial position of the bronchoscope in the three-dimensional airway model. The electromagnetic pose information is the spatial pose information of the end of the bronchoscope, and the cavity structure features are the structural features of the cavity where the end of the bronchoscope is located. The path navigation module is used to guide the bronchoscope to the target area based on the deviation between the real-time spatial location and the planned path. The probe control module is used to: control the bronchoscope to extend the confocal imaging probe and receive the microscopic tissue structure information collected by the confocal imaging probe; The region determination module is used to determine whether the bronchoscope has reached the region where the target lesion is located, based on the microscopic tissue structure information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the dual-scale multimodal bronchial lesion navigation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dual-scale multimodal bronchial lesion navigation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the dual-scale multimodal bronchial lesion navigation method as described in any one of claims 1 to 6.

Images